Wanted: provide a list of version information

[zhmushan]

related: denolib/setup-deno#11
Using Github API will limit the rate.

[MarkTiedemann]

The GitHub API has a rate-limit, but their website does not.

async function versions(set = new Set(), after) {
  let res = await fetch("https://github.com/denoland/deno/releases?after=" + after);
  if (!res.ok) return Deno.exit(1);
  let text = await res.text();
  let lines = text.split("\n");
  let ver;
  for (let line of lines) {
    let match = line.match(/denoland\/deno\/releases\/download\/(.*)\/deno_linux_x64\.gz/);
    if (match !== null) {
      ver = match[1];
      if (set.has(ver)) {
        console.log(Array.from(set).join("\n"));
        return Deno.exit(0);
      } else {
        set.add(ver);
      }
    }
  }
  versions(set, ver);
}

versions();

$ deno --allow-net versions.js
v0.27.0
v0.26.0
v0.25.0
v0.24.0
v0.23.0
v0.22.0
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.16.0
v0.15.0
v0.14.0
v0.13.0
v0.12.0
v0.11.0
v0.10.0
v0.9.0
v0.8.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.3.11
v0.3.10
v0.3.9
v0.3.8
v0.3.7
v0.3.6
v0.3.5
v0.3.4
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.11
v0.2.10
v0.2.9
v0.2.8
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.12
v0.1.11
v0.1.10
v0.1.9
v0.1.8
v0.1.7
v0.1.6
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0

This may be a little brittle, but hopefully it's enough for your use-case for now. :)

[MarkTiedemann]

Even better: Listing all versions with a single request:

async function main() {
  let res = await fetch("https://raw.githubusercontent.com/denoland/deno/master/Releases.md");
  let text = await res.text();
  console.log(
    [...text.matchAll(/### (v\d+\.\d+\.\d+)/g)]
    .map(m => m[1])
    .join("\n")
  );
}

main();

This one also returns v0.0.0. :)

[ry]

It's worth noting that on the website we use the Github API to get directory listings and it's working fine. The reason we don't get rated limited is the API calls are made from the client side, so any rate limitations apply per visitor, not for the whole site. So I think just using AJAX requests to the API is the best way to do this.

[MarkTiedemann]

any rate limitations apply per visitor

The problem is that "per visitor" does not apply to CI environments, such as, GitHub Actions or Travis, or rather any service where you can't decide which server your code is gonna be run on.

The same happened in the CI of our installer, by the way. Don't have a link now, but I remember it failed because it was rate-limited. This is why we're parsing the website to get the latest version: https://github.com/denoland/deno_install/blob/master/install.sh#L24

[ry]

Oh sorry, I misunderstood the issue. I thought this was about providing branch information for deno.land/x/ modules like unpkg.org has.

[zhmushan]

I just wanted a no rate limit api like https://golang.org/dl/?mode=json&include=all

[MarkTiedemann]

@zhmushan

As far as I know, there's no rate-limiting for https://raw.githubusercontent.com. So, to me, this seems like a good temporary work-around until a Release API is provided.

async function getReleases(): Promise<string[]> {
	let res = await fetch("https://raw.githubusercontent.com/denoland/deno/master/Releases.md");
	if (!res.ok) throw new Error(res.statusText);
	let text = await res.text();
	let matches = text.matchAll(/### (v\d+\.\d+\.\d+)/g);
	let releases = [...matches].map(m => m[1]);
	return releases;
}

The code wouldn't be too different either if there was an API.

async function getReleases(): Promise<string[]> {
	let res = await fetch("https://deno.land/releases.json");
	if (!res.ok) throw new Error(res.statusText);
	let releases = await res.json();
	return releases;
}

Same for Node:

const https = require("https");

function getReleases() {
	return new Promise((resolve, reject) => {
		https.get("https://raw.githubusercontent.com/denoland/deno/master/Releases.md", async res => {
			try {
				if (res.statusCode !== 200) throw new Error(res.statusMessage);
				let buf = [];
				for await (let data of res) buf.push(data);
				let text = Buffer.concat(buf).toString();
				let matches = text.matchAll(/### (v\d+\.\d+\.\d+)/g);
				let releases = [...matches].map(m => m[1]);
				resolve(releases);
			} catch (err) {
				reject(err);
			}
		}).on("error", reject);
	});
}

function getReleases() {
	return new Promise((resolve, reject) => {
		https.get("https://deno.land/releases.json", async res => {
			try {
				if (res.statusCode !== 200) throw new Error(res.statusMessage);
				let buf = [];
				for await (let data of res) buf.push(data);
				let text = Buffer.concat(buf).toString();
				let releases = JSON.parse(text);
				resolve(releases);
			} catch (err) {
				reject(err);
			}
		}).on("error", reject);
	});
}

EDIT: There's some more or less hidden assumptions in the above code: e.g. Deno will continue to use a master branch, releases will be documented in the Releases.md file, releases will be documented as a third-level heading (### ...), they will continue to be formatted as v$major.$minor.$patch, etc. So perhaps you may want to try using the GitHub API first, and only use this as a fallback in case rate-limiting actually kicks in.

EDIT 2: On a related note, you can also easily polyfill the Fetch API for Node in such a way that the getReleases function looks the same for both Deno and Node...

function fetch(url) {
  return new Promise((resolve, reject) => {
    https.get(url, res => {
      let text = async () => {
        let buf = [];
        for await (let data of res) buf.push(data);
        return Buffer.concat(buf).toString();
      };
      resolve({
        ok: res.statusCode === 200,
        statusText: res.statusMessage,
        text,
        json: async () => JSON.parse(await text())
      })
    }).on("error", reject);
  });
}

[zhmushan]

@MarkTiedemann Looks good, thank you!

[MarkTiedemann]

Happy to help. :) PS: I edited my comment above.

[brandonkal]

@MarkTiedemann why not just query the RSS feed https://github.com/denoland/deno/releases.atom
That would be cleaner than trying to parse the markdown file.

[MarkTiedemann]

@brandonkal Didn't know about that. That's pretty sweet! Thanks for sharing. :)

[axetroy]

@brandonkal

Nice!

[shian15810]

@brandonkal Fetching from https://github.com/denoland/deno/releases.atom won't provide information about prerelease and draft, so you can't filter out version that is not ready for production, such as this: https://github.com/denoland/deno/releases/tag/v0.3.11

[lucacasonato]

You can get version info from https://cdn.deno.land/MODULENAME/meta/versions.json now. It lists all released versions, and the latest.

[MarkTiedemann]

@lucacasonato This issue is not about the version of a module, but about Deno's version.

This is similar to denoland/deno_install#115.

[lucacasonato]

@lucacasonato This issue is not about the version of a module, but about Deno's version.

https://cdn.deno.land/deno/meta/versions.json

[MarkTiedemann]

https://cdn.deno.land/deno/meta/versions.json

That's nice.

Unfortunately, I think it's not usable for determining the latest Deno version for downloading the latest Deno binary. That is because there is a time difference between the git tag being pushed and the binaries being built and uploaded. For example, v1.2.2 was tagged at 2020-07-31T19:14:25Z, but the GitHub release was created at 2020-07-31T21:06:46Z, almost 2h later, (and the binaries may have been added to the release even later?). That's why, in deno_install, we're currently parsing the HTML of the GitHub releases page to figure out the "latest release for the specific platform" - "latest" meaning "binary is available for download".

A quick and dirty solution would be something like this: https://gist.github.com/MarkTiedemann/6ec5dc6b4f968296fa7e7b578ca91ad6 A better solution would need to be integrated with the release process.

Also note that, ideally, at least for the use case of deno_install, the API would not require any kind of parsing - not even JSON parsing - since that's non-trivial to do in a shell script. That's why https://deno.maju.workers.dev/version/x86_64-pc-windows-msvc, for example, just returns the plain text version.

[lucacasonato]

The versions.json file in this repo contains a list of Deno CLI versions. It is updated manually after artifacts are published.

[MarkTiedemann]

Sweet, that's perfect! Thanks. :)

---

@lucacasonato Where can I find the source code / docs for the API?

[MarkTiedemann]

@lucacasonato Where can I find the source code / docs for the API?

I'd like to add a feature for returning the latest version in plain text. :)

[lucacasonato]

@MarkTiedemann You can find the source code at https://github.com/denoland/deno_registry2. The API is not stable though and is likely to change. There is a little documentation at https://github.com/denoland/deno_registry2/blob/main/API.md. If we want to make the API more extensive I think we should spec it out first. I'll open an issue on the repository.