feature request: index.json manifest for releases

[jasonkarns]

Both nodejs.org and iojs.org maintain an index.json file from their respective download pages. The json file is essentially a manifest of paths and checksums for all their releases. This is a lovely file that makes it possible to easily scrape their downloads page (or rather, just the manifest) for new releases.

https://nodejs.org/dist/index.json

It would be awesome if jxcore had a similar manifest file on its download page.

The manifest file could have, for each jxcore version, a listing of the names for the source tarball, compiled tarballs for each platform, and version information about which v8, spidermonkey, npmjx, node, and npm are bundled.

The carrot for this particular feature, is that the node-build scraper, which currently adds new builds of iojs and node to its definitions by scraping their respective index.json files, would be able to be run against jxcore. This would keep jxcore's definitions up to date without the definitions needing to be created manually (which is simple but tiresome when one accounts for including the package checksums).

[ktrzeciaknubisa]

Would something like this be sufficient?

[
  {
    "node": "0.10.40",
    "jxcore": "0.3.0.7",
    "v8": "3.14.5.9",
    "sm": 0,
    "embedded": { "sqlite": "3.8.4.3" },
    "ares": "1.9.0-DEV",
    "uv": "0.10.36",
    "zlib": "1.2.3",
    "modules": "11",
    "openssl": "1.0.1p",

    "source" : {
      "zip" : "https://github.com/jxcore/jxcore/archive/v0.3.0.7.zip",
      "tar" : "https://github.com/jxcore/jxcore/archive/v0.3.0.7.tar.gz"
    },

    "dist" : {
      "url" : "https://jxcore.s3.amazonaws.com/0307/",
      "platforms" : [
        "jx_ub32sm.zip", "jx_iosFATsm.zip", 
        // etc
      ]
    }
  }
]

[jasonkarns]

Yeah! That's nearly perfect. The only changes I would make, are to include the checksums.

nodejs (and iojs) publish all their distributable packages under a directory per release (as you do). They then include a SHASUMS256.txt within the release directory that has the checksums for each dist file. (example)

You could certainly copy their lead with a checksum file in the release directory. Though you could also just include them in the main manifest.

Perhaps the dist object could look like:

"dist": {
  "url": "https://jxcore.s3.amazonaws.com/0307/",
  "platforms": {
    "ubuntu-x86": {
      "file": "jx_ub32sm.zip",
      "sha256": "1723abf50ee9b2b2209af06374523ae657c5562166bdc44b7b8d32801484c572",
      "sha1": "edca2a0c728388247db251c4a45159863b2d1314"
    },
    //etc
  }
}

So platforms is an object rather than array. And each platform object contains file and checksum keys, for whichever types of checksums you publish. (sha256 recommended, of course, since sha1 is deprecated)

The platforms object could still be an array of these platform objects. But I think being able to 'name' each platform gives you some flexibility besides relying on the filename itself. Since as a scraper of this file, we would need to determine the OS/arch for each dist file. Having a platform name would let you change your filename format in the future. Otherwise, scrapers would need to derive the OS/arch from the filename itself.

[ktrzeciaknubisa]

@jasonkarns I've prepared a full-sample file for you. For now it is a temporary location: https://github.com/ktrzeciaknubisa/jxcore-issues/blob/master/index.json .

I gave up on your proposal to give a name to the platform:

"platforms": {
    "ubuntu-x86": {

because I would need to add an engine to it ("ubuntu-x86-v8", "ubuntu-x86-sm") so you would still need to parse it. However if you still want this way, I can do it.

Also we have some binaries suitable for multiple platforms, e.g. "RH/Centos/Fedora". Would you like me to split them?

[jasonkarns]

so you would still need to parse it

Ah, right, good point. It doesn't make a difference to me on that; either way is fine. Having the properties split out is probably best long term anyway.

Also we have some binaries suitable for multiple platforms, e.g. "RH/Centos/Fedora". Would you like me to split them?

Short answer, no, I don't think that's necessary. node-build derives the host platform (for the most part) from lsb_release -sir. I believe that redhat, centos, and fedora all match *"Red Hat"* from lsb_release -sir, which we then normalize to redhat. So the node-build definitions for redhat/centos/fedora will be a single entry named as redhat-x64:

binary redhat-x64 "https://jxcore.s3.amazonaws.com/0307/jx_rh64v8.zip#929d1266a5979f30eff590c276f0e18cefbd0771f4dd47eaa7294ebee5b0335c"

So this is all 👍 from me!

Since node-build is (for now, at least) the only consumer of this file, would you mind keeping this issue open and not officially publishing this file until we can get our scraper to consume it? That way if we run into any issues, we can address them on our side or your side as is appropriate? I'll open an issue on node-build to update the scraper.

[jasonkarns]

First couple things I've noticed:

1. sha checksums are missing for the source packages
2. version numbers for every version pre 0.3.0.7 has a 'Beta-' prefix, which seems odd. If those are all the betas, where are the non betas for those version numbers? And shouldn't the pre-release designation be suffix instead of prefix? (ie, 0.3.0.6-beta would be a prerelease for 0.3.0.6)

[ktrzeciaknubisa]

Thanks @jasonkarns

1. I'll add those checksums
2. the "Beta-xxx" is a string that we were using for jx -jxv (same as process.jxversion). Thus I left them in index.jxson to be fully correspondent to those strings. Furthermore those Beta releases were only beta. As for prefix/suffix - we're done with betas already. I don't suppose we'll be changing the version string format for those obsolete releases and recompiling all binaries etc.

I also didn't respond to your previous question:

would you mind keeping this issue open and not officially publishing this file until we can get our scraper to consume it?

Sure, no problem :)

[jasonkarns]

Fine to leave as is. Does that mean future releases will comply with semver?

The unfortunate side effect of those particular releases not following semver means that the scraper will generate invalid build definitions for them. I can blacklist Beta-* if no new releases will begin with that prefix.

[jasonkarns]

I guess it's worth mentioning, then, that the existing build definitions for those releases in node-build do not have the Beta- prefix.

See the jxcore releases at the bottom of: https://github.com/OiNutter/node-build/tree/master/share/node-build

[ktrzeciaknubisa]

Does that mean future releases will comply with semver?

I cannot say "yes" at the moment. This is a decision for our team to make.

I can blacklist Beta-* if no new releases will begin with that prefix.

I think you may just remove "Beta" string and things will be fine?

I guess it's worth mentioning, then, that the existing build definitions for those releases in node-build do not have the Beta- prefix.

Yeap, that's what i say.

[jasonkarns]

I think you may just remove "Beta" string and things will be fine?

Yeah, I guess. I would have like to not have that in the scraper in case future versions have a similar prefix.

Another note:

The dists for FreeBSD are identical for FreeBSD 9 and FreeBSD 10. The only differentiator is in the filename/url (jx_bsd9 vs jx_bsd10). As the version of the OS is technically a different platform, would you be opposed to using FreeBSD9/FreeBSD10 or something?

[ktrzeciaknubisa]

in case future versions have a similar prefix

I don't think we're going to follow that pattern anymore.

would you be opposed to using FreeBSD9/FreeBSD10 or something?

Yes, exactly. I'll add the numbers too.

[jasonkarns]

Oh, another thing that I need (one after the other :) )...

I need info of which npmjx should go with each release. Something like this would be sufficient (you could add the npm version numbers if you like, but all I need is the url and sha).

"npm": {
  "url": "https://s3.amazonaws.com/nodejx/npmjx307.tar.gz",
  "sha256": "7b99397f17f5fdb592842ccd62e25806b2a05ae9997c4508b47412f194e8ba6e"
}

[jasonkarns]

I now have a working scraper that's missing only the items already mentioned:

1. sha for source tarball
2. url and sha for npm tarball
3. os version included in freebsd platform name

[ktrzeciaknubisa]

@jasonkarns 1st and 3rd are done, please recheck index.json under same previous link. No 2 is coming next days.

[jasonkarns]

Sweet! 1 and 3 check out

[ktrzeciaknubisa]

@jasonkarns The 2nd is also done now.

[jasonkarns]

nice! just updated the scraper, looks good!

Unless there are any more changes you expect to see, I'm ready to ship the scraper! We just need to know the official URL where it will be hosted! (hopefully under https, since we need to be able to trust the urls and checksums contained therein)

[ktrzeciaknubisa]

I've located it here: https://jxcore.s3.amazonaws.com/releases.json

[jasonkarns]

Merged!! nodenv/node-build-update-defs#4

Thanks!