Downloading node client binaries at runtime

[hiddentao]

Instead of bundling client binaries at build time we will download then on startup at runtime. This will allow us to update client binaries for all Mist users without having to publish a new Mist release, and it will also reduce the size of Mist/wallet releases. And we will no longer track binaries in the repo (which we should never have done in the first place!). This makes use of ethereum-client-binaries to facilitate this.

1. It first checks for ethereumBinaries.json online in the master repo of Mist project
2. If not found or invalid it will load the one bundled with Mist
3. It will then call into ethereum-client-binaries lib with the JSON config and scan for binaries (only Geth supported at the mo)
4. If not matching version of a binary found it will download one and verify again
5. It will then start the app as normal

[hiddentao]

Ready for review. Eth is still bundled as normal whilst I work out how best to fetch it on the fly. Gulp script simplified.

[luclu]

Works flawlessly (tested on mac and linux) :)

Could you also cleanup your shrinkwrap by running npm install; npm prune; npm shrinkwrap?
Without the prune there will be dev-deps included that won't be available under linux (#1208).
This will also fix the travis build.

[hiddentao]

Thanks @luclu , shrinkwrap updated. Also removed some redundant deps.

[luclu]

hmm, seems it didn't work as expected, could you repeat npm install; npm prune; npm shrinkwrap while doing rm -r node_modules before?

[hiddentao]

Tried again, after removing node_modules

[luclu]

Weird, I pushed my npm install; npm prune; npm shrinkwrap's result.

[luclu]

Ok, made travis happy again while on it.

[hiddentao]

Thanks @luclu

With the frequent new releases of Geth that have been happening recently, this PR would really help in rolling out Geth updates for Mist users.

[hiddentao]

Annoyingly am getting build errors again. I've done a fresh npm shrinkwrap but to no avail. @luclu I think I need your help again.

[frozeman]

One problem with your config is that it downloads all files from peters cross builds. We should download the ones on the github release page from there and only take from peter what we don't get there.

@hiddentao if the config file contains the links to the versions to download, how does the client updates the geth nodes automatically, or checks for new versions? Also how do we restrict updates to only patch versions, as minor amen major versions could contain backwards incompatible changes.

EDIT i see we download the config from github master. Can we also include the links from the github release page and can you sth-client package download and unzip them as well?

[hiddentao]

@frozeman The download links are arbitrary and can be set to whatever. As long as they link to Zips and Tarballs (i.e. no installers) it's fine.

As you saw, it first checks for a config on the master branch. Thus, to update Geth we have to update the master branch config and then tell users to restart their Mist client. Nothing is automatic.

[hiddentao]

Parity binaries available here: https://vanity-service.ethcore.io/parity-binaries

[luclu]

@hiddentao no problem, I'm at your disposal ;)

Annoyingly am getting build errors again. I've done a fresh npm shrinkwrap but to no avail. @luclu I think I need your help again.

[hiddentao]

Thanks @luclu !

[evertonfraga]

Testing checklist

• Start mist without geth nodes on folder
• Empty nodes with newer geth on remote json
• Start mist with geth nodes on folder
• Start mist with JSON failing to load
• Old node; Newer geth on remote json
  • failed to link to newer geth binary

Silly tests

• Test with version check (sanity) failing on purpose
  • it does fails to install/update the node.

[evertonfraga]

@hiddentao I've raised https://github.com/hiddentao/ethereum-client-binaries/pull/2 in order to make node updates more reliable.

I'll keep testing.

[hiddentao]

@evertonfraga Nice one. Only a minor change needed to that PR and it can be merged.

[evertonfraga]

Looks really good to me :)
@hiddentao after integrating that PR, we're good to go

[hiddentao]

Why can't we fetch binaries from the Github servers? e.g. links on https://github.com/ethereum/go-ethereum/releases/tag/v1.4.17

[hiddentao]

Fixed CI, ready to merge.

[lock]

This thread has been automatically locked because it has not had recent activity. Please open a new issue for related bugs and link to relevant comments in this thread.