Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding solc binaries to solc-bin #9226

Closed
alcuadrado opened this issue Jun 17, 2020 · 16 comments
Closed

Adding solc binaries to solc-bin #9226

alcuadrado opened this issue Jun 17, 2020 · 16 comments
Assignees
Labels

Comments

@alcuadrado
Copy link
Member

I'm creating this issue to discuss/track adding Windows, Linux, and MacOS binaries of solc to solc-bin.

We've already talked about this in multiple places, but there was no issue to keep track of its progress. The closes thing was #3168, but that's ready so that issue is no longer relevant.

Abstract & Motivation

The reasons to do this are:

  1. It can be used by JS tools like truffle and buidler to improve the compilation speed.

  2. Tools that are not written in JS need this to simplified their required setup. Now they only work with docker or a github API key. @iamdefinitelyahuman can provide more info about this if needed.

Specification

Just like with asm.js and wasm, binary versions for each of the major OS should be added.

Backwards Compatibility

This doesn't introduce any change to the language.

To make the effort effective all the versions in solc-bin should be compiled and published, except maybe nightlies.

@chriseth
Copy link
Contributor

It turns out that in the meantime, we hit the size limit of github-pages, so we need to find a new solution on where to put released binaries anyway. It is a good idea to combine different architectures.

Maybe to clarify this a bit more: This repository would contain statically-linked binaries that are as portable as possible. Distributing other binaries should be done through the respective package manager of the OS.

@iamdefinitelyahuman
Copy link
Contributor

Very excited about this!

With py-solc-x I'm pulling the binaries from Github, and can confirm that it creates a headache when used in the CI. Without an access token it almost always fails from rate limiting. Having binaries served elsewhere will be a huge help to me.

Re OSX: I wrote a wiki article to field the repeated questions, but I think more often than not I end up losing potential new users who hit the issue. Especially when working on the mainnet Brownie installs many solc versions to locally compile source codes from Etherscan. The delay in compiling solc, combined with uncertainty that it won't fail at 90%, is just too much.

@cameel
Copy link
Member

cameel commented Jul 17, 2020

A little update on the progress here:

  • Linux, MacOS and Windows binaries are now available in solc-bin repo. You can see the matrix of available versions in my comment in #42
    • We currently only have MacOS builds starting from v0.6.9 on but we're planning to build older versions and add them to solc-bin too.
  • Due to size constraints of GH pages the new binaries are not accessible via solc-bin.ethereum.org yet. They can't be included until we start hosting them elsewhere and switch over the domain. We're in the process of setting that up.
    • By the way, please don't use the solc-bin.github.io domain directly. We don't control that domain so we can't transparently switch that over. We'll keep the soljson.js files available there not to break tools that have it hard-coded but we won't be able to make binaries for other platforms available that way.
  • We set up a new directory structure for the binaries, segregated by platform. The old soljson.js binaries are still available in bin/ and wasm/ but they're also symlinked under new names. Please use the new ones whenever possible.
  • We'd like to make the content of solc-bin available via IPFS as well, though not necessarily as a part of this task.

@alcuadrado
Copy link
Member Author

This is great news, @cameel! Thanks for the update.

@cameel
Copy link
Member

cameel commented Jul 23, 2020

If anyone wants to try the new setup, binaries are already available via https://solc-bin.s3.eu-central-1.amazonaws.com with no rate limiting. The paths are exactly as in solc-bin repo, e.g. https://solc-bin.s3.eu-central-1.amazonaws.com/macosx-amd64/list.json.

I have just finished setting up mirroring of solc-bin to this S3 bucket and it contains all the releases (including the latest one). My mirroring script still has to go through review and the official domain has not been switched over to point at S3 so please don't treat it as production-ready yet but I expect the transition to be smooth.

Now I need to deal with some smaller left-over tasks related to solc-bin and solc-js and when I'm done, the Mac OS X builds are next on my todo list.

@cameel
Copy link
Member

cameel commented Jul 30, 2020

The mirroring is now up and running. I have also just asked the devops team to switch over the domain (https://github.com/ethereum/devops/issues/499) so you can expect solc-bin.ethereum.org to start pointing at the S3 bucket very soon.

@cameel
Copy link
Member

cameel commented Aug 5, 2020

I got info from the devops team that the domain has been switched over today in the morning.
Changes are not visible in DNS yet but once the cached old DNS entries expire, https://solc-bin.ethereum.org should start serving from S3.
Please let me know if you notice any breakage.

@cameel
Copy link
Member

cameel commented Aug 8, 2020

We had some unexpected problems with the switchover (DNS was updated but Cloudfront kept serving the data from GH pages) but it's solved and the new binaries are finally available at solc-bin.ethereum.org.

@cameel
Copy link
Member

cameel commented Sep 28, 2020

Good news. I managed to rebuild all solc versions down to 0.3.6 on MacOS. For now they're available in my fork of solc-bin: https://github.com/cameel/solc-bin/tree/macosx-static-binaries/macosx-amd64 (ethereum/solc-bin#54). Please try them out and let me know if you notice any issues (especially with the older ones).

The script for building them is currently being reviewed (ethereum/solc-bin#53).

@cameel
Copy link
Member

cameel commented Sep 28, 2020

Also, another piece of news related to the binaries:

We recently updated our CI to produce completely static Windows builds (#9811) so packing them in a .zip is no longer necessary. In one of the upcoming releases we're going to start putting uncompressed .exe files in solc-bin.

@alcuadrado
Copy link
Member Author

Awesome news, @cameel ! Thanks

@jcaracciolo
Copy link
Contributor

Great news! Will definitely improve and accelerate tooling development

@cameel
Copy link
Member

cameel commented Sep 29, 2020

The PR has been merged. MacOS binaries are now available at https://solc-bin.ethereum.org

@alcuadrado
Copy link
Member Author

Thanks @cameel! We are already integrating this into Buidler 🥳

@fzeoli
Copy link

fzeoli commented Sep 29, 2020 via email

@cameel
Copy link
Member

cameel commented Sep 30, 2020

I think we're done here so I'm closing the issue. There are a few minor things related to the release process covered by #9258 (see #9258 (comment)) but I think that the core issue is solved.

@cameel cameel closed this as completed Sep 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants