Make official ChakraCore binaries easily available to developers on *nix platforms. (e.g. deb and brew feeds)

[dilijev]

As discussed in #85 (especially #85 (comment)), we should have developer feeds for Linux and OS X for the binaries we release for those platforms, since NuGet is probably not the primary choice for development for Native apps on those platforms.

Since the Linux flavor we primary support is Linux, we should start with an apt-get feed for .deb packages.

The goal is to enable developers to do e.g. apt-get install libchakracore and begin development with the ability to link against our official binaries.

For OS X, the obvious choice is Brew.

/cc @obastemur

Work Items:

• Signing linux binaries
• Signing OS X binaries
• Script for installation of official packages which works for all *nix platforms

Extra work items (if deemed valuable):

• apt-get feed
• Brew feed
  • Send a PR to update https://github.com/Homebrew/homebrew-core/blob/master/Formula/chakra.rb

[obastemur]

@dilijev we were considering another approach.

ChakraCore runs on many *nix distros without a problem and we would like to cover a wider range with our binary distro without dealing with all the distribution channels one by one.

We may end up with a bash script that detects the environment and installs the appropriate ChakraCore binary.

i.e. $ curl https://aka.ms/abcd | bash (edit: <- this is just a sample line, DO NOT USE IT!)

No matter what *nix platform you are on, the command above should install the latest ChakraCore. We may even provide additional options here to select among the versions, ask some questions etc..

[dilijev]

@obastemur sounds great! However, maybe there's still an opportunity to aid ease of entry for developers on some platforms by integrating with the existing package services?

[dilijev]

Updated title to more generically capture the goal here.

[obastemur]

However, maybe there's still an opportunity to aid ease of entry for developers on some platforms by integrating with the existing package services?

I don't know how that would be easier than the option I have mentioned above. Could you please clarify?

[cosinusoidally]

How about distributing a single binary that works across every supported Linux distribution (for a given architecture)? Much the same as how node.js is distributed. The Linux versions of node.js are distributed as gpg signed tarballs (one per architecture). Their binaries work on a wide range of Linux distributions as they build their binaries on a CentOS 6 system (therefore avoiding depending on newer versions of glibc, which is the cause of a large part of Linux's binary compatibility problems).

[obastemur]

How about distributing a single binary that works across every supported Linux distribution (for a given architecture)? Much the same as how node.js is distributed.

@cosinusoidally that's the plan though and we already have ChakraCore single binary is working on all the recent distros. (once it combined with --embed-icu --static options)

IIRC libc libstdc++ adds aprox 1mb to file size. So, we are not actually worried on this one.

So.. The one-liner I was mentioning above is going to detect unix/posix and target-arch and then download the tarball / print sha / install.. Perhaps it's a good idea to share the tarball links for people who doesn't want to use our automated installer.

edit: fixed a typo

[cosinusoidally]

How about distributing a single binary that works across every supported Linux distribution (for a given architecture)? Much the same as how node.js is distributed.

@cosinusoidally that's the plan though and we already have ChakraCore single binary is working on all the recent distros. (once it combined with --embed-icu --static options)

Just to clarify, are you planning on distributing ChakraCore as a shared library (say libChakraCore.so) plus some headers?

IIRC libc adds aprox 1mb to file size. So, we are not actually worried on this one.

I assume you mean libstdc++ ? I can see that the build.sh script defaults to statically linking libstdc++. Or are you planning to statically link libc (which is glibc on the majority of Linux distros)? I'm finding it difficult to find good info on this but apparently it is a bad idea to statically link glibc (eg, this blog post warns against: http://insanecoding.blogspot.co.uk/2012/07/creating-portable-linux-binaries.html).

[obastemur]

Just to clarify, are you planning on distributing ChakraCore as a shared library (say libChakraCore.so) plus some headers?

@cosinusoidally Yes, that's the plan. BTW; fixed the typo above.

[chicoxyzzy]

Would be great if ChakraCore will be available with Node ChakraCore via nvm. Related nvm issue nvm-sh/nvm#1067

[dilijev]

@obastemur It's definitely the case that a general script that works for all platforms is the most value and lowest cost to maintain for the value. It may be the case that platform-specific feeds which require setup in any case would be any additional value. Your discretion :)

[mathiasbynens]

I ran into an issue while putting together a Homebrew formula for ChakraCore: #2417 Any help there would bring us closer to a working formula.

[mathiasbynens]

ChakraCore can now be installed with Homebrew! brew install chakra {--with-icu4c}

Please consider adding the following item to the ChakraCore release checklist:

1. send a PR to update https://github.com/Homebrew/homebrew-core/blob/master/Formula/chakra.rb

[mathiasbynens]

Also, consider hosting pre-built binaries for each OS/architecture somewhere official. This would make it easier for developers to test different ChakraCore versions: instead of having to build them all from source, they could just wget and extract a .tar.gz file. (For example, the Homebrew formula could then be updated to just download + extract instead of building from source.)

You could even attach the official binaries as downloads to the GitHub release, e.g. https://github.com/Microsoft/ChakraCore/releases/tag/v1.4.0#downloads.

[dilijev]

@mathiasbynens: @obastemur is working on hosting the xplat binaries at an official location (for example, try curl -SL https://aka.ms/chakracore/install | bash on Linux).

send a PR to update https://github.com/Homebrew/homebrew-core/blob/master/Formula/chakra.rb

Added to the list in the description of this issue.

[dilijev]

@obastemur have we determined whether signing on Linux/OSX is possible/realistic?

[mathiasbynens]

@obastemur is working on hosting the xplat binaries at an official location (for example, try curl -SL https://aka.ms/chakracore/install | bash on Linux).

🎉 The URL format of https://aka.ms/chakracore/${BINARY_NAME}_${CHAKRA_VERSION}.tar.gz looks great.

[obastemur]

@mathiasbynens @ilovzfs for 2.0 release, can we use https://aka.ms/chakracore/${BINARY_NAME}_${CHAKRA_VERSION}.tar.gz instead of compiling from source?

[mathiasbynens]

@obastemur For homebrew-core, building from source is a requirement.

I would definitely use the precompiled binaries for https://github.com/mathiasbynens/homebrew-ecmascript, though. WIP PR: mathiasbynens/homebrew-ecmascript#5

[cosinusoidally]

Any plans to support older Linux distros? I noticed that the x64 binary https://aka.ms/chakracore/cc_linux_x64_2_0_0.tar.gz requires GLIBC 2.17. This means that the ChakraCore binary won't run on older distros (such as CentOS 6, Debian Wheezy, etc)

(precise)ljw@localhost:/tmp/ChakraCoreFiles/bin$ ./ch 
./ch: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found (required by ./ch)

The ch binary (and also libChakraCore.so) requires one symbol from GLIBC 2.14 and one from GLIBC 2.17:

(precise)ljw@localhost:/tmp/ChakraCoreFiles/bin$ objdump -T ./ch |grep GLIBC_2\\.[0-9][0-9]
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.14  memcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.17  clock_gettime

To support CentOS 6 you would need to target GLIBC 2.12.

You could probably get your current build system to generate CentOS 6 compatible binaries by putting something like this:

__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver clock_gettime,clock_gettime@GLIBC_2.2.5");

in a header file somewhere (plus, obviously, you'd also need to test on CentOS 6). Failing that you could build on CentOS 6.

[obastemur]

@cosinusoidally First of all thank you for sharing all those details. I saw something similar on Wheezy today.

We have to target the particular version of some of the binaries but glibc is luckily not one of them.
I did already setup a machine to test binary re-targeted. Once it succeeds, I will update the RC binary on the link.

[dilijev]

@obastemur is it possible to have an RC link and have an RC option for the script to download an RC rather than latest stable, so that the link is not presumed to be a final release?

[obastemur]

@dilijev shall we close this?

[dilijev]

@obastemur Happy to :)

[cosinusoidally]

As mentioned in an earlier comment the Linux binaries do not work on CentOS 6. Any plans to provide binaries that will work with Linux systems all the way back to CentOS 6? I mention CentOS 6 because if you can get your binaries working there then they should work on pretty much every single currently supported x86_64 version of Linux.

I've managed to hack together a workaround for my purposes by patching the CharkraCore release binaries (see https://github.com/cosinusoidally/ChakraCore-experiments/tree/master/centos6). This works, but there are a couple of caveats to that approach. In an ideal world the release binaries would just work unmodified on every Linux distro.

[dilijev]

@cosinusoidally No specific plans at the moment, but we wouldn't be opposed as long as we don't end up breaking support for currently-supported platforms. Feel free to open a new issue so we can have an open issue to track it. (You can link back to this conversation.)

[obastemur]

Aren't you able to build ChakraCore on Centos 6? If you are having trouble to find latest Clang, under 'tools' folder, we have a custom build script

[cosinusoidally]

@obastemur see #3494 . I've managed to build ChakraCore on CentOS 6 . I did have a look at your Clang build script, but I could see that there were some missing build options that would have yielded a slightly misconfigured copy of Clang if I had attempted to use it on CentOS 6 (since I am not using the standard system copy of GCC I needed to set GCC_INSTALL_PREFIX). For security purposes I also wanted to verify the signatures/checksums of any tarballs I was fetching. I also wanted to use Clang 3.8.0 to match what you run on your Jenkins bots. Also, in order to save build time I only wanted to build the X86 backend. In the end I just wrote my own script.

[obastemur]

I'm not sure I understood why you did all of that? Probably Clang 4.0 / 4.1 with Gold(LTO) has failed to build there?

If you can send a PR on tarball hash etc checking, it could be really nice. FYI; we tend to fetch the latest here!

What else was needed? Appreciate the PRs to current build system.

Thanks!

[cosinusoidally]

You're welcome.

It was just a matter of picking an approach which I thought would give me the best chance of success. My approach also generates the whole build environment (it sets up a CentOS 6 chroot, installs all the development requirements, and builds the required versions of cmake and clang). I used proot rather than chroot so there was no requirement to do anything as root. This also means that the generated root filesystem could be used in, say, a CI environment where a user may not have root access. Using proot does slow down the build process to some extent, but a standard chroot could be used if the performance hit is too much (I haven't measured it, but in my experience it is tolerable).

Building Clang is pretty time consuming, so I didn't want to end up doing it multiple times. Clang 4 and LTO with Gold would have been nice, but I wasn't sure if they would build so I didn't try. I also suspected LTO would have used too much RAM (the machine I was using has 2GiB). I had a look at one of your Jenkins Linux build jobs and noted that you were using Clang 3.8.0 (eg see https://ci.dot.net/job/Microsoft_ChakraCore/job/master/job/shared_ubuntu_linux_release/533/console). Since this was a known working compiler I figured using Clang 3.8.0 would give me the best chance of success. Tantalisingly there is actually a "Clang for CentOS 6 x86_64" tarball on http://releases.llvm.org/download.html#3.8.0 . Unfortunately that official release of Clang for CentOS 6 doesn't actually work (somehow they have managed to compile it against a newer GLIBC and they have also dynamically linked to a newer libstdc++, so it doesn't work on a stock CentOS 6). Since the official binary wouldn't work I built my own.

Regarding https://github.com/Microsoft/ChakraCore/blob/master/tools/compile_clang.sh I can try and put together a PR next week. Something like the following:

diff --git a/tools/compile_clang.sh b/tools/compile_clang.sh
index dcb4477..ad5f8f1 100755
--- a/tools/compile_clang.sh
+++ b/tools/compile_clang.sh
@@ -58,6 +58,8 @@ ROOT=${PWD}/cc-toolchain/
 GOLD_PLUGIN=""
 
 if [ ! -d ./cc-toolchain/src/llvm/projects/compiler-rt ]; then
+# get "Hans Wennborg <hans@chromium.org>" gpg key (he signed the Clang 4.0.0 release)
+    gpg --recv-key  B6C8F98282B944E3B0D5C2530FC3042E345AD05D
     rm -rf cc-toolchain
     mkdir cc-toolchain
     cd cc-toolchain
@@ -106,6 +108,14 @@ if [ ! -d ./cc-toolchain/src/llvm/projects/compiler-rt ]; then
 
     echo "Downloading LLVM ${LLVM_VERSION}"
     wget --quiet "http://llvm.org/releases/${LLVM_VERSION}/llvm-${LLVM_VERSION}.src.tar.xz" >/dev/null 2>&1
+    wget --quiet "http://llvm.org/releases/${LLVM_VERSION}/llvm-${LLVM_VERSION}.src.tar.xz.sig" >/dev/null 2>&1
+    gpg --verify "llvm-${LLVM_VERSION}.src.tar.xz.sig"
+    if [ $? == 0 ]; then
+        echo "llvm-${LLVM_VERSION}.src.tar.xz.sig is ok"
+    else
+        echo "Error checking signature: llvm-${LLVM_VERSION}.src.tar.xz.sig"
+        exit 1
+    fi
     tar -xf "llvm-${LLVM_VERSION}.src.tar.xz"
     if [ $? == 0 ]; then
         rm "llvm-${LLVM_VERSION}.src.tar.xz"
@@ -117,6 +127,14 @@ if [ ! -d ./cc-toolchain/src/llvm/projects/compiler-rt ]; then
     cd llvm/tools/
     echo "Downloading Clang ${LLVM_VERSION}"
     wget --quiet "http://llvm.org/releases/${LLVM_VERSION}/cfe-${LLVM_VERSION}.src.tar.xz" >/dev/null 2>&1
+    wget --quiet "http://llvm.org/releases/${LLVM_VERSION}/cfe-${LLVM_VERSION}.src.tar.xz.sig" >/dev/null 2>&1
+    gpg --verify "cfe-${LLVM_VERSION}.src.tar.xz.sig"
+    if [ $? == 0 ]; then
+        echo "cfe-${LLVM_VERSION}.src.tar.xz.sig is ok"
+    else
+        echo "Error checking signature: cfe-${LLVM_VERSION}.src.tar.xz.sig"
+        exit 1
+    fi
     tar -xf "cfe-${LLVM_VERSION}.src.tar.xz"
     if [ $? == 0 ]; then
         mv "cfe-${LLVM_VERSION}.src" clang
@@ -129,6 +147,14 @@ if [ ! -d ./cc-toolchain/src/llvm/projects/compiler-rt ]; then
     cd ../projects/
     echo "Downloading Compiler-RT ${LLVM_VERSION}"
     wget --quiet "http://llvm.org/releases/${LLVM_VERSION}/compiler-rt-${LLVM_VERSION}.src.tar.xz" >/dev/null 2>&1
+    wget --quiet "http://llvm.org/releases/${LLVM_VERSION}/compiler-rt-${LLVM_VERSION}.src.tar.xz.sig" >/dev/null 2>&1
+    gpg --verify "compiler-rt-${LLVM_VERSION}.src.tar.xz.sig"
+    if [ $? == 0 ]; then
+        echo "compiler-rt-${LLVM_VERSION}.src.tar.xz.sig is ok"
+    else
+        echo "Error checking signature: compiler-rt-${LLVM_VERSION}.src.tar.xz.sig"
+        exit 1
+    fi
     tar -xf "compiler-rt-${LLVM_VERSION}.src.tar.xz"
     if [ $? == 0 ]; then
         mv "compiler-rt-${LLVM_VERSION}.src" compiler-rt

should verify the signatures of the Clang source tarballs. The signature of Binutils would also need to be checked in a similar way. Is there a reason you get it from git rather than using a release tarball?