jenkins: add llnode pipeline and shell scripts

[gibfahn]

The llnode CI isn't working right now, but hopefully by committing this
we can then iterate in GitHub.

This also makes the node download script runnable manually, and adds docs for it.

@joyeecheung I was thinking if we land this, you can raise a PR to build, and build the jobs directly from your PR. Thoughts?

[gibfahn]

Can the parameters for llnode-pipeline also be added into the pipeline definition?

Good point, added. Also added a check, as the parameters don't always show up until the second time you run the pipeline.

[mmarchini]

Ping @gibfahn

[mmarchini]

@gibfahn would you mind if I move forward with this? Looks like Travis on llnode just broke again (our 2 last builds are stuck), so it would be nice to move our tests to Jenkins.

[mmarchini]

Commit updating Node.js versions: mmarchini@b918f24
(I don't have permission to push to this branch)

I tested with GIT_REPO="nodejs/llnode" GIT_BRANCH="master" NODE_VERSION="v8" sh jenkins/scripts/llnode-continuous-integration.sh and it's working. @nodejs/build can we move forward with this and improve in further iterations?

[Trott]

@nodejs/build can we move forward with this and improve in further iterations?

Works for me, but it would be good to have some input from more Jenkins-y folks like @joaocgreis @maclover7 @rvagg @refack ....

[maclover7]

There are a few outstanding review comments that @gibfahn should look at (I, at least personally, am not super familiar with this PR and how it will work, and don't really feel comfortable trying to address those) -- if we don't hear back for a little while longer, we can take another look

[refack]

@mmarchini I applied your tweak to the pipeline spec. Since the job runs using the code from @gibfahn's fork it should now be updated.

[mmarchini]

@refack I just uploaded a new patch, can you cherry-pick it into gibfahn:llnode-ci?
mmarchini@417fa90

This patch will fix the test runs for Linux and Mac OS by downloading LLDB from http://releases.llvm.org/download.html. It assumes we're only testing on Ubuntu 16.04 and Mac OS for now, which I think is fine for now, since those are the tests we currently have on llnode with Travis.

I also added nightly and canary as possible parameters for Node.js version. Is it possible to discard the result of those versions on the result of llnode-pipeline, so we can keep the same behavior from our current Travis setup? For reference:

• https://github.com/nodejs/llnode/blob/master/.travis.yml
• https://travis-ci.org/nodejs/llnode/builds/379496130

[refack]

More shaving got us to:

https://ci.nodejs.org/job/llnode-continuous-integration/37/LLDB=3.8,MACHINE=ubuntu1604-64/console

seems like lldb is not part of the tarball

root@test-rackspace-ubuntu1604-x64-2:/home/iojs/build/workspace/llnode-continuous-integration/LLDB/4.0/MACHINE/ubuntu1604-64/llvm-bin# ls bin
bugpoint                  clang-change-namespace  clang-func-mapping     clang-rename          ld.lld    llvm-as          llvm-c-test   llvm-dlltool    llvm-link        llvm-mt          llvm-profdata  llvm-size        llvm-xray   scan-build
c-index-test              clang-check             clang-import-test      clang-reorder-fields  llc       llvm-bcanalyzer  llvm-cvtres   llvm-dsymutil   llvm-lto         llvm-nm          llvm-ranlib    llvm-split       modularize  scan-view
clang                     clang-cl                clang-include-fixer    clang-tidy            lld       llvm-cat         llvm-cxxdump  llvm-dwarfdump  llvm-lto2        llvm-objcopy     llvm-rc        llvm-stress      obj2yaml    verify-uselistorder
clang++                   clang-cpp               clang-offload-bundler  find-all-symbols      lld-link  llvm-cfi-verify  llvm-cxxfilt  llvm-dwp        llvm-mc          llvm-objdump     llvm-readelf   llvm-strings     opt         wasm-ld
clang-6.0                 clangd                  clang-query            git-clang-format      lli       llvm-config      llvm-diff     llvm-extract    llvm-mcmarkup    llvm-opt-report  llvm-readobj   llvm-symbolizer  sancov      yaml2obj
clang-apply-replacements  clang-format            clang-refactor         ld64.lld              llvm-ar   llvm-cov         llvm-dis      llvm-lib        llvm-modextract  llvm-pdbutil     llvm-rtdyld    llvm-tblgen      sanstats

[mmarchini]

Interesting. I tested only with 3.9.0, and lldb was there. Let me see why it's missing from 6.0.0

[mmarchini]

Btw, does this build matrix still make sense now that LLDB_VERSIONS is a parameter?

[mmarchini]

I tested only with 3.9.0, and lldb was there. Let me see why it's missing from 6.0.0

I actually forgot the global LLDB in my path while testing, my bad. clang+llvm doesn't come with LLDB, and I'm having trouble finding a source to download it from.

[refack]

Btw, does this build matrix still make sense now that LLDB_VERSIONS is a parameter?

Killed the matrix.

[joyeecheung]

I have not yet find a way to get reliable prebuilt lldb binaries either.

Here is a gist that I wrote some time ago when I needed to test llnode on different linux distros:
https://gist.github.com/joyeecheung/f458f9a668e0e9546d48d899298e7331

[mmarchini]

Is Xcode installed in our OSX machines? If so we can use the LLDB that comes with it, and use Docker to test on Linux. What do you think?

[refack]

We should have "Xcode Command Line Tools" on the macOS machines.

[mmarchini]

New patch, tested on Ubuntu 16.04 (this time without having a global lldb installed):
mmarchini@acb541d

I went with the "download the .deb and extract it" approach.

OSX will still fail, but let's fix one OS at a time :)

[refack]

Shaved it some more (curl needed --compressed and a missing pipe)

Reading lldb version...
/home/iojs/build/workspace/llnode-continuous-integration/default/lldb-bin/usr/bin/lldb: error while loading shared libraries: libLLVM-6.0.so.1: cannot open shared object file: No such file or directory
{ Error: Command failed: /home/iojs/build/workspace/llnode-continuous-integration/default/lldb-bin/usr/bin/lldb -v
/home/iojs/build/workspace/llnode-continuous-integration/default/lldb-bin/usr/bin/lldb: error while loading shared libraries: libLLVM-6.0.so.1: cannot open shared object file: No such file or directory

I saw this while trying to to the deb dance, there's a long chain of dependencies needed to NVM was just missing 1 more; libllvm$LLDB_VERSION.*$arch.deb and... success.

https://ci.nodejs.org/job/llnode-continuous-integration/47/default/

[mmarchini]

@refack is there anything left to be done here? If so, can I help?

[refack]

@mmarchini you should now have write permissions to the underlying branch - ttps://github.com/gibfahn/build/tree/llnode-ci