Allow alternate binaries download urls generation and convert GoDistribution and LLVM subsystems to use it

[cosmicexplorer]

Problem

BinaryTool is a great recent development which makes using binaries downloaded lazily from a specified place much more declarative and much more extensible. However, it's still only able to download from either our S3 hosting, or a mirror.

The previous structure requires the urls provided to the global option --binaries-baseurls to point to an exact mirror of the hierarchy we provide in our S3 hosting, but that can change at any time. It's not incredibly difficult to write a script to mirror our hosting into an internal network, but in general there's no reason the layout of binaries in ~/.cache/pants/bin/ needs to determine where those binaries are downloaded from.

Our bandwidth costs in S3 have recently increased due to the introduction of clang and gcc in #5490. See #5777 and #5779 for further context on S3 hosting. There are reliable binary downloads for some of these tools, which we would be remiss not to use if we can do it in a structured way.

Solution

• Introduce a urls= argument to multiple methods of BinaryUtil for BinaryTools that don't download from our s3.
• Add support for extracting (not creating) .tar.xz archives by adding the xz BinaryTool (see add xz package binaries#66) and integrating it into BinaryTool's archive_type selection mechanism.
• Use the above to download the go and llvm binaries from their official download urls.
  • Also, rename the Clang subsystem to LLVM as the binary download we use now (for ubuntu 16.04, currently) also contains many other LLVM tools, including e.g. lld.

Result

Urls for binary downloads can now be created in a structured way for external downloads, with the --force-baseurls option as an escape hatch. Some binaries now default to external urls provided for public use by the maintainers of the software to download, thanks to the introduction of the xz binary tool. Two out of the three largest bandwidth users among our provided binaries have been switched to use the download urls provided by the maintainers of each project (LLVM and Go). gcc still needs to be fixed, which will happen in a separate PR.

[cosmicexplorer]

CI is failing because Travis doesn't have xz installed -- could I add a package manager command in some setup script to fix this?

  Running setup.py bdist_wheel for pyliblzma ... error
  Complete output from command /travis/workdir/build-support/pants_dev_deps.venv/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lScRhR/pyliblzma/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-14EddE --python-tag cp27:
  Package liblzma was not found in the pkg-config search path.
  Perhaps you should add the directory containing `liblzma.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'liblzma' found
  Package liblzma was not found in the pkg-config search path.
  Perhaps you should add the directory containing `liblzma.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'liblzma' found
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  copying liblzma.py -> build/lib.linux-x86_64-2.7
  running build_ext
  building 'lzma' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/src
  In file included from src/liblzma.c:1:0:
  src/liblzma.h:24:10: fatal error: lzma.h: No such file or directory
   #include <lzma.h>
            ^~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pyliblzma

[benjyw]

So I was initially thinking we'd do this by leveraging --binaries-baseurls, with a more sophisticated interpolation scheme. But I like your way better - basically a BinaryTool can optionally say "there is a canonical place to fetch me from", and fall back to "there is no canonical place, so Pants are hosting me".

However this means that some binary tools use --binaries-baseurls and others don't. It's a bit weird to pass urls around as arguments, in a class that also has its own way of finding urls, that may or may not actually be used.

The underlying design flaw, I think, is that BinaryUtil does two orthogonal things: it computes URLs for requested binaries, and it handles fetching. It might be best to break it up into two separate pieces, along those lines. Then the distinction between "how we get a URL" (which used to only have one answer but now has two) and "how we download" gets clarified.

[benjyw]

The comment references the fact that we put the urls into a set. You've removed that for whatever reason (why?) so the comment is no longer relevant.

[cosmicexplorer]

Wow, that's a good catch. I was thinking that we might want to allow intentional retrying of urls, but realized after your comment that we should be doing that in the Fetcher, if anywhere, so added the OrderedSet() call in b29bca3. Thanks!

[cosmicexplorer]

@benjyw there are 3 or 4 comments describing a few more test cases I should add, but having added docstrings, this is in a reviewable state.

[cosmicexplorer]

btw, splitting up BinaryUtil into fetching and url generation was a fantastic idea, imo. I'm totally fine with iterating on it more if the +/- is too high, but this turned out pretty neat, I think.

[benjyw]

This came out really nicely! Just one comment that I think you can satisfy by renaming some things.

[benjyw]

I don't love this. It's weird to provide a really neat API, and then have this out-of-band way of overriding it, that requires an elaborate comment in the interface, and that will be completely non-obvious to the reader/debugger of this code.

I don't immediately have a better idea. But at the very least, let's name this something more obvious, like "--allow-external-binary-tool-downloads" (with a default of True, of course).

And then you can change the name of url_generator() to get_external_url_generator(), to clarify that it is affected by this option, and that if it returns None then "internal" (i.e., pants-hosted, or org-hosted) urls are used.

[cosmicexplorer]

Yeah, the only reason for this was for migration purposes, which isn't great. I would much prefer / tried adding an option to do this for individual BinaryTools, but then you would just have to override every one of them (if running pants on an internal network). As a pants.ini option, I might not use it, but as a command-line option I can see it being useful. I agree on not liking it and there is probably a better way.

I'll implement the option name change and the method name change you describe.

[cosmicexplorer]

c8bb85d has the changes. I think I got all the references to the old option, but we will see what CI has to say.

[cosmicexplorer]

I did some work on deepening the testing for archive.py and the xz decompressor along with further BinaryTool testing (since the current test cases don't test things that BinaryUtil does test, but we obviously want to move away from relying on BinaryUtil eventually), but it was a little orthogonal to this PR, so I stashed it and will put up another PR when this is merged.

[benjyw]

I'm not sure what this means? In what way would a BinaryTool subclass be affected by the download location, and how would an author mitigate that in code?

[benjyw]

Nit: Also, I think the word is spelled "implementer".

[cosmicexplorer]

Yeah, I was thinking about how to make that more clear here. I was thinking specifically about what is done in the LLVM subsystem by overriding the select() method (and was going to explicitly call out that example here, but stopped short of doing that). Should I add in a comment describing that a bit (basically: we check if there's a single top-level directory and go down a level if so, or just return the result of select() if the extraction doesn't put everything into a single top-level directory inside our specified extraction directory), or take this clause out of the docstring? Let me know if that made sense.

The internet confirms that "implementer" is the correct spelling.

[benjyw]

Now the name and the help text mesh together nicely!

[cosmicexplorer]

Totally agree!

[benjyw]

Great! Thanks for the thoughtful changes.

[benjyw]

Ah, I see. So the issue is that the internal and external artifacts may have different internal structure? How about saying something more along the lines of "Implementations of BinaryTool must be aware of differences (e.g., in archive structure) between the external and internal versions of the downloaded tool, if any."

[cosmicexplorer]

That sounds perfect and I tried changing that paragraph to the exact text of your comment in 55e6fb3 -- the bit about being further overridden by a subclass in particular was highly extraneous, so this makes it a lot clearer. I think this is a clear description of the concern I wanted to state there.