Add support for binary builds with PyInstaller

[crwood]

This PR adds support for building portable, self-contained tahoe binary distributions for all major desktop platforms (GNU/Linux, Mac, and Windows) via PyInstaller, partly resolving trac ticket 2729. This is implementation is similar to what I've used elsewhere (and extensively tested) for the Gridsync project but with a some minor cleanups and additions that I thought might be useful for integrating with Tahoe-LAFS' own buildbot/CI/deployment pipeline (like exporting artifacts in platform-prefered archive-formats, labeling their bitness in the resultant filename, and printing out their SH256 hash digests along the way).

Although probably not strictly necessary, I've decided to implement this using a tox testenv in order to provide stronger isolation during the build process (since I set PYTHONHASHSEED=1 to provide partial reproducibility and patch the setuptools check out of _auto_deps.py at buildtime to prevent a runtime exception -- either of which could lead to issues if you're parallelizing your test run with detox or similar); to make a binary distribution, simply run the following command (with tox installed):

tox -e pyinstaller

Assuming all goes well, when the build process completes, you'll find an appropriately-labeled archive inside of dist/ (e.g., "Tahoe-LAFS-Windows-32bit.zip", "Tahoe-LAFS-MacOS-64bit.tar.gz") containing everything needed to run Tahoe-LAFS on that target platform. The end-user need only extract the archive somewhere on their system and run the tahoe executable contained therein (tahoe.exe on Windows); no python2 installation is required to be present on the host system.

Please note, however, that the various caveats listed in trac ticket 2729 still apply. In particular, PyInstaller's binaries are not guaranteed to be backwards-compatible and, in some cases, are certainly not (as one example, binaries built with Travis-CI's osx images -- which run Mac OS X 10.10 -- will not run on the old iMac in my office -- which runs OS X 10.9 -- however, binaries built on my iMac will run without issue on Travis and other machines that I've tested running 10.10). Accordingly, it is recommended to distribute artifacts that were built on older operating systems so as to increase the likelihood of them running on as many systems as possible.

On that note, once this PR lands -- or even if it doesn't -- I would be happy to donate some Mac and Windows VMs/buildslaves to tahoe-lafs.org's buildbot fleet (which I'm already running for Gridsync anyway) and/or help streamline deployments through Travis-CI/AppVeyor (the latter of which, I think, we already depend on for building wheels?).

[warner]

looks like the appveyor test failed because of a transient test_crawler timing problem.. I've kicked off a new build, hopefully it'll pass this time.

[codecov-io]

Codecov Report

Merging #421 into master will decrease coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #421      +/-   ##
==========================================
- Coverage   88.02%   87.96%   -0.07%     
==========================================
  Files         147      147              
  Lines       27833    27833              
  Branches     3977     3977              
==========================================
- Hits        24500    24483      -17     
- Misses       2621     2632      +11     
- Partials      712      718       +6

Impacted Files	Coverage Δ
src/allmydata/web/status.py	82.5% <0%> (-0.89%)	⬇️
src/allmydata/stats.py	85.34% <0%> (-0.87%)	⬇️
src/allmydata/util/dictutil.py	97.19% <0%> (-0.71%)	⬇️
src/allmydata/mutable/servermap.py	93.55% <0%> (-0.65%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11ae1dc...5a9cf3e. Read the comment docs.

[meejah]

Weird, what's up with the coverage?

[warner]

We've got some intermittent coverage problems in the unit tests. I'm listing them in https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2891#comment:5 , so we can slowly track them down and stomp them.

This looks great. I've rebased it to current master and will land momentarily.

[crwood]

I was just going to say: it may actually be a good idea to hold off on this temporarily (or at least not incorporate the build step into your buildbots/travis/appveyor configs yet), as I just discovered this morning that, for some package-versioning/autodeps-related reason, the introduction of the newly-added magic-wormhole dependency via PR#418 is causing the resultant binary builds to fail at runtime. See, e.g., this log (which occurs on all three operating systems): https://buildbot.gridsync.io/builders/frozen_tahoe_linux/builds/34/steps/run/logs/stdio

I'm not yet sure whether something else needs to be added to src/allmydata/_auto_deps.py to properly resolve the issue (or whether it's caused, perhaps, as a result of magic-wormhole's usage of versioneer to dynamically generate the version strings) but I've been able to work around the issue on my end by simply patching out the "magic-wormhole" line in _auto_deps.py at build time (which isn't a very elegant solution, obviously).. That said, any advice on how to best address the issue would be appreciated -- the workings of _auto_deps.py the various related hooks in __init__.py and elsewhere are rather mysterious to me..