Packaging into binaries

[AdamISZ]

I've been looking into how to build stand-alone binaries. I'm keeping track in this issue.

Good news from today's efforts: I was able to build a yield-generator binary that runs on Ubuntu 14.10 and 12.04. File size is about 12MB. Been running in the test pit OK.

What was needed:

1. Install pyinstaller. Version 3.0 had problems with ctypes. After some googling I saw issues raised about this recently on the pyinstaller repo, and it seems they fixed these issues. Running off the current development branch commit, this issue was removed. I also needed to direct it to the cffi module. The command I used to successfully build the binary was:

python /path/to/my/pyinstaller/repo/pyinstaller.py -F -p lib --hidden-import=cffi yield-generator.py

of course, running from within my joinmarket directory.

The 12MB output binary seems to run fine e.g. ./yieldgenerator testseed.

I've just tried to build sendpayment the same way, seems to work (unsurprisingly). Clearly this is the actually important case (and tumbler.py).

Testing will have to be a lot more thorough, but already I noticed a problem on Debian:

./yield-generator: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.14' not found (required by /tmp/_MEI7dTLDG/libz.so.1)

Will investigate; some incompatibility with library versions. Once I can run sendpayment.py on Ubuntu and Debian I will try to do the same thing on Windows. Doubtless there will be some difficulties! But it's the more important case.

[AdamISZ]

Update:
sendpayment built OK using the exact same command as above, also 12MB. Just did a successful tx on testnet, console output is normal, argument handling also normal. If everyone in the world used Ubuntu I could stop now :)

[AdamISZ]

Built for debian 32 bit, working on TAILS. For this, the joinmarket.cfg should have:

socks5=false
host=<irc onion>

The command should be something like torify ./yield-generator seed/wallet. For some reason the logs/ subdirectory needs to be created in advance or you get a file access error.

Tested OK on cyberguerrilla testpit (don't forget /mode username -R !!)

From testing we seem to need independent 32 and 64 bit builds, not surprising probably. More surprising was that Ubuntu build didn't work on Debian; there was an incompatibility with the libc shared libraries.

Once those are built (won't take long), will start trying to make a Windows build.

[AdamISZ]

So yield-generator was mainly for testing, I forgot to mention from now I'm focusing on sendpayment, tumbler and wallet-tool

[AdamISZ]

A few notes on building these binaries in case I forget:

It's essential to to follow the instructions here as produced by @fivepiece to set up the environment correctly (this is for the new secp256k1 branch, see #291 and #303). Including and especially the env settings into .bashrc. After this, you can run pyinstaller (remember to take dev branch, see commit above) with the command listed above against a .py script and it should run OK.

Extra notes for TAILS are: you have to run apt-get update before running apt-get install as listed in that INSTALL_UBUNTU.md. Then the pre-build should work fine. Remember to disable socks5 and set the onion host, then the binary should run OK. Also remember to mkdir logs before running, as mentioned above, not yet sure why.

[AdamISZ]

Linux binaries at https://github.com/AdamISZ/TJMBinary.git should run without any preparation on a clean machine, the only thing you should need to do is add a joinmarket.cfg (if you don't want the default) and add a logs/ directory in the same place as the binary.

(Choose debian 32 for Tails. I didn't have Ubuntu 32bit so only built 64 so far)

Reports as to whether this works would be greatly appreciated.

[tailsjoin]

Seems to need a wallet directory created as well:

$ ./wallet-tool_debian_64 generate
Write down this wallet recovery seed

blah blah seed...

Enter wallet encryption passphrase: 
Reenter wallet encryption passphrase: 
Input wallet file name (default: wallet.json): test-one
Traceback (most recent call last):
  File "<string>", line 132, in <module>
IOError: [Errno 2] No such file or directory: 'wallets/test-one'
wallet-tool returned -1

Why not just add empty dirs for logs and wallets to the repo?

[AdamISZ]

@tailsjoin yes absolutely. This is still in a preparatory stage. The notes I'm leaving here are to make sure nothing's forgotten. Thanks for the wallets/ thing; we'll do that.

Does it work OK with wallets/ and logs/ ?

[tailsjoin]

Yes, generated a wallet and got a help menu from the rest. Because this is using libsecp256k1, I can't communicate with the pit, correct?

[AdamISZ]

@tailsjoin no, you can. it works fine. i've been running libsecp256k1 in the live pit for a couple of weeks.

[tailsjoin]

I'll keep testing then.

[tailsjoin]

How do I modify yieldgen settings with this?

[AdamISZ]

Quite :) That's one reason I'm not focusing on yieldgen, as mentioned above. I believe yieldgen should be run from source (Python anyway). Of course, there's nothing stopping us moving all configuration to the joinmarket.cfg file.

[tailsjoin]

Just caught this in the console during yieldgen:

[2015/11/11 21:00:00] ran unconfirmfun
/bin/bash: symbol lookup error: /lib/x86_64-linux-gnu/libncurses.so.5: undefined symbol: _nc_putchar

This is on debian 64, not Tails.

Edit: Happened on run confirmfun too (maybe expected).

[AdamISZ]

@tailsjoin at first glance I have no idea what's going on there; wondering where a dependency on ncurses comes from. Thanks for doing this testing, much appreciated. I will come back and try to reproduce this when I've finished with Windows.

[tailsjoin]

Let me know any other outputs you need, including logs. This is a test environment, so there's no privacy issues.

[fivepiece]

@tailsjoin can you tell which version of debian you're running?
Thanks

[tailsjoin]

jessie

[tailsjoin]

I don't know if it makes a difference, but I dropped the *debian_64 files in ~/bin/ and I'm running them from within my joinmarket/ dir.

[fivepiece]

Thanks, I have a jessie vm ready, can you tell the output of apt-cache policy libncurses5 ?

[tailsjoin]

$ apt-cache policy libncurses5
libncurses5:
  Installed: 5.9+20140913-1+b1
  Candidate: 5.9+20140913-1+b1
  Version table:
 *** 5.9+20140913-1+b1 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

[fivepiece]

yep, we're running the same version. I'll try re-creating this on #joinmarket-pit-test

[fivepiece]

Currently running 3 yigens, one tumbler, and one sendpayment on -test, seems like it's working

[tailsjoin]

I'm not sure. I just finished a successful tumble session, but with this same bug popping up on every tx.

[fivepiece]

Maybe it's related to if using bitcoin-cli or not. I'll try bitcoin-cli tomorrow, see if it happens here.

[tailsjoin]

Yes, I'm using bitcoin-rpc.

[AdamISZ]

(A bit of a mess, but so far it works)

Building for Windows:
Do the secp256k1 dll build on Ubuntu. Note the dependencies on mingw dev packages for this kind of build (dang forgot the names, can be found pretty easily I think).

Add --host=i686-w64-mingw32 to the ./configure call. Add LDFLAGS = -no-undefined to Makefile before making (to allow a dynamic library to be built). This generates a libsecp256k1-0.dll in the .libs/ directory.

(TODO: forgot to --use-endomorphism, should be important for perf.)

Moving over to Windows,

Assuming already have pip, need to pip install cffi. (NB: I'm a bit hazy now on whether there's a MSVCRT type dependency here, or somewhere else. It was eminently googlable).

I found that the easiest way to access the dll was by using the first described method in the cffi documentation. Copy the definitions variable (with the header definitions) from build.py and place it into the secp256k1.py file, and then make these calls:

from cffi import FFI
ffi = FFI()
ffi.cdef(definitions)
lib = ffi.dlopen("libsecp256k1.dll")

(or whatever you've called the dll). This means putting the dll in the cwd (well, if I remember right).

Also, don't forget to get hold of libsodium.dll as is described in the "install for Win7" page on the wiki here.

At this point the python repo will run correctly.

Final step is pyinstaller. Didn't pip install it, for the reasons mentioned near the start of this thread. Instead git clone it (you need git ofc).
Direct application of the command above: C:\path\to\pyinstaller\pyinstaller.py -F -p lib --hidden-import=cffi sendpayment.py will create an exe file, but it won't be valid. It doesn't have the libsodium reference. To fix that, after running the previous command, edit the created sendpayment.spec file, and in the EXE section alter:

a.binaries,

to:

a.binaries+[('libsodium.dll','libsodium.dll','BINARY')],

and then run C:\path\to\pyinstaller\pyinstaller.py sendpayment.spec (no need for the options this time). A new exe will be created in dist/ folder, and this will work.

I've now tested (done a regtest style test) with three exes created like this, and it was successful on my Win7 box. Next step will be to test it on a fresh machine, so not at all sure this job is finished yet.

[AdamISZ]

The win exes are in the repo mentioned above now, I'll be testing them more tomorrow.

[AdamISZ]

OK, so I've bundled the result of all this into a new repository: https://github.com/AdamISZ/JMBinary (not TJMBinary).

Testing on a fresh Windows 7 seems to work fine. I've done a couple of regtest runs with the yield-generator and sendpayment executables OK. The performance of the wallet loading is not good, some combination of Windows, a smaller machine, the way the exe is built/packaged over Python etc. But as long as the performance is not bad on Linux I don't think it's a big deal. I also rebuilt secp256k1 library with --enable-endomorphism which should help.

See the notes in https://github.com/AdamISZ/JMBinary/README.md . I don't think this is hugely significant right now, except in one sense: it makes it much easier to use joinmarket on Windows. What @fivepiece has done is more important probably, in as much as it allows us to have an install script including secp256k1. @chris-belcher let me know how you want to progress with that? The branch is ready (and I myself have been running it live for quite a while).

[tailsjoin]

How does one review the code in these? I have to admit I didn't try too hard, but still couldn't figure it out.

[AdamISZ]

@tailsjoin I don't know if there's a way to do that. The best method for trusting binaries is a gitian build. That's a non-trivial endeavour, though.

I can make pgp signatures for these files (I've started signing the commits at least), but it's a poor level of security (trust me) without a deterministic build.

[AdamISZ]

Figured out how to cross compile for Windows better today; include gmp compilation with mingw using the instructions here: https://wiki.filezilla-project.org/Cross_Compiling_FileZilla_3_for_Windows_under_Ubuntu_or_Debian_GNU/Linux. This allowed rebuilding the secp256k1.dll file with gmp for bignum (the other platforms already had it).

Then looked at performance and discovered the main source of slowness: secp256k1_context_create calls happening for every key instantiation, as in: https://github.com/ludbb/secp256k1-py/blob/master/secp256k1.py#L27. This is wrong, see: https://github.com/bitcoin/secp256k1/blob/master/include/secp256k1.h#L31-L33 . Pulling this out into a global created a huge speedup, so much so that wallet creation is basically instant.

I rebuilt binaries based on that for Windows. I will "backport" this speed up onto the secp256k1 branch next, so all versions (including not binary, just Python) have it.

Also added a note to the windows directory re: VC++ redistributable download, which is needed for sendpayment and tumbler (because they use libsodium) but not wallet-tool.

[AdamISZ]

OK, applied that speedup to the secp256k1 branch and updated all the binaries with it in the JMBinary repo. I will close this for now. Feel free to open it again.

[ofek]

@AdamISZ Did you successfully build gmp for Windows? libsecp256k1's configure can't find it for me

$ wget -q https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2 && tar -xjpf gmp-*.tar.bz2 && cd gmp* && ./configure --prefix=$HOME/prefix-win32 --host=i686-w64-mingw32 --disable-shared --enable-fat && sudo make && sudo make install && cd ..

then when configuring libsecp256k1

$ ./configure --prefix=$HOME/prefix-win32 --disable-shared --enable-static --with-bignum=gmp --host=i686-w64-mingw32

checking gmp.h usability... no
checking gmp.h presence... no
checking for gmp.h... no
configure: error: gmp bignum explicitly requested but libgmp not available

[AdamISZ]

@ofek I can only say I thought I did - but such a long time ago, I'd have to go back and try again.