added bionic builder host, others minor fixes

gitian-building.md

@@ -44,10 +44,10 @@ Any kind of virtualization can be used, for example:
 
 Please refer to the following documents to set up the operating systems and Gitian.
 
-|                                   | Debian                                                                             | Fedora                                                                             |
-|-----------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
-| Setup virtual machine (optional)  | [Create Debian VirtualBox](./gitian-building/gitian-building-create-vm-debian.md) | [Create Fedora VirtualBox](./gitian-building/gitian-building-create-vm-fedora.md) |
-| Setup Gitian                      | [Setup Gitian on Debian](./gitian-building/gitian-building-setup-gitian-debian.md) | [Setup Gitian on Fedora](./gitian-building/gitian-building-setup-gitian-fedora.md) |
+|                     | Ubuntu                                                                             | Debian                                                                             | Fedora                                                                             |
+|---------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
+| Setup VM (optional) | [Create Ubuntu VirtualBox](./gitian-building/gitian-building-create-vm-ubuntu.md)  | [Create Debian VirtualBox](./gitian-building/gitian-building-create-vm-debian.md)  | [Create Fedora VirtualBox](./gitian-building/gitian-building-create-vm-fedora.md)  |
+| Setup Gitian        | [Setup Gitian on Ubuntu](./gitian-building/gitian-building-setup-gitian-ubuntu.md) | [Setup Gitian on Debian](./gitian-building/gitian-building-setup-gitian-debian.md) | [Setup Gitian on Fedora](./gitian-building/gitian-building-setup-gitian-fedora.md) |
 
 Note that a version of `lxc-execute` higher or equal to 2.1.1 is required.
 You can check the version with `lxc-execute --version`.
@@ -96,13 +96,22 @@ To speed up the build, use `-j 5 -m 5000` as the first arguments, where `5` is t
 
 If all went well, this produces a number of (uncommited) `.assert` files in the gitian.sigs repository.
 
-You need to copy these uncommited changes to your host machine, where you can sign them:
+You need to copy these uncommited changes to your host machine:
 
 ```
 export NAME=satoshi
-gpg --output $VERSION-linux/$NAME/bitcoin-linux-0.16-build.assert.sig --detach-sign 0.16.0rc1-linux/$NAME/bitcoin-linux-0.16-build.assert 
-gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-osx-0.16-build.assert.sig --detach-sign 0.16.0rc1-osx-unsigned/$NAME/bitcoin-osx-0.16-build.assert 
-gpg --output $VERSION-win-unsigned/$NAME/bitcoin-win-0.16-build.assert.sig --detach-sign 0.16.0rc1-win-unsigned/$NAME/bitcoin-win-0.16-build.assert 
+cd gitian.sigs
+scp -r gitian:gitian.sigs/0.16.0rc1-linux/$NAME/ 0.16.0rc1-linux/
+scp -r gitian:gitian.sigs/0.16.0rc1-osx-unsigned/$NAME/ 0.16.0rc1-osx-unsigned/
+scp -r gitian:gitian.sigs/0.16.0rc1-win-unsigned/$NAME/ 0.16.0rc1-win-unsigned/
+```
+
+Then you can sign them:
+
+```
+gpg --output 0.16.0rc1-linux/$NAME/bitcoin-linux-0.16-build.assert.sig --detach-sign 0.16.0rc1-linux/$NAME/bitcoin-linux-0.16-build.assert 
+gpg --output 0.16.0rc1-osx-unsigned/$NAME/bitcoin-osx-0.16-build.assert.sig --detach-sign 0.16.0rc1-osx-unsigned/$NAME/bitcoin-osx-0.16-build.assert 
+gpg --output 0.16.0rc1-win-unsigned/$NAME/bitcoin-win-0.16-build.assert.sig --detach-sign 0.16.0rc1-win-unsigned/$NAME/bitcoin-win-0.16-build.assert 
 ```
 
 Make a PR (both the `.assert` and `.assert.sig` files) to the
@@ -128,6 +137,6 @@ This will create the `.sig` files that can be committed together with the `.asse
 Gitian build.
 
 
- `./gitian-build.py --detach-sign -s satoshi 0.16.0rc1 --nocommit`
+ `./gitian-build.py --detach-sign -s satoshi 0.16.0rc1 --no-commit`
 
 Make another pull request for these.

gitian-building/gitian-building-create-vm-ubuntu.md

@@ -0,0 +1,238 @@
+# Setup Ubuntu virtual machine on VirtualBox
+
+Table of Contents
+------------------
+
+- [Create a new VirtualBox VM](#create-a-new-virtualbox-vm)
+- [Connecting to the VM](#connecting-to-the-vm)
+
+Create a new VirtualBox VM
+---------------------------
+In the VirtualBox GUI click "New" and choose the following parameters in the wizard:
+
+![](figs/create_new_vm_ubuntu.png)
+
+- Type: Linux, Ubuntu (64-bit)
+
+![](figs/create_vm_memsize.png)
+
+- Memory Size: at least 3000MB, anything less and the build might not complete.
+
+![](figs/create_vm_hard_disk.png)
+
+- Hard Disk: Create a virtual hard disk now
+
+![](figs/create_vm_hard_disk_file_type.png)
+
+- Hard Disk file type: Use the default, VDI (VirtualBox Disk Image)
+
+![](figs/create_vm_storage_physical_hard_disk.png)
+
+- Storage on physical hard disk: Dynamically Allocated
+
+![](figs/create_vm_file_location_size.png)
+
+- File location and size: at least 40GB
+- Click `Create`
+
+After creating the VM, we need to configure it.
+
+- Click the `Settings` button, then go to `System` tab and `Processor` sub-tab. Increase the number of processors to the number of cores on your machine if you want builds to be faster.
+
+![](figs/system_settings.png)
+
+- Go to the `Network` tab. Adapter 1 should be attached to `NAT`.
+
+![](figs/network_settings.png)
+
+- Click `Advanced`, then `Port Forwarding`. We want to set up a port through which we can reach the VM to get files in and out.
+- Create a new rule by clicking the plus icon.
+
+![](figs/port_forwarding_rules.png)
+
+- Set up the new rule the following way:
+  - Name: `SSH`
+  - Protocol: `TCP`
+  - Leave Host IP empty
+  - Host Port: `22222`
+  - Leave Guest IP empty
+  - Guest Port: `22`
+
+- Click `Ok` twice to save.
+
+Get the [Ubuntu 18.04.x minimal iso image](http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/mini.iso) (a more recent minor version should also work, see also [Installation/MinimalCD](https://help.ubuntu.com/community/Installation/MinimalCD)).
+This DVD image can be [validated](https://help.ubuntu.com/community/Installation/MinimalCD) using a SHA1 hashing tool, for example on
+Unixy OSes by entering the following in a terminal:
+
+    echo "cce936c1f9d1448c7d8f74b76b66f42eb4f93d4a  mini.iso" | sha1sum -c
+    # (must return OK)
+
+Replace `sha1sum` with `shasum` on OSX.
+
+Then start the VM. On the first launch you will be asked for a CD or DVD image. Choose the downloaded ISO.
+
+![](figs/select_startup_disk_ubuntu.png)
+
+Installing Ubuntu
+------------------
+
+This section will explain how to install Ubuntu on the newly created VM.
+
+- Choose the non-graphical installer (only if you will use the standard iso image instead of the suggested mini.iso, you will see the graphical installer choice).  We do not need the graphical environment; it will only increase installation time and disk usage.
+
+![](figs/ubuntu_install_1_boot_menu.png)
+
+**Note**: Navigating in the Ubuntu installer:
+To keep a setting at the default and proceed, just press `Enter`.
+To select a different button, press `Tab`.
+
+- Choose locale and keyboard settings (doesn't matter, you can just go with the defaults or select your own information)
+
+![](figs/ubuntu_install_2_select_a_language.png)
+![](figs/ubuntu_install_3_select_location.png)
+![](figs/ubuntu_install_4_configure_keyboard1.png)
+![](figs/ubuntu_install_4_configure_keyboard2.png)
+
+- The VM will detect network settings using DHCP, this should all proceed automatically
+- Set the hostname `ubuntu`.
+<!--
+  - Leave domain name empty.
+-->
+![](figs/ubuntu_install_5_network1_hostname.png)
+
+  - Choose a mirror (any will do)
+
+![](figs/ubuntu_install_5_network2_mirror_country.png)
+![](figs/ubuntu_install_5_network3_mirror.png)
+
+  - Enter proxy information (unless you are on an intranet, leave this empty)
+
+![](figs/ubuntu_install_5_network4_proxy.png)
+<!--
+![](figs/ubuntu_install_6_domain_name.png)
+
+- You can leave the root password empty. Otherwise, enter it twice and remember it for later.
+
+![](figs/ubuntu_install_6a_set_up_root_password.png)
+-->
+
+- Wait a bit while 'Loading additional components' runs
+- Name the new user `gitianuser` (the full name doesn't matter, you can leave it empty)
+- Set the account username as `gitianuser`
+
+![](figs/ubuntu_install_7_set_up_user_fullname.png)
+![](figs/ubuntu_install_8_set_up_username.png)
+
+- Choose a user password and enter it twice (remember it for later)
+
+![](figs/ubuntu_install_9_user_password.png)
+
+- The installer will set up the clock using a time server; this process should be automatic
+- Set up the clock: choose a time zone (depends on the locale settings that you picked earlier; specifics don't matter)  
+
+![](figs/ubuntu_install_10_configure_clock.png)
+
+- Disk setup
+  - Partitioning method: Guided - Use the entire disk
+
+![](figs/ubuntu_install_11_partition_disks.png)
+
+  - Select disk to partition: SCSI1 (0,0,0)
+
+![](figs/ubuntu_install_12_choose_disk.png)
+
+  - Finish partitioning and write changes to disk -> *Yes* (`Tab`, `Enter` to select the `Yes` button)
+
+![](figs/ubuntu_install_15_write_changes.png)
+
+- The base system will be installed, this will take some minutes
+- Select how to manage system upgrades -> *No automatic updates*
+
+![](figs/ubuntu_install_17_software_updates.png)
+
+- Wait a bit while 'Select and install software' runs
+- Choose software to install. We need just the base system.
+- Make sure only 'OpenSSH server' and 'Basic Ubuntu Server' are checked
+
+![](figs/ubuntu_install_19_software_selection.png)
+
+- Install the GRUB boot loader to the master boot record? -> Yes
+
+![](figs/ubuntu_install_20_install_grub.png)
+
+- Is the system clock set to UTC? -> *Yes*
+
+![](figs/ubuntu_install_21_configure_clock.png)
+
+- Installation Complete -> *Continue*
+- After installation, the VM will reboot and you will have a working Ubuntu VM. Congratulations!
+
+![](figs/ubuntu_install_22_finish_installation.png)
+
+Connecting to the VM
+----------------------
+
+After the VM has booted you can connect to it using SSH, and files can be copied from and to the VM using a SFTP utility.
+Connect to `localhost`, port `22222` (or the port configured when installing the VM).
+On Windows you can use [putty](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) and [WinSCP](http://winscp.net/eng/index.php).
+
+For example, to connect as `gitianuser` from a Linux command prompt use
+
+      $ ssh gitianuser@localhost -p 22222
+      The authenticity of host '[localhost]:22222 ([127.0.0.1]:22222)' can't be established.
+      RSA key fingerprint is ae:f5:c8:9f:17:c6:c7:1b:c2:1b:12:31:1d:bb:d0:c7.
+      Are you sure you want to continue connecting (yes/no)? yes
+      Warning: Permanently added '[localhost]:22222' (RSA) to the list of known hosts.
+      gitianuser@localhost's password: (enter gitianuser password configured during install)
+
+      System information as of Sat Sep 22 19:33:49 EDT 2018
+
+      System load:  0.0               Processes:             97
+      Usage of /:   8.8% of 39.12GB   Users logged in:       0
+      Memory usage: 4%                IP address for enp0s3: 10.0.2.15
+      Swap usage:   0%
+
+    0 packages can be updated.
+    0 updates are security updates.
+
+
+
+    The programs included with the Ubuntu system are free software;
+    the exact distribution terms for each program are described in the
+    individual files in /usr/share/doc/*/copyright.
+
+    Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
+    applicable law.
+
+    To run a command as administrator (user "root"), use "sudo <command>".
+    See "man sudo_root" for details.
+
+    gitianuser@ubuntu:~$
+
+Optional - Easier login to the VM
+---------------------------------
+
+You'll need to generate an SSH key, e.g. by following the instructions under "Generating a new SSH key" [here](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
+
+After that, login to the VM and enter:
+
+```bash
+mkdir .ssh
+```
+
+On your machine edit or create `~/.ssh/config` and add:
+
+```bash
+Host gitian
+    HostName localhost
+    Port 22222
+    User gitianuser
+```
+
+Open a new terminal tab and enter:
+
+```bash
+scp ~/.ssh/id_rsa.pub gitian:.ssh/authorized_keys
+```
+
+Next time you need to login to the VM, just use: `ssh gitian`

gitian-building/gitian-building-setup-gitian-debian.md

@@ -39,6 +39,7 @@ chmod +x /etc/rc.local
 echo 'export USE_LXC=1' >> /home/gitianuser/.profile
 echo 'export GITIAN_HOST_IP=10.0.3.1' >> /home/gitianuser/.profile
 echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/gitianuser/.profile
+[ -f /etc/default/lxc-net ] && sed -i 's/USE_LXC_BRIDGE="true"/USE_LXC_BRIDGE="false"/' /etc/default/lxc-net
 reboot
 ```
 

gitian-building/gitian-building-setup-gitian-ubuntu.md

@@ -0,0 +1,99 @@
+Table of Contents
+------------------
+
+- [Setting up Ubuntu for Gitian building](#setting-up-ubuntu-for-gitian-building)
+- [Installing Gitian](#installing-gitian)
+- [Setting up the Gitian image](#setting-up-the-gitian-image)
+
+
+Setting up Ubuntu for Gitian building
+--------------------------------------
+
+In this section we will be setting up the Ubuntu installation for Gitian building.
+We assume that a user `gitianuser` was previously added.
+
+First we need to set up dependencies. Type/paste the following in the terminal:
+
+```bash
+sudo apt-get install git ruby apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils make curl firewalld
+```
+
+Then set up LXC and the rest with the following, which is a complex jumble of settings and workarounds:
+
+```bash
+sudo -s
+# the version of lxc-start in Ubuntu needs to run as root, so make sure
+# that the build script can execute it without providing a password
+echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
+echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
+# make /etc/rc.local script that sets up bridge between guest and host
+echo '#!/bin/sh -e' > /etc/rc.local
+echo 'brctl addbr br0' >> /etc/rc.local
+echo 'ip addr add 10.0.3.2/24 broadcast 10.0.3.255 dev br0' >> /etc/rc.local
+echo 'ip link set br0 up' >> /etc/rc.local
+echo 'firewall-cmd --zone=trusted --add-interface=br0' >> /etc/rc.local
+echo 'exit 0' >> /etc/rc.local
+chmod +x /etc/rc.local
+# make sure that USE_LXC is always set when logging in as gitianuser,
+# and configure LXC IP addresses
+echo 'export USE_LXC=1' >> /home/gitianuser/.profile
+echo 'export GITIAN_HOST_IP=10.0.3.2' >> /home/gitianuser/.profile
+echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/gitianuser/.profile
+[ -f /etc/default/lxc-net ] && sed -i 's/USE_LXC_BRIDGE="true"/USE_LXC_BRIDGE="false"/' /etc/default/lxc-net
+reboot
+```
+
+At the end Ubuntu is rebooted to make sure that the changes take effect. The steps in this
+section only need to be performed once.
+
+**Note**: When sudo asks for a password, enter the password for the user `gitianuser`.
+
+Installing Gitian
+------------------
+
+Re-login as the user `gitianuser` that was created during installation.
+The rest of the steps in this guide will be performed as that user.
+
+There is no `python-vm-builder` package in Ubuntu, so we need to install it from source ourselves,
+
+```bash
+wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
+echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e  vm-builder_0.12.4+bzr494.orig.tar.gz" | sha256sum -c
+# (verification -- must return OK)
+tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
+cd vm-builder-0.12.4+bzr494
+sudo python setup.py install
+cd ..
+```
+
+**Note**: When sudo asks for a password, enter the password for the user `gitianuser`.
+
+Clone the git repositories for bitcoin and Gitian.
+
+```bash
+git clone https://github.com/devrandom/gitian-builder.git
+git clone https://github.com/bitcoin/bitcoin
+git clone https://github.com/bitcoin-core/gitian.sigs.git
+git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git
+```
+
+Setting up the Gitian image
+-------------------------
+
+Gitian needs a virtual image of the operating system to build in.
+Currently this is Ubuntu Bionic x86_64 and the script gitian-build.py setup it for you.
+For previous releases of bitcoin the image was Trusty x86_64, so if you want to build bitcoin 0.16.x or earlier you have to create it now manually.
+This image will be copied and used every time that a build is started to
+make sure that the build is deterministic.
+Creating the image will take a while, but only has to be done once.
+
+Execute the following as user `gitianuser`:
+
+```bash
+cd gitian-builder
+bin/make-base-vm --lxc --arch amd64 --suite trusty
+```
+
+There will be a lot of warnings printed during the build of the image. These can be ignored.
+
+**Note**: When sudo asks for a password, enter the password for the user `gitianuser` not for `root`.