|
| 1 | +Deterministic / Gitian building Mac OSX |
| 2 | +======================================= |
| 3 | + |
| 4 | +Working OS X DMGs are created in Linux by combining a recent clang, |
| 5 | +the Apple binutils (ld, ar, etc) and DMG authoring tools. |
| 6 | + |
| 7 | +Apple uses clang extensively for development and has upstreamed the necessary |
| 8 | +functionality so that a vanilla clang can take advantage. It supports the use |
| 9 | +of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary |
| 10 | +when building for OS X. |
| 11 | + |
| 12 | +Apple's version of binutils (called cctools) contains lots of functionality |
| 13 | +missing in the FSF's binutils. In addition to extra linker options for |
| 14 | +frameworks and sysroots, several other tools are needed as well such as |
| 15 | +install_name_tool, lipo, and nmedit. These do not build under linux, so they |
| 16 | +have been patched to do so. The work here was used as a starting point: |
| 17 | +[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). |
| 18 | + |
| 19 | +In order to build a working toolchain, the following source packages are needed |
| 20 | +from Apple: cctools, dyld, and ld64. |
| 21 | + |
| 22 | +These tools inject timestamps by default, which produce non-deterministic |
| 23 | +binaries. The ZERO_AR_DATE environment variable is used to disable that. |
| 24 | + |
| 25 | +This version of cctools has been patched to use the current version of clang's |
| 26 | +headers and and its libLTO.so rather than those from llvmgcc, as it was |
| 27 | +originally done in toolchain4. |
| 28 | + |
| 29 | +To complicate things further, all builds must target an Apple SDK. These SDKs |
| 30 | +are free to download, but not redistributable. |
| 31 | +To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). |
| 32 | + |
| 33 | +This file is several gigabytes in size, but only a single directory inside is |
| 34 | +needed: |
| 35 | + |
| 36 | +```bash |
| 37 | +Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk |
| 38 | +``` |
| 39 | + |
| 40 | +## How To: |
| 41 | + |
| 42 | +On the host machine, register for a free Apple [developer account](https://developer.apple.com/register/), then download the SDK [here](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). |
| 43 | + |
| 44 | +### MacOS host |
| 45 | + |
| 46 | +Using Mac OS X, you can mount the dmg, and then extract the SDK with: |
| 47 | + |
| 48 | +```bash |
| 49 | + hdiutil attach Xcode_7.3.1.dmg |
| 50 | + tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk |
| 51 | +``` |
| 52 | + |
| 53 | +Clean up the files you don't need: |
| 54 | + |
| 55 | +```bash |
| 56 | +diskutil unmount /Volumes/Xcode |
| 57 | +rm MacOSX10.11.sdk.tar.gz Xcode_7.3.1.dmg |
| 58 | +``` |
| 59 | + |
| 60 | +### Non-MacOS host: |
| 61 | +-------- |
| 62 | + |
| 63 | +Alternatively, you can use 7zip and SleuthKit to extract the files one by one. |
| 64 | +The script [extract-osx-sdk.sh](..//contrib/macdeploy/extract-osx-sdk.sh) automates this. First ensure |
| 65 | +the dmg file is in the current directory, and then run the script. |
| 66 | + |
| 67 | +You may wish to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when |
| 68 | +you've confirmed the extraction succeeded. |
| 69 | + |
| 70 | +```bash |
| 71 | +apt-get install p7zip-full sleuthkit |
| 72 | +contrib/macdeploy/extract-osx-sdk.sh |
| 73 | +rm -rf 5.hfs MacOSX10.11.sdk |
| 74 | +``` |
| 75 | + |
| 76 | + |
| 77 | +### Alternatively - Download it: |
| 78 | + |
| 79 | +Alternatively you can download the SDK from `phracker` github, and create a tar.gz file |
| 80 | +using the following commands: |
| 81 | + |
| 82 | +```bash |
| 83 | +wget https://github.com/phracker/MacOSX-SDKs/releases/download/MacOSX10.11.sdk/MacOSX10.11.sdk.tar.xz |
| 84 | +tar xvf MacOSX10.11.sdk.tar.xz |
| 85 | +tar -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk |
| 86 | +``` |
| 87 | + |
| 88 | + |
| 89 | + |
| 90 | +## (Gitian builder only) Copy SDK to Gitian VM : |
| 91 | + |
| 92 | +Copy it to the Gitian VM, e.g.: |
| 93 | + |
| 94 | +```bash |
| 95 | +scp MacOSX10.11.sdk.tar.gz gitian: |
| 96 | +``` |
| 97 | + |
| 98 | +Login to the VM and: |
| 99 | + |
| 100 | +```bash |
| 101 | +mkdir gitian-builder/inputs |
| 102 | +mv MacOSX10.11.sdk.tar.gz gitian-builder/inputs |
| 103 | +``` |
| 104 | + |
| 105 | + |
| 106 | +## Info |
| 107 | +The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries |
| 108 | +which are created using these tools. The build process has been designed to |
| 109 | +avoid including the SDK's files in Gitian's outputs. All interim tarballs are |
| 110 | +fully deterministic and may be freely redistributed. |
| 111 | + |
| 112 | +genisoimage is used to create the initial DMG. It is not deterministic as-is, |
| 113 | +so it has been patched. A system genisoimage will work fine, but it will not |
| 114 | +be deterministic because the file-order will change between invocations. |
| 115 | +The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). |
| 116 | +No effort was made to fix this cleanly, so it likely leaks memory badly. But |
| 117 | +it's only used for a single invocation, so that's no real concern. |
| 118 | + |
| 119 | +genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the |
| 120 | +libdmg-hfsplus project is used to compress it. There are several bugs in this |
| 121 | +tool and its maintainer has seemingly abandoned the project. It has been forked |
| 122 | +and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). |
| 123 | + |
| 124 | +The 'dmg' tool has the ability to create DMGs from scratch as well, but this |
| 125 | +functionality is broken. Only the compression feature is currently used. |
| 126 | +Ideally, the creation could be fixed and genisoimage would no longer be necessary. |
| 127 | + |
| 128 | +Background images and other features can be added to DMG files by inserting a |
| 129 | +.DS_Store before creation. This is generated by the script |
| 130 | +contrib/macdeploy/custom_dsstore.py. |
| 131 | + |
| 132 | +As of OS X Mavericks (10.9), using an Apple-blessed key to sign binaries is a |
| 133 | +requirement in order to satisfy the new Gatekeeper requirements. Because this |
| 134 | +private key cannot be shared, we'll have to be a bit creative in order for the |
| 135 | +build process to remain somewhat deterministic. Here's how it works: |
| 136 | + |
| 137 | +- Builders use Gitian to create an unsigned release. This outputs an unsigned |
| 138 | + dmg which users may choose to bless and run. It also outputs an unsigned app |
| 139 | + structure in the form of a tarball, which also contains all of the tools |
| 140 | + that have been previously (deterministically) built in order to create a |
| 141 | + final dmg. |
| 142 | +- The Apple keyholder uses this unsigned app to create a detached signature, |
| 143 | + using the script that is also included there. Detached signatures are available from this [repository](https://github.com/navcoin-core/navcoin-detached-sigs). |
| 144 | +- Builders feed the unsigned app + detached signature back into Gitian. It |
| 145 | + uses the pre-built tools to recombine the pieces into a deterministic dmg. |
0 commit comments