Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 351 lines (258 sloc) 18.958 kb
2833436 Ryan Daigle Draft of vulcan article
authored
1 Title: Using Vulcan to Build Binary Dependencies for Heroku
2 Draft: true
3 Date: 15 June, 2012
4
5 Managing an application's code dependencies, which was once a source of constant pain and conflict, is now a solved problem in most modern languages. Ruby has [Bundler](http://gembundler.com/), Node.js has [npm](http://npmjs.org/), Python has [pip](http://pypi.python.org/pypi/pip), Clojure has [Leiningen](https://github.com/technomancy/leiningen)… the list continues.
6
7 What remains unsolved is how to declare and manage system-level dependencies -- external binaries that your application is dependent upon.
8
9 <!--
10 This post will explore explicitly managing and building system-level dependencies using the [Vulcan](https://github.com/heroku/vulcan) build server.
11 -->
12
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
13 ## Problem
2833436 Ryan Daigle Draft of vulcan article
authored
14
15 It's common for an application to shell out to a local executable to perform some computationally intense work where compiled libraries are more performant or robust. Some common examples include [Ghostscript](http://www.ghostscript.com/), which your application might use to manipulate PDF or PostScript files, or [ImageMagick](http://www.imagemagick.org/script/index.php) for resizing and cropping uploaded images.
16
17 Many deployment and hosting services attempt to fill this need by providing a set of common binaries bundled into every environment. This is a fragile approach that boxes you in to out-dated or incompatible versions more than it provides the exact version your app needs.
18
19 Additionally, having the ability to install system-wide binaries in your deployment environment is a poor solution. It merely shifts the burden of systems management from the service provider to you, the application developer. Without application-level isolation, installing binaries on a remote system still exposes you to dependency hell.
20
21 [Twelve-factor](http://www.12factor.net/) makes [the case](http://www.12factor.net/dependencies) to "not rely on the implicit existence of any system tools" for the deleterious effects it may have on future environment migration and upgrade efforts. Instead of solving system dependencies at a system level solve them at the application level by explicitly bundling binaries with the application that requires them.
22
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
23 ## Solution
2833436 Ryan Daigle Draft of vulcan article
authored
24
25 Bundling a binary dependency requires that the binary be built specifically for the remote environment's operating system and processor architecture. This is a non-trivial task when you're faced with having to boot up a local host-VM just to simulate the remote server environment. Even then the processor architecture may be in conflict.
26
27 A better solution is to use the remote environment of your service provider to compile and build the required dependencies. Such a process would look something like this:
28
29 1. Specify the required library's source
30 2. Get a remote shell to your production environment
31 3. Download the library source in the remote shell
32 3. Compile the library remotely
33 4. Download the compiled library for use in your application's source-tree
34
35 While this process is not a lengthy one, your eye should spot that steps 2-4 are ripe for automation. That's where Vulcan comes in.
36
37 [Vulcan](https://github.com/heroku/vulcan) is a utility that bundles and uploads a source-tree to a remote environment, runs the necessary compilation commands on the source tree, and downloads the binary -- all in a single command. Vulcan consists of a Ruby-based CLI and Node.js server component and, though built by and for Heroku, is platform-agnostic.
38
39 **IMAGE(s)**
40
41 ## Setup
42
43 <p class="note">
44 These steps assume you have <a href="http://git-scm.com/">git</a> and <a href="http://www.ruby-lang.org/en/">ruby</a> available from the command line and have already signed up for a <a href="https://api.heroku.com/signup">Heroku account</a>. The <a href="http://toolbelt.heroku.com/">Heroku Toolbelt</a> can get you up and running if you're missing any components.
45 </p>
46
47 Installing the Vulcan CLI is simply a matter of installing the `vulcan` gem:
48
49 <pre lang="bash"><code>
50 $ gem install vulcan
51 Please run 'vulcan update' to update your build server.
52 Successfully installed vulcan-0.7.1
53 1 gem installed
54 </code></pre>
55
56 Vulcan's [build-server](https://github.com/heroku/vulcan/tree/master/server) is a simple Node.js app that should be running _in the same remote environment your application will be deployed_. If your app runs on Heroku, Vulcan can deploy itself with `vulcan create appname`.
57
58 <pre lang="bash"><code>
59 $ vulcan create buildserver-you
60 Creating buildserver-you... done, stack is cedar
61 http://buildserver-you.herokuapp.com/ | git@heroku.com:buildserver-you.git
62 Initialized empty Git repository in /private/var/folders/Uz/UzRCgjzkGIi7Iqz9QNi6NUDrHf6/-Tmp-/d20120614-31875-1qjw1d6/.git/
63 Counting objects: 883, done.
64 ...
65
66 -----> Heroku receiving push
67 -----> Node.js app detected
68 ...
69 Dependencies installed
70 -----> Discovering process types
71 Procfile declares types -> web
72 -----> Compiled slug size is 4.1MB
73 -----> Launching... done, v3
74 http://buildserver-you.herokuapp.com deployed to Heroku
75 </code></pre>
76
77 The Vulcan build server is now running on Heroku at `http://buildserver-you.herokuapp.com` as a (free) single-dyno app.
78
79 <p class="note" markdown="1">
80 If you've manually deployed the build server to another provider you'll need to set its location so the CLI knows where to send build tasks. This is done by [setting](https://github.com/heroku/vulcan/blob/master/lib/vulcan/cli.rb#L40) the `VULCAN_HOST` env var: `$ export VULCAN_HOST=http://myserver.domain.com`
81 </p>
82
83 ## Build
84
85 The Ghostscript library was utilized in a tutorial on the [Heroku Dev Center](http://devcenter.heroku.com) for [processing PDF files](https://devcenter.heroku.com/articles/processing-pdfs-ruby-mongo). It makes for a good example of building a binary application dependency.
86
87 ### Download source
88
89 First, download and expand the Linux x86 64-bit source for the [Ghostscript project](http://www.ghostscript.com/). At the time of this writing it is located at [http://downloads.ghostscript.com/public/binaries/ghostscript-9.05-linux-x86_64.tgz](http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz).
90
91 <p class="note" markdown="1">
92 For the purpose of this article Heroku will be assumed to be the target environment. [Heroku's dynos](https://devcenter.heroku.com/articles/dynos) run on 64-bit Linux kernel. If your target environment differs you will need to select the correct source distribution.
93 </p>
94
95 <pre lang="bash"><code>
96 $ wget http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz
97 $ tar -xvzf ghostscript-9.05.tar.gz
98 x ghostscript-9.05/
99 x ghostscript-9.05/base/
100 x ghostscript-9.05/base/szlibxx.h
101 ...
102 </code></pre>
103
104 You now have a Ghostscript source directory at `./ghostscript-9.05`.
105
106 ### Remote compilation
107
108 Next, use the Vulcan CLI to initiate a build task on the build server. This will send the source to the target environment for compilation. The only required argument is `-s`, location of the Ghostscript source directory. The `-v` flag (verbose) will show the output from the compilation process and is recommended as, depending on the library, compilation can take some time and it's useful to see its progress.
109
110 <pre lang="bash"><code>
111 $ vulcan build -v -s ./ghostscript-9.05
112 Packaging local directory... done
113 Uploading source package... done
114 Building with: ./configure --prefix /app/vendor/ghostscript-9 &amp;&amp; make install
115 checking for gcc... gcc
116 checking whether the C compiler works... yes
117 ...
118 >> Downloading build artifacts to: /tmp/ghostscript-9.tgz
119 (available at http://buildserver-you.herokuapp.com/output/45380fa6-e02a-479b-a7af-d9afb089b81f)
120 </code></pre>
121
122 On completion of the build process the resulting binaries are packages and downloaded for you. In this example the binary package can be found locally at `/tmp/ghostscript-9.tgz`. While the binary package is also available on your build server (here at `http://buildserver-you.herokuapp.com/output/45380fa6-e02a-479b-a7af-d9afb089b81f`) this is a temporary location. Due to the ephemeral nature of cloud filesystems the file is likely not to exist at that location for an extended period of time and should not be relied on.
123
124 Keep in mind, while the result are the library binaries they will not be executable on your local machine due to the difference in processor architecture of your local and target environments.
125
126 <p class="warning" markdown="1">
127 Although the [Vulcan source](https://github.com/heroku/vulcan) indicates the ability to fetch source packages directly from a URL (i.e. `$ vulcan build -s http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz` this tends to result in build errors. The most reliable method is to manually download and expand the source before passing to vulcan.
128 </p>
129
130 ## Customization
131
132 Looking at the output from the Ghostscript example build you can see that a sensible build command is chosen for you: `./configure --prefix /app/vendor/ghostscript-9 && make install`. If you need to specify a non-default build command you can do so with the `-c` flag. Here is an example adding the `--without-ssl` configure option when building `wget`.
133
134 <pre lang="bash"><code>
135 $ vulcan build -v -s ./wget-1.13 -c "./configure --prefix /app/vendor/wget-1.13 --without-ssl &amp;&amp; make install" -p /app/vendor/wget-1.13
136 Packaging local directory... done
137 Uploading source package... done
138 Building with: ./configure --without-ssl &amp;&amp; make install
139 configure: configuring for GNU Wget 1.13
140 ...
141 >> Downloading build artifacts to: /tmp/wget-1.tgz
142 (available at http://buildserver-you.herokuapp.com/output/66ba3e2b-77ef-4409-acc2-fca70650c318)
143 </code></pre>
144
145 The `-p` (prefix) flag is also used here to tell Vulcan where to look on the build server for the compiled artifacts (`/app/vendor/wget-1.13`). To avoid ambiguities it's best to specify this value and to set it to the same value as the `--prefix` flag passed to `./configure`.
146
147 ## Visibility
148
149 There are several utilities available to you to introspect the remote compilation process.
150
151 ### Logging
152
153 Outside of using the `-v` flag to force see the build output ou can also use the logs of the Vulcan build server to gain better visibility into the process. Since the Vulcan build server is just a Node.js app running in your target environment this is a trivial task. On Heroku use `heroku logs` and the `-a` flag with the name of your build server app.
154
155 <pre lang="bash"><code>
156 $ heroku logs -t -a buildserver-you
157 2012-07-04T15:29:20+00:00 app[web.1]: [7f3a7510-400a-44d4-9132-66a2e6c878a5] spawning build
158 2012-07-04T15:29:20+00:00 app[web.1]: valid socket
159 2012-07-04T15:29:21+00:00 heroku[run.1]: Awaiting client
160 2012-07-04T15:29:21+00:00 heroku[run.1]: Starting process with command `bin/make "7f3a7510-400a-44d4-9132-66a2e6c878a5"`
161 2012-07-04T15:29:22+00:00 heroku[run.1]: State changed from starting to up
162 2012-07-04T15:30:10+00:00 heroku[run.1]: Process exited with status 1
163 2012-07-04T15:30:10+00:00 heroku[run.1]: State changed from up to complete
164 </code></pre>
165
166 By default Vulcan will spawn a [on-off dyno](https://devcenter.heroku.com/articles/oneoff-admin-ps) to perform the build command. This is evident by the `run.1` dyno in the log output. In some circumstances this can result in a billable event if your web and one-off dyno usage combined [exceeds 750 hours for any one month](https://devcenter.heroku.com/articles/usage-and-billing).
167
168 If this small overage is meaningful to you you can sacrifice compilation concurrency and force Vulcan to execute the compilation command in-process with the `SPAWN_ENV` [config var](https://devcenter.heroku.com/articles/config-vars).
169
170 <pre lang="bash"><code>
171 $ heroku config:add SPAWN_ENV=local -a buildserver-you
172 Setting config vars and restarting buildserver-you... done, v11
173 SPAWN_ENV: local
174 </code></pre>
175
176 The build command will then execute within the web process:
177
178 <pre lang="bash"><code>
179 $ heroku logs -t -a buildserver-you
180 2012-07-04T15:35:43+00:00 app[web.1]: [bbeab2b8-3941-4d41-94af-24c4f0fa65c0] spawning build
181 2012-07-04T15:36:33+00:00 app[web.1]: 10.125.41.68 - - [Wed, 04 Jul 2012 15:36:33 GMT] "GET /output/bbeab2b8-3941-4d41-94af-24c4f0fa65c0 HTTP/1.1" 200 - "-" "Ruby"
182 2012-07-04T15:36:33+00:00 heroku[router]: GET buildserver-you.herokuapp.com/output/bbeab2b8-3941-4d41-94af-24c4f0fa65c0 dyno=web.1 queue=0 wait=0ms service=35ms status=200 bytes=75
183 </code></pre>
184
185 While this fails to adhere to the [background job](https://devcenter.heroku.com/articles/background-jobs-queueing) pattern it may be acceptable for your use-case.
186
187 ### Remote shell
188
189 If a build fails it is often useful to be able to view the failed artefacts. Since Vulcan performs its work in temporary directories their contents are cleaned up after each build. However, a remote shell can be used to manually invoke the build command and navigate the build results.
190
191 At the start of every build request Vulcan outputs log statements resembling the following:
192
193 <pre lang="bash"><code>
194 2012-07-04T18:57:10+00:00 app[web.1]: [6869e68a-492d-4a7e-8b27-64352811d7dc] saving to couchdb
195 2012-07-04T18:57:10+00:00 app[web.1]: [6869e68a-492d-4a7e-8b27-64352811d7dc] saving attachment - [id:6869e68a-492d-4a7e-8b27-64352811d7dc rev:1-722a96f6734a3511efd73b7cfb9a2aed]
196 </code></pre>
197
198 The attachment id, here `6869e68a-492d-4a7e-8b27-64352811d7dc`, is all that's needed to manually invoke the build yourself. Establish a remote shell to the Vulcan buildserver environment. On Heroku you can use `heroku run bash`:
199
200 <pre lang="bash"><code>
201 $ heroku run bash -a buildserver-you
202 ~ $
203 </code></pre>
204
205 Then invoke the `bin/make` command with the attachment id. You will see the output of the build process and can then browse the output directory yourself.
206
207 <pre lang="bash"><code>
208 $ bin/make "6869e68a-492d-4a7e-8b27-64352811d7dc"
209 configure: configuring for GNU Wget 1.13
210 checking for a BSD-compatible install... /usr/bin/install -c
211 checking whether build environment is sane... yes
212 ...
213 $ cd /app/vendor/wget-1.13
214 </code></pre>
215
216 ## Vendoring
217
218 ## Heroku binaries
219
220 Many binaries were compiled in the course of writing this article for use on Heroku. Here is a list in case they're useful to you:
221
222 <p class="note" markdown="1">
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
223 If you'd like to list a Heroku binary here, please [send a pull request](https://github.com/rwdaigle/ryandaigle.com/blob/master/content/pages/a/using-vulcan-to-build-binary-dependencies-on-heroku.mdown).
2833436 Ryan Daigle Draft of vulcan article
authored
224 </p>
225
226 <table>
227 <tr>
228 <th>Library</th>
229 <th>Build command</th>
230 <th>Binary</th>
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
231 <th>Contributor</th>
2833436 Ryan Daigle Draft of vulcan article
authored
232 </tr>
233 <tr>
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
234 <td><a href="http://ftp.gnu.org/gnu/wget/wget-1.13.tar.gz">GNU Wget v1.13</a></td>
235 <td><code>vulcan build -v -s ./wget-1.13 -c "./configure --prefix /app/vendor/wget-1.13 --without-ssl &amp;&amp; make install" -p /app/vendor/wget-1.13</code></td>
236 <td><a href="http://cl.ly/1B0n121X1T200g3u2a1k/wget-1.tgz">download</a></td>
237 <td><a href="https://twitter.com/rwdaigle">Ryan Daigle</a></td>
238 </tr>
239 <tr>
240 <td><a href="http://www.imagemagick.org/download/ImageMagick.tar.gz">ImageMagick v6.7.8-1</a></td>
241 <td><code>vulcan build -v -s ./ImageMagick-6.7.8-1</code></td>
242 <td><a href="http://cl.ly/011m0E3w2I360X2s1D2d/ImageMagick-6.7.tgz">download</a></td>
243 <td><a href="https://twitter.com/rwdaigle">Ryan Daigle</a></td>
244 </tr>
245 <tr>
246 <td><a href="http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz">Ghostscript v9.05</a></td>
247 <td><code>vulcan build -v -s ./ghostscript-9.05</code></td>
248 <td><a href="http://cl.ly/192z2W1H2i1e3o0W341V/ghostscript-9.tgz">download</a></td>
249 <td><a href="https://twitter.com/rwdaigle">Ryan Daigle</a></td>
2833436 Ryan Daigle Draft of vulcan article
authored
250 </tr>
251 </table>
252
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
253 ## Updates
2833436 Ryan Daigle Draft of vulcan article
authored
254
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
255 Updating Vulcan is quite simple. Update the CLI using ruby gems:
2833436 Ryan Daigle Draft of vulcan article
authored
256
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
257 <pre lang="bash"><code>
258 $ gem install vulcan
259 Please run 'vulcan update' to update your build server.
260 Successfully installed vulcan-0.8.0
261 1 gem installed
262 </code></pre>
2833436 Ryan Daigle Draft of vulcan article
authored
263
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
264 And use the `vulcan update` command to update the build server.
2833436 Ryan Daigle Draft of vulcan article
authored
265
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
266 <p class="warning" markdown="1">
267 Be aware that updating the build server while active compilations are running will cause them to be aborted.
268 </p>
2833436 Ryan Daigle Draft of vulcan article
authored
269
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
270 <pre lang="bash"><code>
271 $ vulcan update
272 Initialized empty Git repository in /private/var/folders/tt/7f38d4b14qq5xglpj3yl0smr0000gn/T/d20120704-53816-m4n0rn/.git/
273 Counting objects: 883, done.
274 ...
2833436 Ryan Daigle Draft of vulcan article
authored
275
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
276 -----> Heroku receiving push
277 -----> Node.js app detected
278 -----> Resolving engine versions
279 Using Node.js version: 0.6.18
280 Using npm version: 1.1.4
281 ...
282 -----> Launching... done, v15
283 http://buildserver-you.herokuapp.com deployed to Heroku
2833436 Ryan Daigle Draft of vulcan article
authored
284
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
285 To git@heroku.com:buildserver-you.git
286 + a5f27be...a934704 master -> master (forced update)
287 </code></pre>
288
289 ## Troubleshooting
290
291 ### Invalid secret
2833436 Ryan Daigle Draft of vulcan article
authored
292
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
293 If working across multiple development environments or some other non-default workflow you may see the following build server error logged when attempting to invoke a build:
294
295 <pre><code>
296 2012-07-04T17:39:47+00:00 app[web.1]: [672b5df6-ad8c-49ed-9831-515207e2dc4f] ERROR: invalid secret
2833436 Ryan Daigle Draft of vulcan article
authored
297 2012-07-04T17:39:47+00:00 app[web.1]: invalid secret
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
298 </code></pre>
299
300 This occurs when the CLI secret hash, created when the build server was created with `vulcan create`, either doesn't exist or doesn't match the server-side secret. The most common cause is that the `~/.vulcan` configuration file doesn't exist in your environment. You can create it with the following contents:
301
302 <h5 class="file">~/.vulcan</h5>
303 <pre lang="yaml"><code>
2833436 Ryan Daigle Draft of vulcan article
authored
304 ---
305 :app: buildserver-you
306 :host: buildserver-you.herokuapp.com
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
307 :secret: reallylonghash12df
308 </code></pre>
2833436 Ryan Daigle Draft of vulcan article
authored
309
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
310 If you don't have access to your original `.vulcan` file you can find your secret on Heroku using `heroku config`:
311
312 <pre lang="bash"><code>
313 $ heroku config:get SECRET -a buildserver-you
314 reallylonghash12df
315 </code></pre>
316
317 <p class="note" markdown="1">
318 Copy your `~/.vulcan` file to each development machine from which you wish to invoke builds.
319 </p>
2833436 Ryan Daigle Draft of vulcan article
authored
320
321 <!--
322
e873d6b Ryan Daigle Add troubleshooting and other supporting sections
authored
323 ## Notes
324
325 Vendoring/In practice: switch between local dev and remote (using the bin/ env)
326
327 style table
328
2833436 Ryan Daigle Draft of vulcan article
authored
329 ## Dependencies
330
331 Unlike Ghostscript, many libraries themselves have dependencies that are not available in your target environment. `wget` is a useful utility that requires the [GNU transport layer security library (gnutls)](http://www.gnu.org/software/gnutls/download.html) for SSL connections. To compile wget first compile gnutls.
332
333 <p class="note" markdown="1">
334 Recent versions of gnutls are [distributed](http://ftp.gnu.org/gnu/gnutls/) with [XZ](http://tukaani.org/xz/) file compression. It will be left as an excercise for the reader to download and decompress the gnutls source.
335 </p>
336
337 <pre lang="bash"><code>
338 $ vulcan build -v -s ./gnutls-3.0.21
339 Packaging local directory... done
340 Uploading source package... done
341 Building with: ./configure --prefix /app/vendor/ghostscript-9 && make install
342 checking for gcc... gcc
343 checking whether the C compiler works... yes
344 ...
345 </code></pre>
346
347 xz compressions: http://tukaani.org/xz/
348
349 wget as example, requires gnutils
350
351 -->
Something went wrong with that request. Please try again.