Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 347 lines (255 sloc) 18.997 kB
2833436 @rwdaigle Draft of vulcan article
authored
1 Title: Using Vulcan to Build Binary Dependencies for Heroku
2 Draft: true
3 Date: 15 June, 2012
4
5f7d6be @rwdaigle Small edits
authored
5 Managing an application's code dependencies, once a source of constant pain and conflict, is now a solved problem in most modern languages. Ruby has [Bundler](http://gembundler.com/), Node.js has [npm](http://npmjs.org/), Python has [pip](http://pypi.python.org/pypi/pip), Clojure has [Leiningen](https://github.com/technomancy/leiningen)… the list continues.
2833436 @rwdaigle Draft of vulcan article
authored
6
5f7d6be @rwdaigle Small edits
authored
7 What remains unsolved is how to declare and manage system-level dependencies -- external binaries on which your application is dependent. This article explores explicitly managing and building system-level dependencies using the [Vulcan](https://github.com/heroku/vulcan) build server.
2833436 @rwdaigle Draft of vulcan article
authored
8
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
9 ## Problem
2833436 @rwdaigle Draft of vulcan article
authored
10
5f7d6be @rwdaigle Small edits
authored
11 It's common for an application to shell out to a local executable to perform some computationally intense work where compiled libraries are more performant or robust. Examples include [Ghostscript](http://www.ghostscript.com/), which your application might use to manipulate PDF or PostScript files and [ImageMagick](http://www.imagemagick.org/script/index.php) for resizing and cropping uploaded images.
12
13 Many deployment services attempt to fill this need by providing a set of common binaries bundled into every environment. This is a fragile approach that boxes you in to out-dated or incompatible library versions.
2833436 @rwdaigle Draft of vulcan article
authored
14
5f7d6be @rwdaigle Small edits
authored
15 Having the ability to install system-wide binaries in your deployment environment is also poor solution. It merely shifts the burden of dependency management from the service provider to you, the application developer.
2833436 @rwdaigle Draft of vulcan article
authored
16
5f7d6be @rwdaigle Small edits
authored
17 [Twelve-factor](http://www.12factor.net/) is firm in [its stance on system-level dependencies](http://www.12factor.net/dependencies) to
2833436 @rwdaigle Draft of vulcan article
authored
18
5f7d6be @rwdaigle Small edits
authored
19 > Twelve-factor apps also do not rely on the implicit existence of any system tools... While these tools may exist on many or even most systems, there is no guarantee that they will exist on all systems where the app may run in the future, or whether the version found on a future system will be compatible with the app. If the app needs to shell out to a system tool, that tool should be vendored into the app.
2833436 @rwdaigle Draft of vulcan article
authored
20
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
21 ## Solution
2833436 @rwdaigle Draft of vulcan article
authored
22
5f7d6be @rwdaigle Small edits
authored
23 Vendoring a binary dependency requires that the binary be built specifically for the remote environment's operating system and processor architecture. Even with the benefits of virtual machine software this is a non-trivial task.
2833436 @rwdaigle Draft of vulcan article
authored
24
5f7d6be @rwdaigle Small edits
authored
25 A better solution is to use the remote environment of your service provider to compile and build the required dependencies. Such a process would roughly adhere to the following steps:
2833436 @rwdaigle Draft of vulcan article
authored
26
27 1. Specify the required library's source
28 2. Get a remote shell to your production environment
29 3. Download the library source in the remote shell
30 3. Compile the library remotely
31 4. Download the compiled library for use in your application's source-tree
32
33 While this process is not a lengthy one, your eye should spot that steps 2-4 are ripe for automation. That's where Vulcan comes in.
34
35 [Vulcan](https://github.com/heroku/vulcan) is a utility that bundles and uploads a source-tree to a remote environment, runs the necessary compilation commands on the source tree, and downloads the binary -- all in a single command. Vulcan consists of a Ruby-based CLI and Node.js server component and, though built by and for Heroku, is platform-agnostic.
36
37 ## Setup
38
39 <p class="note">
40 These steps assume you have <a href="http://git-scm.com/">git</a> and <a href="http://www.ruby-lang.org/en/">ruby</a> available from the command line and have already signed up for a <a href="https://api.heroku.com/signup">Heroku account</a>. The <a href="http://toolbelt.heroku.com/">Heroku Toolbelt</a> can get you up and running if you're missing any components.
41 </p>
42
43 Installing the Vulcan CLI is simply a matter of installing the `vulcan` gem:
44
45 <pre lang="bash"><code>
46 $ gem install vulcan
47 Please run 'vulcan update' to update your build server.
48 Successfully installed vulcan-0.7.1
49 1 gem installed
50 </code></pre>
51
5f7d6be @rwdaigle Small edits
authored
52 Vulcan's [build-server](https://github.com/heroku/vulcan/tree/master/server) is a simple Node.js app that runs _in the same target environment your application will be deployed_. If your app runs on Heroku, Vulcan can deploy itself to Heroku with `vulcan create appname`.
2833436 @rwdaigle Draft of vulcan article
authored
53
54 <pre lang="bash"><code>
55 $ vulcan create buildserver-you
56 Creating buildserver-you... done, stack is cedar
57 http://buildserver-you.herokuapp.com/ | git@heroku.com:buildserver-you.git
58 Initialized empty Git repository in /private/var/folders/Uz/UzRCgjzkGIi7Iqz9QNi6NUDrHf6/-Tmp-/d20120614-31875-1qjw1d6/.git/
59 Counting objects: 883, done.
60 ...
61
62 -----> Heroku receiving push
63 -----> Node.js app detected
64 ...
65 Dependencies installed
66 -----> Discovering process types
67 Procfile declares types -> web
68 -----> Compiled slug size is 4.1MB
69 -----> Launching... done, v3
70 http://buildserver-you.herokuapp.com deployed to Heroku
71 </code></pre>
72
5f7d6be @rwdaigle Small edits
authored
73 The Vulcan build server is now running on Heroku at `http://buildserver-you.herokuapp.com` as a (free) single-dyno app. It's important to note there's nothing blessed about Vulcan running on Heroku. It's just a normal application running in user-space, giving you all the [visibility](#visibility) and management tools you're used to.
2833436 @rwdaigle Draft of vulcan article
authored
74
75 <p class="note" markdown="1">
5f7d6be @rwdaigle Small edits
authored
76 If you've manually deployed the build server to another provider you'll need to set its location so the [CLI knows where to send build tasks](https://github.com/heroku/vulcan/blob/master/lib/vulcan/cli.rb#L40). This is done by setting the `VULCAN_HOST` env var: `$ export VULCAN_HOST=http://myserver.domain.com`..
2833436 @rwdaigle Draft of vulcan article
authored
77 </p>
78
79 ## Build
80
5f7d6be @rwdaigle Small edits
authored
81 The Ghostscript library utilized in the [processing PDF files](https://devcenter.heroku.com/articles/processing-pdfs-ruby-mongo) tutorial on the [Heroku Dev Center](http://devcenter.heroku.com) makes for a good example of building a binary application dependency.
2833436 @rwdaigle Draft of vulcan article
authored
82
83 ### Download source
84
85 First, download and expand the Linux x86 64-bit source for the [Ghostscript project](http://www.ghostscript.com/). At the time of this writing it is located at [http://downloads.ghostscript.com/public/binaries/ghostscript-9.05-linux-x86_64.tgz](http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz).
86
87 <p class="note" markdown="1">
88 For the purpose of this article Heroku will be assumed to be the target environment. [Heroku's dynos](https://devcenter.heroku.com/articles/dynos) run on 64-bit Linux kernel. If your target environment differs you will need to select the correct source distribution.
89 </p>
90
91 <pre lang="bash"><code>
92 $ wget http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz
93 $ tar -xvzf ghostscript-9.05.tar.gz
94 x ghostscript-9.05/
95 x ghostscript-9.05/base/
96 x ghostscript-9.05/base/szlibxx.h
97 ...
98 </code></pre>
99
100 You now have a Ghostscript source directory at `./ghostscript-9.05`.
101
102 ### Remote compilation
103
5f7d6be @rwdaigle Small edits
authored
104 Next, use the Vulcan CLI to initiate a build task on the build server with `vulcan build`. This will send the source to the target environment for compilation. The only required argument is `-s`, the location of the Ghostscript source directory. The `-v` flag (verbose) will show the output from the compilation process and is recommended as, depending on the library, compilation can take some time and it's useful to see its progress.
2833436 @rwdaigle Draft of vulcan article
authored
105
106 <pre lang="bash"><code>
107 $ vulcan build -v -s ./ghostscript-9.05
108 Packaging local directory... done
109 Uploading source package... done
110 Building with: ./configure --prefix /app/vendor/ghostscript-9 &amp;&amp; make install
111 checking for gcc... gcc
112 checking whether the C compiler works... yes
113 ...
114 >> Downloading build artifacts to: /tmp/ghostscript-9.tgz
115 (available at http://buildserver-you.herokuapp.com/output/45380fa6-e02a-479b-a7af-d9afb089b81f)
116 </code></pre>
117
118 On completion of the build process the resulting binaries are packages and downloaded for you. In this example the binary package can be found locally at `/tmp/ghostscript-9.tgz`. While the binary package is also available on your build server (here at `http://buildserver-you.herokuapp.com/output/45380fa6-e02a-479b-a7af-d9afb089b81f`) this is a temporary location. Due to the ephemeral nature of cloud filesystems the file is likely not to exist at that location for an extended period of time and should not be relied on.
119
120 Keep in mind, while the result are the library binaries they will not be executable on your local machine due to the difference in processor architecture of your local and target environments.
121
122 <p class="warning" markdown="1">
5f7d6be @rwdaigle Small edits
authored
123 Although the [Vulcan source](https://github.com/heroku/vulcan) indicates the ability to fetch source packages directly from a URL (i.e. `$ vulcan build -s http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz` this tends to result in build errors. At this time the most reliable method is to manually download and expand the source before passing to vulcan.
2833436 @rwdaigle Draft of vulcan article
authored
124 </p>
125
126 ## Customization
127
128 Looking at the output from the Ghostscript example build you can see that a sensible build command is chosen for you: `./configure --prefix /app/vendor/ghostscript-9 && make install`. If you need to specify a non-default build command you can do so with the `-c` flag. Here is an example adding the `--without-ssl` configure option when building `wget`.
129
130 <pre lang="bash"><code>
131 $ vulcan build -v -s ./wget-1.13 -c "./configure --prefix /app/vendor/wget-1.13 --without-ssl &amp;&amp; make install" -p /app/vendor/wget-1.13
132 Packaging local directory... done
133 Uploading source package... done
134 Building with: ./configure --without-ssl &amp;&amp; make install
135 configure: configuring for GNU Wget 1.13
136 ...
137 >> Downloading build artifacts to: /tmp/wget-1.tgz
138 (available at http://buildserver-you.herokuapp.com/output/66ba3e2b-77ef-4409-acc2-fca70650c318)
139 </code></pre>
140
141 The `-p` (prefix) flag is also used here to tell Vulcan where to look on the build server for the compiled artifacts (`/app/vendor/wget-1.13`). To avoid ambiguities it's best to specify this value and to set it to the same value as the `--prefix` flag passed to `./configure`.
142
143 ## Visibility
144
145 There are several utilities available to you to introspect the remote compilation process.
146
147 ### Logging
148
5f7d6be @rwdaigle Small edits
authored
149 Outside of using the `-v` flag to force see the build output you can also use the logs of the Vulcan build server to gain better visibility into the process. Since the Vulcan build server is just a Node.js app running in your target environment this is a trivial task. On Heroku use `heroku logs` and the `-a` flag with the name of your build server app.
2833436 @rwdaigle Draft of vulcan article
authored
150
151 <pre lang="bash"><code>
152 $ heroku logs -t -a buildserver-you
153 2012-07-04T15:29:20+00:00 app[web.1]: [7f3a7510-400a-44d4-9132-66a2e6c878a5] spawning build
154 2012-07-04T15:29:20+00:00 app[web.1]: valid socket
155 2012-07-04T15:29:21+00:00 heroku[run.1]: Awaiting client
156 2012-07-04T15:29:21+00:00 heroku[run.1]: Starting process with command `bin/make "7f3a7510-400a-44d4-9132-66a2e6c878a5"`
157 2012-07-04T15:29:22+00:00 heroku[run.1]: State changed from starting to up
158 2012-07-04T15:30:10+00:00 heroku[run.1]: Process exited with status 1
159 2012-07-04T15:30:10+00:00 heroku[run.1]: State changed from up to complete
160 </code></pre>
161
162 By default Vulcan will spawn a [on-off dyno](https://devcenter.heroku.com/articles/oneoff-admin-ps) to perform the build command. This is evident by the `run.1` dyno in the log output. In some circumstances this can result in a billable event if your web and one-off dyno usage combined [exceeds 750 hours for any one month](https://devcenter.heroku.com/articles/usage-and-billing).
163
164 If this small overage is meaningful to you you can sacrifice compilation concurrency and force Vulcan to execute the compilation command in-process with the `SPAWN_ENV` [config var](https://devcenter.heroku.com/articles/config-vars).
165
166 <pre lang="bash"><code>
167 $ heroku config:add SPAWN_ENV=local -a buildserver-you
168 Setting config vars and restarting buildserver-you... done, v11
169 SPAWN_ENV: local
170 </code></pre>
171
172 The build command will then execute within the web process:
173
174 <pre lang="bash"><code>
175 $ heroku logs -t -a buildserver-you
176 2012-07-04T15:35:43+00:00 app[web.1]: [bbeab2b8-3941-4d41-94af-24c4f0fa65c0] spawning build
177 2012-07-04T15:36:33+00:00 app[web.1]: 10.125.41.68 - - [Wed, 04 Jul 2012 15:36:33 GMT] "GET /output/bbeab2b8-3941-4d41-94af-24c4f0fa65c0 HTTP/1.1" 200 - "-" "Ruby"
178 2012-07-04T15:36:33+00:00 heroku[router]: GET buildserver-you.herokuapp.com/output/bbeab2b8-3941-4d41-94af-24c4f0fa65c0 dyno=web.1 queue=0 wait=0ms service=35ms status=200 bytes=75
179 </code></pre>
180
181 While this fails to adhere to the [background job](https://devcenter.heroku.com/articles/background-jobs-queueing) pattern it may be acceptable for your use-case.
182
183 ### Remote shell
184
185 If a build fails it is often useful to be able to view the failed artefacts. Since Vulcan performs its work in temporary directories their contents are cleaned up after each build. However, a remote shell can be used to manually invoke the build command and navigate the build results.
186
187 At the start of every build request Vulcan outputs log statements resembling the following:
188
189 <pre lang="bash"><code>
190 2012-07-04T18:57:10+00:00 app[web.1]: [6869e68a-492d-4a7e-8b27-64352811d7dc] saving to couchdb
191 2012-07-04T18:57:10+00:00 app[web.1]: [6869e68a-492d-4a7e-8b27-64352811d7dc] saving attachment - [id:6869e68a-492d-4a7e-8b27-64352811d7dc rev:1-722a96f6734a3511efd73b7cfb9a2aed]
192 </code></pre>
193
5f7d6be @rwdaigle Small edits
authored
194 The attachment id, here `6869e68a-492d-4a7e-8b27-64352811d7dc`, is all that's needed to manually invoke the build yourself. Establish a remote shell to the Vulcan build server environment. On Heroku you can use `heroku run bash`:
2833436 @rwdaigle Draft of vulcan article
authored
195
196 <pre lang="bash"><code>
197 $ heroku run bash -a buildserver-you
198 ~ $
199 </code></pre>
200
201 Then invoke the `bin/make` command with the attachment id. You will see the output of the build process and can then browse the output directory yourself.
202
203 <pre lang="bash"><code>
204 $ bin/make "6869e68a-492d-4a7e-8b27-64352811d7dc"
205 configure: configuring for GNU Wget 1.13
206 checking for a BSD-compatible install... /usr/bin/install -c
207 checking whether build environment is sane... yes
208 ...
209 $ cd /app/vendor/wget-1.13
210 </code></pre>
211
212 ## Vendoring
213
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
214 ## Updates
2833436 @rwdaigle Draft of vulcan article
authored
215
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
216 Updating Vulcan is quite simple. Update the CLI using ruby gems:
2833436 @rwdaigle Draft of vulcan article
authored
217
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
218 <pre lang="bash"><code>
219 $ gem install vulcan
220 Please run 'vulcan update' to update your build server.
221 Successfully installed vulcan-0.8.0
222 1 gem installed
223 </code></pre>
2833436 @rwdaigle Draft of vulcan article
authored
224
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
225 And use the `vulcan update` command to update the build server.
2833436 @rwdaigle Draft of vulcan article
authored
226
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
227 <p class="warning" markdown="1">
228 Be aware that updating the build server while active compilations are running will cause them to be aborted.
229 </p>
2833436 @rwdaigle Draft of vulcan article
authored
230
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
231 <pre lang="bash"><code>
232 $ vulcan update
233 Initialized empty Git repository in /private/var/folders/tt/7f38d4b14qq5xglpj3yl0smr0000gn/T/d20120704-53816-m4n0rn/.git/
234 Counting objects: 883, done.
235 ...
2833436 @rwdaigle Draft of vulcan article
authored
236
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
237 -----> Heroku receiving push
238 -----> Node.js app detected
239 -----> Resolving engine versions
240 Using Node.js version: 0.6.18
241 Using npm version: 1.1.4
242 ...
243 -----> Launching... done, v15
244 http://buildserver-you.herokuapp.com deployed to Heroku
2833436 @rwdaigle Draft of vulcan article
authored
245
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
246 To git@heroku.com:buildserver-you.git
247 + a5f27be...a934704 master -> master (forced update)
248 </code></pre>
249
250 ## Troubleshooting
251
252 ### Invalid secret
2833436 @rwdaigle Draft of vulcan article
authored
253
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
254 If working across multiple development environments or some other non-default workflow you may see the following build server error logged when attempting to invoke a build:
255
256 <pre><code>
257 2012-07-04T17:39:47+00:00 app[web.1]: [672b5df6-ad8c-49ed-9831-515207e2dc4f] ERROR: invalid secret
2833436 @rwdaigle Draft of vulcan article
authored
258 2012-07-04T17:39:47+00:00 app[web.1]: invalid secret
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
259 </code></pre>
260
261 This occurs when the CLI secret hash, created when the build server was created with `vulcan create`, either doesn't exist or doesn't match the server-side secret. The most common cause is that the `~/.vulcan` configuration file doesn't exist in your environment. You can create it with the following contents:
262
263 <h5 class="file">~/.vulcan</h5>
264 <pre lang="yaml"><code>
2833436 @rwdaigle Draft of vulcan article
authored
265 ---
266 :app: buildserver-you
267 :host: buildserver-you.herokuapp.com
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
268 :secret: reallylonghash12df
269 </code></pre>
2833436 @rwdaigle Draft of vulcan article
authored
270
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
271 If you don't have access to your original `.vulcan` file you can find your secret on Heroku using `heroku config`:
272
273 <pre lang="bash"><code>
274 $ heroku config:get SECRET -a buildserver-you
275 reallylonghash12df
276 </code></pre>
277
278 <p class="note" markdown="1">
279 Copy your `~/.vulcan` file to each development machine from which you wish to invoke builds.
280 </p>
2833436 @rwdaigle Draft of vulcan article
authored
281
5f7d6be @rwdaigle Small edits
authored
282 ## Heroku binaries
283
284 During the course of writing this article the following binaries were compiled for use on Heroku.
285
286 <p class="note" markdown="1">
287 If you'd like to list a Heroku binary here, please [send a pull request](https://github.com/rwdaigle/ryandaigle.com/blob/master/content/pages/a/using-vulcan-to-build-binary-dependencies-on-heroku.mdown).
288 </p>
289
290 <table>
291 <tr>
292 <th>Library</th>
293 <th>Build command</th>
294 <th>Binary</th>
295 <th>Contributor</th>
296 </tr>
297 <tr>
298 <td><a href="http://ftp.gnu.org/gnu/wget/wget-1.13.tar.gz">GNU Wget v1.13</a></td>
299 <td><code>vulcan build -v -s ./wget-1.13 -c "./configure --prefix /app/vendor/wget-1.13 --without-ssl &amp;&amp; make install" -p /app/vendor/wget-1.13</code></td>
300 <td><a href="http://cl.ly/1B0n121X1T200g3u2a1k/wget-1.tgz">download</a></td>
301 <td><a href="https://twitter.com/rwdaigle">@rwdaigle</a></td>
302 </tr>
303 <tr>
304 <td><a href="http://www.imagemagick.org/download/ImageMagick.tar.gz">ImageMagick v6.7.8-1</a></td>
305 <td><code>vulcan build -v -s ./ImageMagick-6.7.8-1</code></td>
306 <td><a href="http://cl.ly/011m0E3w2I360X2s1D2d/ImageMagick-6.7.tgz">download</a></td>
307 <td><a href="https://twitter.com/rwdaigle">@rwdaigle</a></td>
308 </tr>
309 <tr>
310 <td><a href="http://downloads.ghostscript.com/public/ghostscript-9.05.tar.gz">Ghostscript v9.05</a></td>
311 <td><code>vulcan build -v -s ./ghostscript-9.05</code></td>
312 <td><a href="http://cl.ly/192z2W1H2i1e3o0W341V/ghostscript-9.tgz">download</a></td>
313 <td><a href="https://twitter.com/rwdaigle">@rwdaigle</a></td>
314 </tr>
315 </table>
316
2833436 @rwdaigle Draft of vulcan article
authored
317 <!--
318
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
319 ## Notes
320
321 Vendoring/In practice: switch between local dev and remote (using the bin/ env)
322
5f7d6be @rwdaigle Small edits
authored
323 style table, list, block quote
e873d6b @rwdaigle Add troubleshooting and other supporting sections
authored
324
2833436 @rwdaigle Draft of vulcan article
authored
325 ## Dependencies
326
327 Unlike Ghostscript, many libraries themselves have dependencies that are not available in your target environment. `wget` is a useful utility that requires the [GNU transport layer security library (gnutls)](http://www.gnu.org/software/gnutls/download.html) for SSL connections. To compile wget first compile gnutls.
328
329 <p class="note" markdown="1">
330 Recent versions of gnutls are [distributed](http://ftp.gnu.org/gnu/gnutls/) with [XZ](http://tukaani.org/xz/) file compression. It will be left as an excercise for the reader to download and decompress the gnutls source.
331 </p>
332
333 <pre lang="bash"><code>
334 $ vulcan build -v -s ./gnutls-3.0.21
335 Packaging local directory... done
336 Uploading source package... done
9a9a686 @rwdaigle Fix double && parse error
authored
337 Building with: ./configure --prefix /app/vendor/ghostscript-9 &amp;&amp; make install
2833436 @rwdaigle Draft of vulcan article
authored
338 checking for gcc... gcc
339 checking whether the C compiler works... yes
340 ...
341 </code></pre>
342
343 xz compressions: http://tukaani.org/xz/
344
345 wget as example, requires gnutils
346
347 -->
Something went wrong with that request. Please try again.