forked from cloudfoundry/docs-buildpacks
/
upgrading_dependency_versions.html.md.erb
238 lines (155 loc) · 15.3 KB
/
upgrading_dependency_versions.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
---
title: Upgrading Dependency Versions
owner: Buildpacks
---
This topic describes how to upgrade a dependency version in a custom buildpack. These procedures enable Cloud Foundry (CF) operators to maintain custom buildpacks that contain dependencies outside of the dependencies in the CF system buildpacks.
##<a id="buildpacks-team"></a>Cloud Foundry Buildpacks Team Process
<p class="note"><strong>Note</strong>: The procedures in this topic refer to the tools used by the CF buildpacks team, but they do not require the specific tools listed below. You can use any continuous integration (CI) system and workflow management tool to update dependencies in custom buildpacks.</p>
The CF buildpacks team uses the following tools to update dependencies:
* A [Concourse](https://concourse.ci) deployment of the [buildpacks-ci](https://github.com/cloudfoundry/buildpacks-ci) pipelines
* The [public-buildpacks-ci-robots](https://github.com/cloudfoundry/public-buildpacks-ci-robots) GitHub repository
* [Pivotal Tracker](https://www.pivotaltracker.com) for workflow management
When the [New Releases](https://buildpacks.ci.cf-app.com/pipelines/notifications/jobs/New%20Releases) job in the [notifications pipeline](https://buildpacks.ci.cf-app.com/pipelines/notifications)
detects a new version of a tracked dependency in a buildpack, it creates a [Tracker](https://www.pivotaltracker.com) story about building and including the new version of the dependency in the buildpack manifests. It also posts a message as the `dependency-notifier` to the [#buildpacks channel](https://cloudfoundry.slack.com/messages/buildpacks/) in the Cloud Foundry Slack channel.
##<a id="building"></a>Build the Binaries
For all dependencies, you must build the binary from source or acquire the binary as a tarball from a trusted source. For most dependencies, the CF buildpacks team builds the binaries from source.
<p class="note"><strong>Note</strong>: The steps below assume you are using a Concourse deployment of the <code>buildpacks-ci</code> pipelines and Pivotal Tracker.</p>
To build the binary for a dependency, perform the following steps:
1. Navigate to the `public-buildpacks-ci-robots` directory and verify no uncommitted changes exist.
<pre class='terminal'>
$ cd ~/workspace/public-buildpacks-ci-robots
$ git status
</pre>
1. Run the `git pull` command in the directory to ensure it contains most recent version of the contents.
<pre class="terminal">
$ git pull -r
</pre>
1. Navigate to the `binary-builds` directory.
<pre class="terminal">
$ cd binary-builds
</pre>
1. Locate the YAML file for the buildpack you want to build a binary for. The directory contains YAML files for all the packages and dependencies tracked by the CF buildpacks team. Each YAML file correlates to the build queue for one dependency or package, and uses the naming format `DEPENDENCY-NAME.yml`. For example, the YAML file tracking the build queue for Ruby is named `ruby-builds.yml` and contains the following contents:
```yaml
---
ruby: []
```
1. Different buildpacks use different signatures for verification. Determine which signature your buildpack requires by consulting the list in the [buildpacks](#buildpacks) section of this topic and run follow the instructions to locate the SHA256, MD5, or GPG signature for the binary:
* For the SHA256 of a file, run `shasum -a 256 FILE-NAME`.
* For the MD5 of a file, run `md5 FILE-NAME`.
* For the GPG signature (for Nginx), see the [Nginx Downloads](http://nginx.org/en/download.html) page.
1. Add the version and verification for the new binary to the YAML file as attributes of an element under the dependency name. For example, to build the Ruby 2.3.0 binary verified with SHA256, add the following to the YAML file:
```yaml
---
ruby:
- version: 2.3.0
sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
```
<p class="note"><strong>Note</strong>: Do not preface the version number with the name of the binary or language. For example, specify <code>2.3.0</code> for <code>version</code> instead of <code>ruby-2.3.0</code>.</p>
You can enqueue builds for multiple versions at the same time. For example, to build both the Ruby 2.3.0 binary and the Ruby 2.3.1 binary, add the following to the YAML file:
```yaml
---
ruby:
- version: 2.3.0
sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
- version: 2.3.1
sha256: b87c738cb2032bf4920fef8e3864dc5cf8eae9d89d8d523ce0236945c5797dcd
```
1. Use the `git add` command to stage your changes:
<pre class="terminal">$ git add .</pre>
1. Use the `git commit -m "YOUR-COMMIT-MESSAGE [#STORY-NUMBER]"` command to commit your changes using the Tracker story number. Replace `YOUR-COMMIT-MESSAGE` with your commit message, and `STORY-NUMBER` with the number of your Tracker story.
<pre class="terminal">$ git commit -m "make that change [#1234567890]"</pre>
1. Run `git push` to push your changes to the remote origin.
<pre class="terminal">$ git push</pre>
1. Pushing your changes triggers the binary building process, which you can monitor at the [binary-builder pipeline](https://buildpacks.ci.cf-app.com/pipelines/binary-builder)
of your own `buildpacks-ci` Concourse deployment. When the build completes, it adds a link to the Concourse build run to the Tracker story specified in the commit message for the new release.
<p class="note"><strong>Note</strong>: Binary builds are executed by the Cloud Foundry <a href="https://github.com/cloudfoundry/binary-builder">Binary Builder</a> and the <a href="https://github.com/cloudfoundry/buildpacks-ci/blob/master/pipelines/dependency-builds.yml.erb">dependency-builds</a> pipeline.</p>
##<a id="manifest-updates"></a>Update Buildpack Manifests
After you build the binary for a dependency that you can access and download from a URL, follow the instructions below to add the dependency version to the buildpack manifest.
<p class="note"><strong>Note</strong>: The steps below assume you are using a Concourse deployment of the <code>buildpacks-ci</code> pipelines and Pivotal Tracker.</p>
1. Navigate to the directory of the buildpack for which you want to update dependencies and run `git checkout develop` to check out the `develop` branch.
<pre class="terminal">
$ cd ~/workspace/ruby-buildpack
$ git checkout develop
</pre>
1. Edit the `manifest.yml` file for the buildpack to add or remove dependencies.
```yaml
dependencies:
- name: ruby
version: 2.3.0
md5: 535342030a11abeb11497824bf642bf2
uri: https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/ruby/ruby-2.3.0-linux-x64.tgz
cf_stacks:
- cflinuxfs3
```
* Follow the current structure of the manifest. For example, if the manifest includes the two most recent patch versions for each minor version of the language, you should also include the two most recent patch versions for each minor version of the language, such as both `ruby-2.1.9` and `ruby-2.1.8`.
* Copy the `uri` and the `md5` from the `build-BINARY-NAME` job that ran in the Concourse dependency-builds pipeline and add them to the manifest.
<p class="note"><strong>Note</strong>: In the PHP buildpack, you may see a <code>modules</code> line for each PHP dependency in the manifest. Do not include this <code>modules</code> line in your new PHP dependency entry. The <code>modules</code> line will be added to the manifest by the <code>ensure-manifest-has-modules</code> Concourse job in the <code>php-buildpack</code> when you commit and push your changes. You can see this in the output logs of the <code>build-out</code> task.</p>
1. Replace any other mentions of the old version number in the buildpack repository with the new version number. The CF buildpack team uses [Ag](https://github.com/ggreer/the_silver_searcher) for text searching.
<pre class="terminal">$ ag OLD-VERSION</pre>
1. Run the following command to package and upload the buildpack, set up the org and space for tests in the specified CF deployment, and run the CF buildpack tests.
<pre class="terminal">$ BUNDLE_GEMFILE=cf.Gemfile buildpack-build</pre>
If the command fails, you may need to fix or change the tests, fixtures, or other parts of the buildpack.
1. Once the test suite completely passes, use git commands to stage, commit, and push your changes:
<pre class="terminal">
$ git add .
$ git commit -m "YOUR-MESSAGE[#TRACKER-STORY-ID]"
$ git push
</pre>
1. Monitor the `LANGUAGE-buildpack` pipeline in Concourse. Once the test suite builds, the `specs-lts-develop` job and `specs-edge-develop` job, pass for the buildpack, you can deliver the Tracker story for the new Dependency release. Copy and paste links for the successful test suite builds into the Tracker story.
##<a id='buildpacks'></a>Buildpacks
The following list contains information about the buildpacks maintained by the CF buildpacks team.
###<a id="build-go-buildpack"></a>Go Buildpack
__Go__:
* **Built from**: A tarred binary, `GO-VERSION.linux-amd64.tar.gz`, provided by [Google on the Go Downloads](https://golang.org/dl/) page
* **Verified with**: The MD5 of the tarred binary
* **Example**: [Using the Google Tarred Binary for Go 1.6.2](https://github.com/cloudfoundry/go-buildpack/blob/v1.7.8/manifest.yml#L33-L34)
__Godep__:
* **Built from**: A source code `.tar.gz` file from the [Godep GitHub releases](https://github.com/tools/godep/releases) page
* **Verified with**: The SHA256 of the source
**Example**: [Automated enqueuing of binary build for Godep 72](https://github.com/cloudfoundry/buildpacks-ci/commit/42a361193ea9a4c77a93451228c93bf96f78d50a)
<p class="note"><strong>Note</strong>: The <a href="https://github.com/cloudfoundry/buildpacks-ci">buildpacks-ci</a> <a href="https://github.com/cloudfoundry/buildpacks-ci/blob/master/pipelines/dependency-builds.yml.erb">dependency-builds</a> pipeline automates the process of detecting, uploading, and updating Godep in the manifest.</p>
###<a id="build-nodejs-buildpack"></a>Node.js Buildpack
__Node__:
* **Verified with**: The SHA256 of the `node-vVERSION.tar.gz` file listed on `https://nodejs.org/dist/vVERSION/SHASUMS256.txt` For example, for Node version 4.4.6, the CF buildpacks team verifies with the SHA256 for `node-v4.4.6.tar.gz` on its [SHASUMS256](https://nodejs.org/dist/v4.4.6/SHASUMS256.txt) page.
* **Example**: [Enqueuing binary builds for Node 4.4.5 and 6.2.0](https://github.com/cloudfoundry/buildpacks-ci/commit/1747c697dc29db0180253c5d466804784cb3fcce)
###<a id="build-python-buildpack"></a>Python Buildpack
__Python__:
* **Verified with**: The MD5 of the `Gzipped source tarball`, listed on `https://www.python.org/downloads/release/python-VERSION/`, where `VERSION` has no periods. For example, for Python version `2.7.12`, use the MD5 for the `Gzipped source tarball` on its [downloads](https://www.python.org/downloads/release/python-2712/) page.
* **Example**: [Enqueuing binary build for Python 2.7.12](https://github.com/cloudfoundry/buildpacks-ci/commit/79e4ce891281ce3ec376b00c967465fc8c7383f8)
###<a id='build-java-buildpack'></a>Java Buildpack
__OpenJDK__:
* **Built from**: The tarred [OpenJDK files](https://download.run.pivotal.io/openjdk/trusty/x86_64/index.yml) managed by the CF Java Buildpack team
* **Verified with**: The MD5 of the tarred OpenJDK files
###<a id="build-ruby-buildpack"></a>Ruby Buildpack
__JRuby__:
* **Verified with**: The MD5 of the Source `.tar.gz` file from the [JRuby Downloads](http://jruby.org/download) page
* **Example**: [Enqueuing binary build for JRuby 9.1.2.0](https://github.com/cloudfoundry/buildpacks-ci/commit/1c90967bcb511e7657202a1308e530825e372158)
__Ruby__:
* **Verified with**: The SHA256 of the source from the [Ruby Downloads](https://www.ruby-lang.org/en/downloads/) page
* **Example**: [Enqueuing binary builds for Ruby 2.2.5 and 2.3.1](https://github.com/cloudfoundry/buildpacks-ci/commit/8d06eaa6573146b9771de7f96cebf85e07719d79)
__Bundler__:
* **Verified with**: The SHA256 of the `.gem` file from [Rubygems](https://rubygems.org/gems/bundler)
* **Example**: [Enqueuing binary build for Bundler 1.12.5](https://github.com/cloudfoundry/buildpacks-ci/commit/cd161594a564631dff13e8c101205f922beb5cf8)
###<a id="build-php-buildpack"></a>PHP Buildpack
__PHP__:
* **Verified with**: The SHA256 of the `.tar.gz` file from the [PHP Downloads](http://php.net/downloads.php) page.
* To enqueue builds for PHP, you need to edit a file in the `public-buildpacks-ci-robots` repository. For PHP5 versions, the CF buildpacks team enqueues builds in the `binary-builds/php-builds.yml` file. For PHP7 versions, the CF buildpacks team enqueues builds in the `binary-builds/php7-builds.yml` file.
* **Example**: [Enqueuing binary builds for PHP 7.2.5 and 7.0.30](https://github.com/cloudfoundry/public-buildpacks-ci-robots/commit/3353b2d4343f955fdfb34c04358d92c4045aa4e3)
__Nginx__:
* **Verified with**: The `gpg-rsa-key-id` and `gpg-signature` of the version. The `gpg-rsa-key-id` is the same for each version/build, but the `gpg-signature` will be different. This information is located on the [Nginx Downloads](https://nginx.org/en/download.html) page.
* **Example**: [Enqueuing binary build for Nginx 1.11.0](https://github.com/cloudfoundry/buildpacks-ci/commit/b86a09a46eb071957adaecd3a16d1ebba9cd0973)
__HTTPD__:
* **Verified with**: The MD5 of the `.tar.bz2` file from the [HTTPD Downloads](https://httpd.apache.org/download.cgi) page
* **Example**: [Enqueuing binary build for HTTPD 2.4.20](https://github.com/cloudfoundry/buildpacks-ci/commit/cee826dc978f013840ef6a8193ec6ee75536778b)
__Composer__:
* **Verified with**: The SHA256 of the `composer.phar` file from the [Composer Downloads](https://getcomposer.org/download/) page
* For Composer, there is no build process as the `composer.phar` file is the binary. In the manual process, connect to the appropriate S3 bucket using the correct AWS credentials. Create a new directory with the name of the
composer version, for example `1.0.2`, and put the appropriate `composer.phar` file into that directory. For Composer `v1.0.2`, connect and create the `php/binaries/trusty/composer/1.0.2` directory. Then place the `composer.phar` file into that directory so the binary is available at `php/binaries/trusty/composer/1.0.2/composer.phar`.
<p class="note"><strong>Note</strong>: The <a href="https://github.com/cloudfoundry/buildpacks-ci">buildpacks-ci</a> <a href="https://github.com/cloudfoundry/buildpacks-ci/blob/master/pipelines/dependency-builds.yml.erb">dependency-builds</a> pipeline automates the process of detecting, uploading, and updating Composer in the manifest.</p>
* **Example**: [Automated enqueuing of binary build for Composer 1.1.2](https://github.com/cloudfoundry/buildpacks-ci/commit/d51abbdb919ca51df86628b37366c55311b05f38)
###<a id="build-staticfile-buildpack"></a>Staticfile Buildpack
__Nginx__:
* **Verified with**: The `gpg-rsa-key-id` and `gpg-signature` of the version. The `gpg-rsa-key-id` is the same for each version/build, but the `gpg-signature` will be different. This information is located on the [Nginx Downloads](https://nginx.org/en/download.html) page.
* **Example**: [Enqueuing binary build for Nginx 1.11.0](https://github.com/cloudfoundry/buildpacks-ci/commit/b86a09a46eb071957adaecd3a16d1ebba9cd0973)
###<a id="build-binary-buildpack"></a>Binary Buildpack
The Binary buildpack has no dependencies.