Skip to content

Commit

Permalink
THRIFT-4579: Move up to Ubuntu Bionic for CI builds
Browse files Browse the repository at this point in the history
make dlang library compatible with openssl-1.1 for Ubuntu Bionic
Requires an upstream deimos update to be compatible.
  • Loading branch information
jeking3 committed Jun 16, 2018
1 parent b5471f6 commit f5f430d
Show file tree
Hide file tree
Showing 15 changed files with 386 additions and 84 deletions.
13 changes: 5 additions & 8 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ env:
- SCRIPT="cmake.sh"
- BUILD_ARG=""
- BUILD_ENV="-e CC=gcc -e CXX=g++ -e THRIFT_CROSSTEST_CONCURRENCY=4"
- DISTRO=ubuntu-artful
- DISTRO=ubuntu-bionic
- BUILD_LIBS="CPP C_GLIB HASKELL JAVA PYTHON TESTING TUTORIALS" # only meaningful for CMake builds
- TRAVIS_BUILD_STAGE=test
# DOCKER_REPO (this works for all builds as a source for docker images - you can override for fork builds in your Travis settings)
Expand All @@ -57,13 +57,13 @@ jobs:
- stage: docker
script: true
env:
- JOB="Docker Build ubuntu-xenial 16.04"
- JOB="Docker Build ubuntu-xenial 16.04 LTS"
- DISTRO=ubuntu-xenial
- TRAVIS_BUILD_STAGE=docker
- script: true
env:
- JOB="Docker Build ubuntu-artful 17.10"
- DISTRO=ubuntu-artful
- JOB="Docker Build ubuntu-bionic 18.04 LTS"
- DISTRO=ubuntu-bionic
- TRAVIS_BUILD_STAGE=docker

# ========================= stage: thrift =======================
Expand Down Expand Up @@ -144,8 +144,7 @@ jobs:
# TODO: Remove them once migrated to CMake
- script: build/docker/run.sh
env:
- JOB="Autotools (Ubuntu Artful)"
- DISTRO=ubuntu-artful
- JOB="Autotools (Ubuntu Bionic)"
- SCRIPT="autotools.sh"

- script: build/docker/run.sh
Expand All @@ -158,13 +157,11 @@ jobs:
- script: build/docker/run.sh
env:
- JOB="make dist"
- DISTRO=ubuntu-artful
- SCRIPT="make-dist.sh"

- script: build/docker/run.sh
env:
- JOB="Debian Packages"
- DISTRO=ubuntu-artful
- SCRIPT="dpkg.sh"

### ------------------------- phase: osx -------------------------
Expand Down
130 changes: 81 additions & 49 deletions build/docker/README.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,65 @@
# Docker Integration #

Due to the large number of language requirements to build Apache Thrift, docker containers are used to build and test the project on a variety of platforms to provide maximum test coverage.
Due to the large number of languages supported by Apache Thrift,
docker containers are used to build and test the project on a
variety of platforms to provide maximum test coverage.

## Appveyor Integration ##

At this time the Appveyor scripts do not use docker containers.
Once Microsoft supports Visual Studio Build Tools running inside
nano containers (instead of Core, which is huge) then we will
consider using containers for the Windows builds as well.

## Travis CI Integration ##

The Travis CI scripts use the following environment variables and logic to determine their behavior.
The Travis CI scripts use the following environment variables and
logic to determine their behavior:

### Environment Variables ###

| Variable | Default | Usage |
| -------- | ----- | ------- |
| `DISTRO` | `ubuntu-xenial` | Set by various build jobs in `.travis.yml` to run builds in different containers. Not intended to be set externally.|
| `DISTRO` | `ubuntu-bionic` | Set by various build jobs in `.travis.yml` to run builds in different containers. Not intended to be set externally.|
| `DOCKER_REPO` | `thrift/thrift-build` | The name of the Docker Hub repository to obtain and store docker images. |
| `DOCKER_USER` | `<none>` | The Docker Hub account name containing the repository. |
| `DOCKER_PASS` | `<none>` | The Docker Hub account password to use when pushing new tags. |

For example, the default docker image that is used in builds if no overrides are specified would be: `thrift/thrift-build:ubuntu-xenial`
For example, the default docker image that is used in builds if no overrides are specified would be: `thrift/thrift-build:ubuntu-bionic`

### Forks ###

If you have forked the Apache Thrift repository and you would like to use your own Docker Hub account to store thrift build images, you can use the Travis CI web interface to set the `DOCKER_USER`, `DOCKER_PASS`, and `DOCKER_REPO` variables in a secure manner. Your fork builds will then pull, push, and tag the docker images in your account.
If you have forked the Apache Thrift repository and you would like
to use your own Docker Hub account to store thrift build images,
you can use the Travis CI web interface to set the `DOCKER_USER`,
`DOCKER_PASS`, and `DOCKER_REPO` variables in a secure manner.
Your fork builds will then pull, push, and tag the docker images
in your account.

### Logic ###

The Travis CI build runs in two phases - first the docker images are rebuilt for each of the three supported containers if they do not match the Dockerfile that was used to build the most recent tag. If a `DOCKER_PASS` environment variable is specified, the docker stage builds will attempt to log into Docker Hub and push the resulting tags.
The Travis CI build runs in two phases - first the docker images are rebuilt
for each of the supported containers if they do not match the Dockerfile that
was used to build the most recent tag. If a `DOCKER_PASS` environment
variable is specified, the docker stage builds will attempt to log into
Docker Hub and push the resulting tags.

## Supported Containers ##

The Travis CI (continuous integration) builds use the Ubuntu Trusty, Xenial, and Artful images to maximize language level coverage.
The Travis CI (continuous integration) builds use the Ubuntu Bionic
(18.04 LTS) and Xenial (16.04 LTS) images to maximize language level
coverage.

### Ubuntu ###

* xenial (stable)
* artful (current)
* bionic (stable, current)
* artful (previous stable)
* xenial (legacy)

## Unsupported Containers ##

These containers may be in various states, and may not build everything.
They can be found in the `old/` subdirectory.

### CentOS ###
* 7.3
Expand All @@ -50,7 +73,9 @@ These containers may be in various states, and may not build everything.

## Building like Travis CI does, locally ##

We recommend you build locally the same way Travis CI does, so that when you submit your pull request you will run into fewer surprises. To make it a little easier, put the following into your `~/.bash_aliases` file:
We recommend you build locally the same way Travis CI does, so that when you
submit your pull request you will run into fewer surprises. To make it a
little easier, put the following into your `~/.bash_aliases` file:

# Kill all running containers.
alias dockerkillall='docker kill $(docker ps -q)'
Expand All @@ -76,35 +101,42 @@ We recommend you build locally the same way Travis CI does, so that when you sub
docker run -v $(pwd):/thrift/src -it $1 /bin/bash
}

To pull down the current image being used to build (the same way Travis CI does it) - if it is out of date in any way it will build a new one for you:
Then, to pull down the current image being used to build (the same way
Travis CI does it) - if it is out of date in any way it will build a
new one for you:

thrift$ DOCKER_REPO=thrift/thrift-build DISTRO=ubuntu-xenial build/docker/refresh.sh
thrift$ DOCKER_REPO=thrift/thrift-build DISTRO=ubuntu-bionic build/docker/refresh.sh

To run all unit tests (just like Travis CI):
To run all unit tests (just like Travis CI does):

thrift$ dockerrun ubuntu-xenial
thrift$ dockerrun ubuntu-bionic
root@8caf56b0ce7b:/thrift/src# build/docker/scripts/autotools.sh

To run the cross tests (just like Travis CI):
To run the cross tests (just like Travis CI does):

thrift$ dockerrun ubuntu-xenial
thrift$ dockerrun ubuntu-bionic
root@8caf56b0ce7b:/thrift/src# build/docker/scripts/cross-test.sh

When you are done, you want to clean up occasionally so that docker isn't using lots of extra disk space:

thrift$ dockerclean

You need to run the docker commands from the root of the git repository for them to work.
You need to run the docker commands from the root of the local clone of the
thrift git repository for them to work.

When you are done in the root docker shell you can `exit` to go back to your user host shell. Once the unit tests and cross test passes locally, then submit he changes, and squash the pull request to one commit to make it easier to merge. Thanks. I am going to update the docker README.md with this information so others can leverage it too. Now you are building like Travis CI does!
When you are done in the root docker shell you can `exit` to go back to
your user host shell. Once the unit tests and cross test passes locally,
submit the changes, and if desired squash the pull request to one commit
to make it easier to merge (the committers can squash at commit time now
that GitHub is the master repository). Now you are building like Travis CI does!

## Raw Commands for Building with Docker ##

If you do not want to use the same scripts Travis CI does, you can do it manually:

Build the image:

thrift$ docker build -t thrift build/docker/ubuntu-xenial
thrift$ docker build -t thrift build/docker/ubuntu-bionic

Open a command prompt in the image:

Expand All @@ -114,53 +146,53 @@ Open a command prompt in the image:

Last updated: October 1, 2017

| Tool | ubuntu-xenial | ubuntu-artful | Notes |
| Tool | ubuntu-xenial | ubuntu-bionic | Notes |
| :-------- | :------------ | :------------ | :---- |
| ant | 1.9.6 | 1.9.9 | |
| ant | 1.9.6 | 1.10.3 | |
| autoconf | 2.69 | 2.69 | |
| automake | 1.15 | 1.15 | |
| automake | 1.15 | 1.15.1 | |
| bison | 3.0.4 | 3.0.4 | |
| boost | 1.58.0 | 1.63.0 | artful: stock boost 1.62.0 has problems running unit tests |
| cmake | 3.5.1 | 3.9.1 | |
| cppcheck | 1.72 | 1.80 | |
| flex | 2.6.0 | 2.6.1 | |
| glibc | 2.23 | 2.26 | |
| libevent | 2.0.21 | 2.1 | |
| libstdc++ | 5.4.0 | 7.2.0 | |
| boost | 1.58.0 | 1.65.1 | |
| cmake | 3.5.1 | 3.10.2 | |
| cppcheck | 1.72 | 1.82 | |
| flex | 2.6.0 | 2.6.4 | |
| libc6 | 2.23 | 2.27 | glibc |
| libevent | 2.0.21 | 2.1.8 | |
| libstdc++ | 5.4.0 | 7.3.0 | |
| make | 4.1 | 4.1 | |
| openssl | 1.0.2g | 1.0.2g | |
| qt5 | 5.5.1 | 5.9.1 | |
| openssl | 1.0.2g | 1.1.0g | |
| qt5 | 5.5.1 | 5.9.5 | |

## Compiler/Language Versions per Dockerfile ##

| Language | ubuntu-xenial | ubuntu-artful | Notes |
| Language | ubuntu-xenial | ubuntu-bionic | Notes |
| :-------- | :------------ | :------------ | :---- |
| as of | Mar 06, 2018 | Mar 19, 2018 | |
| as of | Mar 06, 2018 | Jun 6, 2018 | |
| as3 | | | Not in CI |
| C++ gcc | 5.4.0 | 7.2.0 | |
| C++ clang | 3.8 | 4.0 | |
| C++ gcc | 5.4.0 | 7.3.0 | |
| C++ clang | 3.8 | 6.0 | |
| C# (mono) | 4.2.1.0 | 4.6.2.7 | |
| c_glib | 2.48.2 | 2.54.0 | |
| cl (sbcl) | | 1.4.5 | |
| c_glib | 2.48.2 | 2.56.0 | |
| cl (sbcl) | | 1.4.8 | |
| cocoa | | | Not in CI |
| d | 2.075.1 | 2.080.0 | |
| dart | 1.22.1 | 1.24.3 | |
| delphi | | | Not in CI |
| dotnet | 2.1.4 | 2.1.4 | v2.1.4 SDK uses v2.0.5 Runtime |
| erlang | 18.3 | 20.0.4 | |
| go | 1.7.6 | 1.10 | |
| dotnet | 2.1.4 | 2.1.300 | |
| erlang | 18.3 | 20.2.2 | |
| go | 1.7.6 | 1.10.2 | |
| haskell | 7.10.3 | 8.0.2 | |
| haxe | 3.2.1 | 3.4.4 | THRIFT-4352: avoid 3.4.2 |
| java | 1.8.0_151 | 1.8.0_151 | |
| java | 1.8.0_151 | 1.8.0_171 | |
| js | | | Unsure how to look for version info? |
| lua | 5.2.4 | 5.2.4 | Lua 5.3: see THRIFT-4386 |
| nodejs | 6.13.0 | 8.9.4 | |
| ocaml | | 4.04.0 | THRIFT-4517: ocaml 4.02.3 on xenial appears broken |
| perl | 5.22.1 | 5.26.0 | |
| php | 7.0.22 | 7.1.11 | |
| python | 2.7.12 | 2.7.14 | |
| python3 | 3.5.2 | 3.6.3 | |
| ruby | 2.3.1p112 | 2.3.3p222 | |
| rust | 1.17.0 | 1.21.0 | |
| nodejs | 6.13.0 | 8.11.2 | |
| ocaml | | 4.05.0 | THRIFT-4517: ocaml 4.02.3 on xenial appears broken |
| perl | 5.22.1 | 5.26.1 | |
| php | 7.0.22 | 7.2.5 | |
| python | 2.7.12 | 2.7.15rc1 | |
| python3 | 3.5.2 | 3.6.5 | |
| ruby | 2.3.1p112 | 2.5.1p57 | |
| rust | 1.17.0 | 1.24.1 | |
| smalltalk | | | Not in CI |
| swift | | | Not in CI |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit f5f430d

Please sign in to comment.