Skip to content
This repository has been archived by the owner on Aug 26, 2022. It is now read-only.

bug 1352232: Remove Ansible from py27 build #4239

Merged
merged 4 commits into from
May 19, 2017
Merged

bug 1352232: Remove Ansible from py27 build #4239

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f29abbe
bug 1352232: Remove Ansible from py27 build
jwhitlock May 17, 2017
e6d8b66
bug 1352232: Less Travis-specific environment
jwhitlock May 17, 2017
dbb0d94
bug 1127798: Add build with random PYTHONHASHSEED
jwhitlock May 17, 2017
c0e6ec2
bug 1352232: Use appropriate bash
jwhitlock May 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
61 changes: 46 additions & 15 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,69 @@ sudo: required
branches:
only:
- master
cache: pip
cache:
pip: True
directories:
- node_modules
- downloads
language: python
python:
- "2.7"
services:
- docker
- memcached
- mysql
env:
global:
- DATABASE_URL=mysql://root:@127.0.0.1:3306/kuma
- DJANGO_SETTINGS_MODULE=kuma.settings.travis
- DOCKER_COMPOSE_VERSION=1.9.0
# ES_VERSION must come before ES_DOWNLOAD_URL
- ES_VERSION=2.4.5
- ES_DOWNLOAD_URL=https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/$ES_VERSION/elasticsearch-$ES_VERSION.tar.gz
- PIPELINE_CSSMIN_BINARY=$TRAVIS_BUILD_DIR/node_modules/.bin/cssmin
- PIPELINE_CSS_COMPRESSOR=pipeline.compressors.cssmin.CSSMinCompressor
- PIPELINE_JS_COMPRESSOR=pipeline.compressors.uglifyjs.UglifyJSCompressor
- PIPELINE_SASS_BINARY=$TRAVIS_BUILD_DIR/node_modules/.bin/node-sass
- PIPELINE_UGLIFYJS_BINARY=$TRAVIS_BUILD_DIR/node_modules/.bin/uglifyjs
matrix:
- TOXENV=py27
CREATE_DB=kuma
INSTALL_PIPELINE=1
INSTALL_ELASTICSEARCH=1
PYTHONHASHSEED=0
- TOXENV=flake8
- TOXENV=docs
- TOXENV=locales DOCKER_COMPOSE_VERSION=1.9.0
- TOXENV=docker DOCKER_COMPOSE_VERSION=1.9.0 UID=0
- TOXENV=locales
INSTALL_DOCKER_COMPOSE=1
- TOXENV=docker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping it in single line seems more readable to me TOXENV=docker INSTALL_DOCKER_COMPOSE=1 UID=0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the new multi-line approach.

INSTALL_DOCKER_COMPOSE=1
UID=0
- TOXENV=stylelint
global:
- DJANGO_SETTINGS_MODULE=kuma.settings.travis
- DEBIAN_FRONTEND=noninteractive
- DATABASE_URL=mysql://root:kuma@localhost:3306/kuma
- CFLAGS=-O0
# TODO: make this default when http://bugzil.la/1127798 is fixed
- TOXENV=py27
CREATE_DB=kuma
INSTALL_PIPELINE=1
INSTALL_ELASTICSEARCH=1

matrix:
allow_failures:
- env: TOXENV=locales DOCKER_COMPOSE_VERSION=1.9.0
before_install:
- scripts/travis-install
- pip install -U pip
- env: TOXENV=locales INSTALL_DOCKER_COMPOSE=1
- env: TOXENV=py27 CREATE_DB=kuma INSTALL_PIPELINE=1 INSTALL_ELASTICSEARCH=1
install:
- pip install -r requirements/travis.txt
- if [[ $TOXENV == 'py27' ]]; then ansible-playbook -vvv --tags=mysql,pipeline --connection=local --inventory-file=provisioning/inventory provisioning/travis.yml; fi
- nvm install 6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we include all the installs in scripts/travis-install?
We dont need to install pip and other thing in docker testing environment. So it will faster the build

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a good follow-on PR. requirements/travis.txt includes coveralls, which is called at the end of each test, so you'd need to ensure coveralls is not called for the targets that generate coverage data. Which maybe should include the docker target, but currently doesn't. But it could be skipped for the linter tasks.

That's also a lot of work to save at most 10s per build.

- nvm use 6
script: tox -v
- scripts/travis-install
- pip install -U pip
- pip install -r requirements/travis.txt
# Wait for ElasticSearch to be ready
- if [[ "$INSTALL_ELASTICSEARCH" == "1" ]]; then wget -q --waitretry=1 --retry-connrefused -T 10 -O - http://127.0.0.1:9200; fi;
script:
if [[ "$PYTHONHASHSEED" == "0" ]]; then
tox -v --hashseed=noset;
else
tox -v;
fi
after_failure:
- dmesg | tail
after_success:
Expand Down
1 change: 0 additions & 1 deletion requirements/travis.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# this is a standalone requirements file used in .travis.yml to install
# Travis CI specific requirements
ansible==1.9.2
codecov==1.6.3
tox==2.3.1
coverage==4.0.3
2 changes: 0 additions & 2 deletions scripts/docker-compose.travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ services:
dockerfile: Dockerfile-base
environment:
- DJANGO_SETTINGS_MODULE
- DEBIAN_FRONTEND
- CFLAGS

web:
<<: *worker
Expand Down
69 changes: 46 additions & 23 deletions scripts/travis-install
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,57 @@
#!/bin/bash
set -e
set -x
set -e # Exit on non-zero status
set -x # Print commands run
set -u # Treat unset variables as an error

export LC_CTYPE=en_US.UTF-8

if [ "$TOXENV" == "py27" ]
# Download ElasticSearch
if [ ${INSTALL_ELASTICSEARCH:-0} -ne 0 ]
then
sudo apt-get update -qq
sudo apt-get -y install build-essential libxml2-dev libxslt-dev libjpeg8 libjpeg8-dev libfreetype6 libfreetype6-dev zlib1g-dev sqlite3 tidy libtidy-dev libtidy-0.99-0 python-dev libffi-dev libssl-dev
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib
mkdir -p downloads
wget -q -O downloads/elasticsearch-$ES_VERSION.tar.gz $ES_DOWNLOAD_URL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you build the url here?
Rather than passing the url in environment parameters

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, you can't. ElasticSearch keeps changing their downloads domain. Here's a sample:

Version URL
2.2.0 https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.2.0/elasticsearch-2.2.0.tar.gz
1.7.6 https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.6.tar.gz
5.3.0 https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.0.tar.gz

tar -zxf downloads/elasticsearch-$ES_VERSION.tar.gz
fi

# completely and utterly remove Travis' MySQL and let the Ansible role install it
sudo apt-get remove --purge 'mysql*'
sudo apt-get autoremove
sudo apt-get autoclean
sudo rm -rf /var/lib/mysql
# Run ElasticSearch in background
(if [ ${INSTALL_ELASTICSEARCH:-0} -ne 0 ]
then
./elasticsearch-$ES_VERSION/bin/elasticsearch 1> elasticsearch.log 2> elasticsearch.err
fi) &

# Install Elasticsearch
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb
sudo dpkg -i elasticsearch-1.7.2.deb

# limit elasticsearch / java memory usage to avoid OOM Killer
sudo service elasticsearch stop;
echo "ES_HEAP_SIZE=256m" | sudo tee --append /etc/default/elasticsearch
sudo service elasticsearch start;
# Get database ready
if [ -n "${CREATE_DB:-}" ]
then
mysql -e "CREATE DATABASE IF NOT EXISTS ${CREATE_DB};";
fi

# Install pipeline toolchain
if [ ${INSTALL_PIPELINE:-0} -ne 0 ]
then
npm install
npm install cssmin@0.4.3
npm install uglify-js@2.4.13
fi

if [ "$TOXENV" == "docker" -o "$TOXENV" == "locales" ]
# Install docker-compose
if [ ${INSTALL_DOCKER_COMPOSE:--} -ne 0 ]
then
curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
chmod +x docker-compose
sudo mv docker-compose /usr/local/bin
if [ -x $(command -v docker-compose) ]
then
echo "Overwriting existing docker-compose."
docker-compose -v
else
echo "Installing docker-compose ${DOCKER_COMPOSE_VERSION}."
fi
mkdir -p downloads
DOCKER_COMPOSE_FILE=downloads/docker-compose-${DOCKER_COMPOSE_VERSION}
if [ ! -f $DOCKER_COMPOSE_FILE ]
then
wget -q --waitretry=1 --retry-connrefused -T 10 \
-O downloads/docker-compose-${DOCKER_COMPOSE_VERSION} \
https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m`
fi
chmod +x $DOCKER_COMPOSE_FILE
sudo cp $DOCKER_COMPOSE_FILE /usr/local/bin/docker-compose
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we cache this path and check if the version matches with the provided one?

INSTALLED_DOCER_COMPOSE_VERSION=$(docker-compose -v | grep -o '[0-9]\?[0-9].[0-9]\?[0-9].\?[0-9]')
if ! [ INSTALLED_DOCER_COMPOSE_VERSION == DOCKER_COMPOSE_VERSION];
then
    # Download and install
fi

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that will never ever happen. The VM is reset each time, so the version will be 1.8.1 until TravisCI updates their VM image. When they do that, the new default may be later than 1.9.0, and we can stop installing our own docker-compose. Until then, the download folder is cached, so we don't download the file unless we change DOCKER_COMPOSE_VERSION.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jwhitlock We can add /usr/local/bin/docker-compose to cache config to keep the docker-compose

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can cache directories but not files. I think it would be a mistake to cache /usr/local/bin, so I think we're OK caching the file in downloads and copying it each time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the docker-compose file is already in the downloads dir, which is in the cache due to https://github.com/mozilla/kuma/pull/4239/files#diff-354f30a63fb0907d4ad57269548329e3R9

fi
13 changes: 6 additions & 7 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,12 @@ whitelist_externals = make
deps =
-rrequirements/dev.txt
commands =
make compilejsi18n collectstatic coveragetest target={posargs:kuma}
setenv =
PYTHONPATH = .
CFLAGS = -O0
# TODO: remove once http://bugzil.la/1127798 is fixed
PYTHONHASHSEED = 0
passenv=DJANGO_SETTINGS_MODULE DEBIAN_FRONTEND CFLAGS
make compilejsi18n collectstatic clean
py.test --no-migrations --cov=kuma kuma
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If new migration is added, will it fail?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not here, since we aren't running them due to --no-migrations.

passenv =
DATABASE_URL
DJANGO_SETTINGS_MODULE
PIPELINE_*

[testenv:flake8]
basepython = python2.7
Expand Down