Use Travis-ci's container infrastructure (bug 1118887) #390
Conversation
magopian
force-pushed
the
travis-sudo-false
branch
18 times, most recently
from
13b5350
to
15b8335
Compare
| @@ -3,11 +3,10 @@ envlist = es, addons-devhub-editors, main, flake8, docs, assets | |||
|
|
|||
| [testenv] | |||
| basepython = python2.6 | |||
| install_command = pip install --no-deps --exists-action=w --download-cache=/tmp/pip-cache --find-links https://pyrepo.addons.mozilla.org/ {opts} {packages} | |||
| install_command = pip install --no-deps --exists-action=w --download-cache=/tmp/pip-cache --find-links http://mathieu.agopian.info/wheelhouse/ {opts} {packages} | |||
There was a problem hiding this comment.
This is only for testing, it needs to be changed back to pyrepo when the wheels are uploaded.
magopian
force-pushed
the
travis-sudo-false
branch
2 times, most recently
from
ed2fd2b
to
d551b95
Compare
| @@ -3,10 +3,9 @@ envlist = es, addons-devhub-editors, main, flake8, docs, assets | |||
|
|
|||
| [testenv] | |||
| basepython = python2.6 | |||
| install_command = pip install --no-deps --exists-action=w --download-cache=/tmp/pip-cache --find-links https://pyrepo.addons.mozilla.org/ {opts} {packages} | |||
| install_command = pip install --no-deps --exists-action=w --download-cache=/tmp/pip-cache --find-links https://pyrepo.addons.mozilla.org/wheelhouse/ {opts} {packages} | |||
There was a problem hiding this comment.
If packages aren't in the wheelhouse, does it fall back to our mirror or the public one (or none?)?
There was a problem hiding this comment.
It should fallback to using pypi
|
These tests are substantially faster. Would you talk more about what the "fast builds feature" is and what you're doing to enable it? |
| - wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.deb && sudo dpkg -i elasticsearch-1.3.2.deb | ||
| - sudo /usr/share/elasticsearch/bin/elasticsearch -d -D es.path.data=/tmp -D es.gateway.type=none -D es.index.store.type=memory -D es.discovery.zen.ping.multicast.enabled=false | ||
| - wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz && tar xvf elasticsearch-1.3.2.tar.gz | ||
| - elasticsearch-1.3.2/bin/elasticsearch -d -D es.path.data=/tmp -D es.gateway.type=none -D es.index.store.type=memory -D es.discovery.zen.ping.multicast.enabled=false | ||
| install: | ||
| - pip install tox | ||
| before_script: |
There was a problem hiding this comment.
if you add this you'll only have to download packages once:
cache:
directories:
- /tmp/pip-cache
I also wrote a script to bundle elasticsearch @ mozilla/zamboni#2821
There was a problem hiding this comment.
According to the travis documentation:
The caching tars up all the directories listed in the configuration and uploads them to S3, using a secure and protected URL, ensuring security and privacy of the uploaded archives.
Note that this makes our cache not network-local, it's still bound to network bandwidth and DNS resolutions for S3.
That's why I didn't bother dealing with a cache (and the cache invalidation issues that may arise?), even more now that we use wheels (which makes it that much faster than compiling locally). I'm not sure the very small hypothetical gain from downloading from S3 instead of pyrepo (or pypi) is worth the trouble. What do you think?
There was a problem hiding this comment.
It's much faster on the Marketplace. You can see https://travis-ci.org/mozilla/zamboni/builds/46229240 for the latest numbers on my PR. It takes 35-40s to run the pip command now. Before I added the caching, eg. https://travis-ci.org/mozilla/zamboni/builds/46118204 , it took around 130-150s to run.
|
Yeah the tests are much faster using this new architecture. It's based on docker images, and has two advantages: it uses 2 cores (instead of 1.5 for the current architecture, from what I can understand), and is much much faster to start (when the machines are loaded, you could wait for several minutes for your build to start... with the new architecture, it should start in less than 10 seconds). It also seems that this new architecture is running on faster machines, as the total build time is much shorter. The only limitation I can see for now is that you can't use To use this new 'fast architecture' you simply need to add |
magopian
force-pushed
the
travis-sudo-false
branch
from
d551b95
to
769767d
Compare
magopian
changed the title
|
@clouserw i've used your elasticsearch script, and activated the cache, if that still looks good to you, I'll merge ;) |
|
Well, my mistake, I merged before your answer. If you feel this is missing something or should be reverted, please let me know :/ |
Fixes bug 1118887 for olympia