New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trusty 2017/Q3 updates encountered issues and their workarounds #8315

Closed
cotsog opened this Issue Aug 29, 2017 · 32 comments

Comments

Projects
None yet
@cotsog
Copy link

cotsog commented Aug 29, 2017

The new 2017/Q3 Trusty images were released on Thursday, September 7th, 2017. More details can be found at the following links:

Take note that you can add the following in your .travis.yml file to use the previous Trusty image:

group: deprecated-2017Q3

List of issues:

  • TBA
  • ...
@peternewman

This comment has been minimized.

Copy link

peternewman commented Sep 3, 2017

The build environments update page lists clang as changing to be 3.9.0 ( https://docs.travis-ci.com/user/build-environment-updates/2017-09-06/#Changed ), but on an edge build ( https://travis-ci.org/OpenLightingProject/ola/jobs/271392249#L8 ) it's still just listing 3.5.0:
https://travis-ci.org/OpenLightingProject/ola/jobs/271392249#L43

We're actually manually installing 5.0.0, from the clang repos, so it's mostly irrelevant, but I'm just curious why it doesn't seem to have updated?

@edmorley

This comment has been minimized.

Copy link

edmorley commented Sep 3, 2017

Someone from Travis CI will need to confirm this - but that job is a container job, and I believe they aren't able to use the group attribute. If you change to the fully virtualised GCE infrastructure (sudo: required) you should get the edge image.

Perhaps it would be good to document the group attribute somewhere?

@peternewman

This comment has been minimized.

Copy link

peternewman commented Sep 4, 2017

Interesting thanks Ed. As mentioned, it reports, and changes the listed build group, so I'd hope it didn't do that if it wasn't going to change anything.

Some documentation on the default and available groups would certainly be good.

@synth

This comment has been minimized.

Copy link

synth commented Sep 5, 2017

I received Command 'make ' failed. Adding group: deprecated-2017Q3 did not help. Also, we have dist: precise already in our travis.yml. :(

current directory:
/home/travis/build/Recognize/recognize/vendor/bundle/ruby/2.2.0/gems/capybara-webkit-1.14.0
/home/travis/.rvm/rubies/ruby-2.2.2/bin/ruby -r
./siteconf20170905-5943-1o74hch.rb extconf.rb
cd src/ && /usr/bin/qmake
    /home/travis/build/Recognize/recognize/vendor/bundle/ruby/2.2.0/gems/capybara-webkit-1.14.0/src/webkit_server.pro
-o Makefile.webkit_server
cd src/ && make -f Makefile.webkit_server 
make[1]: Entering directory
`/home/travis/build/Recognize/recognize/vendor/bundle/ruby/2.2.0/gems/capybara-webkit-1.14.0/src'
g++ -m64 -pipe -O2 -Wall -W -D_REENTRANT -DQT_NO_DEBUG -DQT_WEBKIT_LIB
-DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED
-I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore
-I/usr/include/qt4/QtNetwork -I/usr/include/qt4/QtGui
-I/usr/include/qt4/QtWebKit -I/usr/include/qt4 -Ibuild -x c++-header -c stable.h
-o build/webkit_server.gch/c++
stable.h:23:23: fatal error: QWebElement: No such file or directory
compilation terminated.
make[1]: *** [build/webkit_server.gch/c++] Error 1
make[1]: Leaving directory
`/home/travis/build/Recognize/recognize/vendor/bundle/ruby/2.2.0/gems/capybara-webkit-    1.14.0/src'
make: *** [sub-src-webkit_server-pro-make_default-ordered] Error 2
Command 'make ' failed
@cotsog

This comment has been minimized.

Copy link

cotsog commented Sep 5, 2017

The update will happen tomorrow i.e. Wednesday, September 6th, 2017. In the meantime, please remove the group: deprecated-2017Q3 directive from your .travis.yml file as it can have undesired side-effects.

@peternewman: can you send us an e-mail at support [at] travis-ci [dot] com with these details?

@synth: we will follow-up over the e-mail you've already sent us.

Thanks!

@MarcoFalke

This comment has been minimized.

Copy link

MarcoFalke commented Sep 6, 2017

pyenv reports that python 3.6 is installed. Is this wanted?

Output:

$ pyenv versions
* system (set by /opt/pyenv/version)
  2.7
  2.7.13
  3.3
  3.3.6
  3.4
  3.4.6
  3.5
  3.5.3
  3.6
  3.6.1
  pypy2-5.6.0

I am using the following yaml:

sudo: required
dist: trusty
os: linux
group: edge
language: generic
before_script:
    - pyenv versions
@edmorley

This comment has been minimized.

Copy link

edmorley commented Sep 6, 2017

@MarcoFalke the language: generic property used to map to the smallest connie image, but in travis-ci/packer-templates#454 was made to map to the largest sugilite image instead. From the date of that PR I think it would have already been in the last round of image updates (and so presumably this not be the cause for the surprise), but @meatballhat would have to confirm.

Anyway, for the sugilite image, Python 3.6.x is expected to be installed:
https://github.com/travis-ci/packer-templates/blob/f33ae65/cookbooks/travis_ci_sugilite/attributes/default.rb#L117-L124

@MarcoFalke

This comment has been minimized.

Copy link

MarcoFalke commented Sep 6, 2017

Thanks for the reply. Though, I can't figure out how to call py3.6.

Doing those in the before_script:

0.06s$ /opt/pyenv/shims/python3.4 -c 'print("test")'
test
0.07s$ /opt/pyenv/shims/python3.6 -c 'print("test")'
pyenv: python3.6: command not found
The `python3.6' command exists in these Python versions:
  3.6
  3.6.1
The command "/opt/pyenv/shims/python3.6 -c 'print("test")'" failed and exited with 127 during .
Your build has been stopped.
@rhs

This comment has been minimized.

Copy link

rhs commented Sep 7, 2017

I'm getting similar python version issues in my builds. This used to work before today:

pyenv: pip3: command not found
The `pip3' command exists in these Python versions:
  3.3
  3.3.6
  3.4
  3.4.6
  3.5
  3.5.3
  3.6
  3.6.1
@cotsog

This comment has been minimized.

Copy link

cotsog commented Sep 7, 2017

@MarcoFalke, @rhs: could you send us an e-mail to support [at] travis-ci [dot] com? Please include a description of your issue, the name of the repo where it's happening and links to builds on travis-ci.org or travis-ci.com showing the issue. Thanks!

@rsimha

This comment has been minimized.

Copy link

rsimha commented Sep 7, 2017

@BanzaiMan, we're seeing a bunch of things breaking on the AMP Project due to the new images.

Errors: https://travis-ci.org/ampproject/amphtml/jobs/272980280
Tracking issue: ampproject/amphtml#11203

I'm switching to the old image for now, while I try to fix these errors. Meanwhile, any input from you re: how to fix them is welcome.

@rsimha

This comment has been minimized.

Copy link

rsimha commented Sep 7, 2017

@BanzaiMan, after switching to the old image, we're still seeing a bunch of errors due to missing plugins, even though yarn.lock is up to date and contains specifications for those plugins. See https://travis-ci.org/ampproject/amphtml/jobs/273011615.

Right now, neither the old image, nor the new image is working, and this is now blocking all development work on our side. Could you please take a quick look at the link above, and let me know if you believe I'm missing something?

@BanzaiMan

This comment has been minimized.

Copy link
Member

BanzaiMan commented Sep 7, 2017

@rsimha-amp Could you try removing (or disabling) caching?

@rsimha

This comment has been minimized.

Copy link

rsimha commented Sep 7, 2017

@BanzaiMan, thanks. Disabling caching did the trick.

@BanzaiMan

This comment has been minimized.

Copy link
Member

BanzaiMan commented Sep 7, 2017

@rsimha-amp Thanks for the report. Please remove existing caches. They are likely culprit in your build failures.

@MarcoFalke

This comment has been minimized.

Copy link

MarcoFalke commented Sep 7, 2017

@cotsog @cfields See here for the minimal example that breaks: https://travis-ci.org/MarcoFalke/bitcoin/jobs/272286278/config

You can see that pyenv versions advertises py3.6, but the actual call fails.

@Shilpi85

This comment has been minimized.

Copy link

Shilpi85 commented Sep 7, 2017

We are running Parallel Cucumber selenium Test using the parallel_tests with 5 Process and with the new Image lots of tests are failing with time out issue.
So for now we added the group: deprecated-2017Q3 and after that the tests are running fine.

@rsimha

This comment has been minimized.

Copy link

rsimha commented Sep 7, 2017

@BanzaiMan, the missing plugin errors went away when I deleted the old caches. However, we're still seeing build errors related to the addition of _JAVA_OPTIONS in the new images.

See ampproject/amphtml#11210

Any idea what's going on here?

Edit: Turns out that the new _JAVA_OPTIONS settings broke the closure compiler we use to build the AMP runtime. We've checked in a workaround that unsets the env var, and we are now back to using the new Travis images.

@riquito

This comment has been minimized.

Copy link

riquito commented Sep 8, 2017

We run into issues because there were new services running

  • mysql
  • memcached

We usually run these services ourselves using docker and the containers started to fail because the ports were already used (3306, 11211). We already had group: deprecated-2017Q3 in our .travis.yml. Fixed by stopping MySQL and memcached services

before_install:
    - sudo service mysql stop
    - sudo service memcached stop
@csantanapr

This comment has been minimized.

Copy link

csantanapr commented Sep 8, 2017

In Apache OpenWhisk we are having issues around pip installing setuptools
Our .travis.yml has something like this

sudo: required

language: scala
scala:
   - 2.11.8

services:
  - docker

before_install:
  - pip install --upgrade pip setuptools six
  - pip3 install --upgrade pip setuptools six

It errors like this:

$ pip install --upgrade pip setuptools six
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already up-to-date: pip in /usr/local/lib/python2.7/dist-packages
Collecting setuptools
  Downloading setuptools-36.4.0-py2.py3-none-any.whl (478kB)
    100% |████████████████████████████████| 481kB 1.8MB/s 
Collecting six
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: setuptools, six
  Found existing installation: setuptools 36.3.0
    Uninstalling setuptools-36.3.0:
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 754, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/usr/lib/python2.7/shutil.py", line 303, in move
    os.unlink(src)
OSError: [Errno 13] Permission denied: '/usr/local/bin/easy_install'
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

Please advise

@stlaz

This comment has been minimized.

Copy link

stlaz commented Sep 8, 2017

Getting the very same pyenv: pip3: command not found issue as @rhs. Is there a way around for this yet or should we just use the older image?
Repo: https://github.com/freeipa/freeipa
Sample build: https://travis-ci.org/freeipa/freeipa/jobs/273180675

@reactive-firewall

This comment has been minimized.

Copy link

reactive-firewall commented Sep 8, 2017

Also getting the same kinds of errors with pip module in python3
Repo: https://www.github.com/reactive-firewall/Pocket-PiAP
Checkout hash: 55a877cbbc69a88524f2bcdf07b6d6d5d8536e5e
Sample Build: (239.1) https://travis-ci.org/reactive-firewall/Pocket-PiAP/jobs/273182690
relevant Log:

$ travis_retry python3 -m pip install tox || python3 -m pip install tox || true ;

/usr/bin/python3: No module named pip

The command "python3 -m pip install tox" failed. Retrying, 2 of 3.

/usr/bin/python3: No module named pip

#UPDATE#
In-case anyone else is still stuck on this:
As a workaround simply adding the addon apt to the .travis.yml config with the packages python3 and python3-pip seem to resolve this issue in the case of python3 -m pip install ...
Example Config: https://travis-ci.org/reactive-firewall/Pocket-PiAP/jobs/273225601/config
Config Workaround Lines:

addons:
  apt:
    packages:
    - python3
    - python3-pip

cc #8378
...additionally this line does not break python 2.7 (seeming default python on Travis Linux installs) when using language: generic although YMMV.
Hope this helps

@daneah

This comment has been minimized.

Copy link

daneah commented Sep 10, 2017

I, among others, see MySQL connection issues when moving from precise to trusty, as documented here. This still seems to be a problem as of this morning!

shane-tomlinson added a commit to mozilla/fxa-content-server that referenced this issue Sep 11, 2017

shane-tomlinson added a commit to mozilla/fxa-content-server that referenced this issue Sep 11, 2017

@audiolion

This comment has been minimized.

Copy link

audiolion commented Sep 11, 2017

My TOX builds are failing for python 3.5 only

https://travis-ci.org/jjkester/django-auditlog/builds/274268676

@hroest

This comment has been minimized.

Copy link

hroest commented Sep 11, 2017

It seems that there currently is an issue with using Clang 3.9 and not having libomp available, therefore making our C++ projects with OpenMP fail in the linking stage when using clang. We can still compile without OpenMP support and we can still use gcc with OpenMP. However, Clang 3.9 does not seem to work with libgomp and we cannot install libomp on the 14.04 system (since it is only available for 16.04). Any help would be appreciate here or over at OpenMS/OpenMS#2906

@eed3si9n

This comment has been minimized.

Copy link

eed3si9n commented Sep 13, 2017

Reported _JAVA_OPTIONS causing problem for me as #8408.

workaround:

# Undo _JAVA_OPTIONS environment variable
before_script:
  - _JAVA_OPTIONS=

Kha added a commit to leanprover/lean that referenced this issue Sep 15, 2017

chore(.travis.yml): undo Travis update
The test_registry test is failing because `python3` seems to have reverted to Python 3.4 in a Travis upgrade. Probably related to travis-ci/travis-ci#8315 (comment).

leodemoura added a commit to leanprover/lean that referenced this issue Sep 15, 2017

chore(.travis.yml): undo Travis update
The test_registry test is failing because `python3` seems to have reverted to Python 3.4 in a Travis upgrade. Probably related to travis-ci/travis-ci#8315 (comment).
@stale

This comment has been minimized.

Copy link

stale bot commented Apr 13, 2018

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

@stale stale bot added the stale label Apr 13, 2018

@stale stale bot closed this Apr 15, 2018

jsirois added a commit to jsirois/pants that referenced this issue Sep 2, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Specify `python: "2.7"` which floats us to the most modern python 2.7
  (2.7.14 currently)
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this issue Sep 3, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to `python: "2.7.14".
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this issue Sep 4, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this issue Sep 4, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this issue Sep 5, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this issue Sep 6, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to pantsbuild/pants that referenced this issue Sep 6, 2018

Tighten travis matrix and python activation. (#6440)
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support #6428 which fixes #6415.

jsirois added a commit to jsirois/pants that referenced this issue Sep 7, 2018

Tighten travis matrix and python activation. (pantsbuild#6440)
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to pantsbuild/pants that referenced this issue Sep 10, 2018

Cherry-pick setup_py fixes (#6472)
* [fix] Pass full path to isdir rather than just basename. (#6453)

* [fix] Pass full path to isdir rather than just basename.

* [test] Add tests for fix.

* Tighten travis matrix and python activation. (#6440)

A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support #6428 which fixes #6415.

* Remove broken pyenv shims from the PATH. (#6469)

It turns out these would still get found in various contexts since the
interpreter cache tries to setup everything it finds on the PATH. Add
improved context in the wrapper script to show exactly which pythons
Pants will have access to.

* Fixup tests involving pexrc. (#6446)

Previously these tests weren't reliably reading a pexrc and had fragile
workarounds and paper-overs as a result. pexrc_util now reliably injects
a pexrc with a temporary HOME.

Also cleanup the affected tests - use @skipIf, respect 100 cols, ensure
path comparisons with judicious application of `os.realpath`.

* Detect ns packages using correct interpreter. (#6428)

Previously `__init__.py`s were parsed in the context of the ambient
interpreter executing the `SetupPy` task which could lead to parse
failures for new or removed language features used in those
`__init__.py`s. We now construct a PEX using the appropriate interpreter
and use it to run ns package detection.

Fixes #6415

* Fix setup.py rendering. (#6439)

We now render setup.py from both python 2 & 3 such that the result
is ingestible by both python 2 & 3.

Fixes #6401
Fixes #6392
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment