Latest YCM devel of robotology-superbuild on Travis with Docker blocks during bootstrap on "Performing build step for 'YCM''

[traversaro]

See https://travis-ci.org/robotology/robotology-superbuild/builds/527715806 . Until yesterday, everything went fine. I already restarted a job, and the same problem appeared again.

I wonder if the problem is related to this commit: 912e0d6

[traversaro]

Apparently the problem is there also with v0.10.2 (see https://travis-ci.org/robotology/robotology-superbuild/builds/527850737?utm_source=github_status&utm_medium=notification), so the problem is probably some weird Travis problem in accessing one of the remote repos used in the YCM bootstrap (ref #105).

[traversaro]

I am unable to reproduce the problem on my local machine.

[traversaro]

I enabled YCM_BOOSTRAP_VERBOSE in a test build (robotology/robotology-superbuild#201) and apparently the problem is some strange interaction of CMake's git repo (even if apparently we are using a GitHub-based mirror) and Travis. The line

 [ 16%] Downloading file Copyright.txt from CMake git repository (ref v3.14.3)

took ~2/3 minutes, and now the build seems stuck to:

[ 17%] Downloading file Modules/BasicConfigVersion-SameMajorVersion.cmake.in from CMake git repository (ref v3.14.3)

see https://travis-ci.org/robotology/robotology-superbuild/jobs/527863239 .

Travis CI - Test and Deploy Your Code with Confidence

After a while, download this file result in Timeout:

[ 17%] Downloading file Modules/BasicConfigVersion-SameMajorVersion.cmake.in from CMake git repository (ref v3.14.3)
-- Cannot download file https://raw.githubusercontent.com/Kitware/CMake/v3.14.3/Modules/BasicConfigVersion-SameMajorVersion.cmake.in
  Network problem or not existing file.
  CMake Error at /home/travis/build/robotology/robotology-superbuild/build/robotology/YCM/cmake-next/CMakeFiles/cmake-v3.14.3.dir/ycm_download_Modules_BasicConfigVersion_SameMajorVersion_cmake_in_real.cmake:9 (file):
  file DOWNLOAD HASH mismatch
    for file: [/home/travis/build/robotology/robotology-superbuild/build/robotology/YCM/cmake-next/CMakeFiles/cmake-v3.14.3.dir/downloads/Modules/BasicConfigVersion-SameMajorVersion.cmake.in]
      expected hash: [f89dc13283012256d1cb702ffb0bc235cee5cc5a]
        actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
             status: [28;"Timeout was reached"].
  Retrying.
[ 18%] Downloading file Modules/BasicConfigVersion-SameMinorVersion.cmake.in from CMake git repository (ref v3.14.3)

[traversaro]

Related tweets:

• https://twitter.com/SleevelessZach/status/1124323582337867777
• https://twitter.com/beaugunderson/status/1124336607333535745

Twitter

SleevelessZach on Twitter

“@TravisCI getting a lot of "No output has been received in the last 10m0s" as of this morning. Any elevated error rates? seems to be happening at different stages of my builds.”

Twitter

beau on Twitter

“anyone else having travis CI issues since mid-day yesterday? all of our builds since then are failing with timeout errors after ~20 minutes 😬️”

[beaugunderson]

i'm in contact with travis support and will let y'all know if i hear anything useful from them :)

[traversaro]

I tried replicating the issue on ycm's Travis, but apparently everything is working correctly there. Probably the issue is related to the fact the the robotology-superbuild Travis build is working inside a docker container.

[traversaro]

i'm in contact with travis support and will let y'all know if i hear anything useful from them :)

Thanks a lot @beaugunderson !

[beaugunderson]

if you want to contact their support as well you can reference our support ticket so they can cross-reference it; it's 6994

[zak10]

I'm not familiar at all with this repository, but my builds have also been getting stuck and timing out. Strangely, it happens during different stages and it is only for one of my repositories. I reran builds from old commits that were fine and those are also breaking so assuming this is a travis problem.

Running a python 3.6 container, tests via docker compose.

[beaugunderson]

fwiw we also use docker/docker-compose in our tests...

edit: verified that we're still pinned to the same docker-compose version, and docker-ce also has an identical version between old successful and now failing builds (18.06.3~ce~3-0~ubuntu)

[zak10]

@beaugunderson do your tests run in multiple threads?

[beaugunderson]

@zak10 yup, we use gnu parallel

edit: and also pytest + multiple xdist workers for the python tests that run inside one of the containers, if that's what you were asking

[alex]

I'm also experiencing this issue -- also in docker builds.

[zak10]

We're using pytest + xdist to run tests in parallel. I'm attempting now to remove xdist completely to see if that alleviates the issue.

edit: I was unable to alleviate any issues and now it seems the issue has spread to a second repository with a similar configuration. Still no word from travis - I think it may be time for me to pull the trigger and switch to a different CI provider.

[beaugunderson]

still no response from travis since last thursday :(

[beaugunderson]

currently trying to force dist: xenial and see if that has any effect, will report back...

[beaugunderson]

Hi Beau,

Thank you for getting in touch, and we apologize for the frustration caused by this ongoing issue with slow or stalling downloads from within Docker containers.

We’ve been receiving similar reports since late on Friday and have proceeded to open an incident to acknowledge the situation here: https://www.traviscistatus.com/incidents/kyf149kl6bvp. We'll be posting updates to the status as we continue to investigate this issue.

In the meantime, you might be able to stabilize your builds and prevent having to manually restart them by adding travis_retry to your failing commands e.g.

travis_retry docker build [...]

This will effectively retry the command up to 3 times if it fails.

Once again we apologize for the inconvenience caused as we work diligently towards a resolution. In the meantime, if you have any questions or concerns, please do not hesitate to get in touch. We'd be happy to assist.

Slow or stalling downloads inside Docker container

Travis CI's Status Page - Slow or stalling downloads inside Docker container.

[beaugunderson]

i feel like this fundamentally misunderstands the issue, sadly

[beaugunderson]

dist: xenial did not help, FYI

[zak10]

yep - got the same canned response from them. at least they're acknowledging the issue now. unfortunately for them, I've already begun the process of migrating to Circle.

best of luck!

[traversaro]

I tried the --network=host workaround mentioned in https://www.traviscistatus.com/incidents/kyf149kl6bvp in https://travis-ci.org/robotology/robotology-superbuild/jobs/528940445, and it seems to be working.

Slow or stalling downloads inside Docker container

Travis CI's Status Page - Slow or stalling downloads inside Docker container.

Travis CI - Test and Deploy Your Code with Confidence

[beaugunderson]

awesome @traversaro; i'm trying that now as well (slightly more complicated with docker-compose, but it seems like network_mode: ${NETWORK_MODE:-bridge} inside the build: option would work, and then specifying NETWORK_MODE=host for travis builds)

[beaugunderson]

this is more of a pain than i thought, especially if you already use networks...

[traversaro]

Related issue: ros-industrial/industrial_ci#364 .

[beaugunderson]

travis is claiming that this issue is fixed but we're still getting 100% failures... around the time they were testing fixes we got a few passes, so it certainly improved for a small period of time, but now it's broken again and i'm having a hard time getting their support to do anything about it

[traversaro]

On our side, in the last day all the build that were affected by that (see https://travis-ci.org/robotology/robotology-superbuild/jobs/529722072) are working correctly. Given @beaugunderson input, probably it make sense to monitor a bit more for possible failures.

Travis CI - Test and Deploy Your Code with Confidence

[traversaro]

On our side, we did not observed Travis failures related to this in the past week. I think we can close the issue, at least for what concern YCM and its bootstrap.