Build fails when enabling caching, but everything shows success in UI

[allanyung]

https://app.shippable.com/github/ImmersiveLabsUkOrg/shippable-test/runs/51/1/console - succeeds
https://app.shippable.com/github/ImmersiveLabsUkOrg/shippable-test/runs/50/1/console - fails

The only difference between these 2 runs is that the failed run has cache: true in the build section. Enabling caching causes the build to fail 100% of the time, but every step shows success in the UI. The debug logs do show an error:

echo Container c.exec.shippable-test.50.1 exited with 99Container c.exec.shippable-test.50.1 exited with 99
sudo docker logs c.exec.shippable-test.50.1 No LSB modules are available.
Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial
Ubuntu confirmed
Looking for apt-get...
Found apt-get at /usr/bin/apt-get
Looking for sudo...
Found sudo at /usr/bin/sudo
Looking for git...
Found git at /usr/bin/git
Looking for ssh-agent
Found ssh-agent at /usr/bin/ssh-agent
Looking for python...
Found python at /usr/bin/python
Booting up CEXEC
Running CEXEC script
ERROR:script_runner - script_runner:Command failed : ssh-agent bash -c 'ssh-add /tmp/ssh/00_sub;ssh-add /tmp/ssh/01_deploy; cd /root && /root/40e139c4-fdb5-44c0-bb82-219be9f58b88.sh'
Exception Invalid or no script tags received
ERROR:script_runner - script_runner:Command failed : ssh-agent bash -c 'ssh-add /tmp/ssh/00_sub;ssh-add /tmp/ssh/01_deploy; cd /root && /root/40e139c4-fdb5-44c0-bb82-219be9f58b88.sh'
Exception Invalid or no script tags received
sudo docker rm -fv $CONTAINER_NAME c.exec.shippable-test.50.1

[trriplejay]

This sounds like a duplicate of #4297 (comment)

As a workaround, can you try changing your runtime version to v6.1.4? You can do this by going to your subscription dashboard, then clicking settings in the top right -> node pools -> edit -> select v6.1.4 from the runtime version drop down box -> save pool.

it seems to be some combination of cache, ruby, and newer runtime versions. we're still narrowing it down.

[trriplejay]

actually, I just realized that runtime version v6.1.4 is not available for u16 dynamic nodes which you are currently on.

Instead, the workaround is to add a pre_ci_boot section to your yml that tells the build to use the v6.1.4 version of the build image. It would look like this:

build:
  pre_ci_boot:
    image_name: drydock/u16ruball
    image_tag: v6.1.4
    pull: true
  ci:
    - echo "your ci commands here"

or you could disable caching.

[allanyung]

Changing the runtime to v6.1.4 stops the caching from breaking the build. Not sure if you want to keep this bug open or not; there's a workaround but the original issue still exists. Cheers.

[a-murphy]

Since this was confirmed to be a duplicate of #4297, I'll close this one to keep all reports linked to a single open issue. Progress on solving the issue will be announced in #4297.