Task fails with "config file not found" after restarting Docker service

[mbjelac]

Disclaimer: There are a couple of issues which seem to be the same as this one. However, I decided to create a new one because:

• they do not describe the same problem (i.e. there are variations which seem important to me)
• one (Task file not found when running in AWS #355) was closed due to lack of reporter response
• the other (3.1.1 jobs fail with "file not found" after flushing workers #1251) was closed and replaced with two technical issues which I don't understand and shouldn't need to (by the way, I don't get this practice of closing user issues and replacing them with one or more tech issues, please stop doing that)

Bug Report

• Concourse version: 3.5.0
• Deployment type (BOSH/Docker/binary): Docker 17.06.0-ce, build 02c1d87
• Infrastructure/IaaS: Linux (privately hosted virtual machine)

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Browser (if applicable): Google Chrome 62.0.3202.89 (Official Build) (64-bit)
• Did this used to work? No, it happens consistently with older versions too.

TL;DR

When the Docker service which hosts Concourse restarts (manually, or by rebooting the virtual or physical machine), all the builds fail on the first task which requires a YAML config file with the message:

task config '<yml-config-file-path>' not found

Reconstruct cookbook

• install Docker on a machine (see version above)
• start Docker service
• start Concourse containers with docker-compose up -d using docker-compose definition:

version: '3.3'

services:
  concourse-db:
    image: postgres:9.5
    volumes: ["/srv/docker/containers/concourse-db/data:/database"]
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"
    environment:
      POSTGRES_DB: concourse
      POSTGRES_USER: concourse
      POSTGRES_PASSWORD: changeme
      PGDATA: /database

  concourse-web:
    image: concourse/concourse
    links: [concourse-db]
    command: web
    depends_on: [concourse-db]
    ports: ["7070:8080"]
    volumes: ["/etc/docker/container/concourse/keys/web:/concourse-keys"]
    restart: unless-stopped # required so that it retries until conocurse-db comes up
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "10"
    environment:
      CONCOURSE_BASIC_AUTH_USERNAME: concourse
      CONCOURSE_BASIC_AUTH_PASSWORD: changeme
      CONCOURSE_EXTERNAL_URL: http://<ci-host>:7070
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: changeme
      CONCOURSE_POSTGRES_DATABASE: concourse

  concourse-worker:
    image: concourse/concourse
    privileged: true
    links: [concourse-web]
    depends_on: [concourse-web]
    command: worker
    volumes: 
      - "/etc/docker/container/concourse/keys/worker:/concourse-keys"
      - "/srv/docker/containers/concourse-worker/work-dir:/work-dir"
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"
    environment:
      CONCOURSE_TSA_HOST: concourse-web
      CONCOURSE_WORK_DIR: /work-dir

• create a pipeline
  • be sure to have jobs with tasks that require external YAML config files, and tasks which have inlined configuration
• start pipeline and verify they work as expected
• on the machine you installed Docker on, do:

docker service stop
docker service start

What happens?

• start a build of a job which has tasks defined in YAML config files
  • the task errors with message: task config '<yml-config-file-path>' not found
• start a build of a job which has tasks with inlined configuration
  • they pass
• start a job which has which has tasks with inlined configuration, but also includes output resources, such as e-mail
  • the PUT step errors with message: file not found
• some inputs are in error: unknown handle: <uuid>
  • this is the probable cause of the task config not found failure ... but what is the parent cause?

Check workers

The single worker appears to be fine:

fly -t ci workers
name          containers  platform  tags  team  state    version
b640b1adefed  10          linux     none  none  running  1.2

After a while...

• start a job - it hangs in the preparing build stage:

---------------
preparing build
---------------
(checkmark) checking pipeline is not paused
(checkmark) checking job is not paused
(checkmark) discovering any new versions of ci-tools
(checkmark) discovering any new versions of source
(checkmark) waiting for a suitable set of input versions
(checkmark) checking max-in-flight is not reached
<waits forever>

Fly still reports the single worker as running.

Quickfix

I always fix this in the same way. Maybe its useful information.

• stop Concourse containers: docker-compose down
• delete worker folder: rm -rf <worker folder path>
• start Concourse containers: docker-compose up -d
• wait for worker to stall (or rather, wait for Fly to report it stalled), then prune it
• everything goes back to normal

[vito]

This should end up being fixed by #1959, where volumes/containers that were hard-removed from a worker will be reaped from the ATC.

[topherbullock]

Since #1959 is done, we should verify that this is no longer an issue with docker compose
We should be able to follow each of the steps outlined by @mbjelac under What happens? and see that these are no longer happening.

[vito]

@topherbullock iirc we ended up punting on this aspect of that issue, so this is probably not actually solved yet

[topherbullock]

@vito
Oh right... I confirmed its def still an issue.
Drafting one for GC-ing records in the DB which aren't on the worker anymore.

[kcmannem]

looks like bind mounts don't persist in the mount table between docker starts and stops. Don't really know how we get around this.

[vito]

This should be somewhat mitigated by #2588, though it will take time for Concourse to confirm that the volume is gone.

For this to result in the errors you saw, it seems like Docker would have had to shut down the containers/etc. and then brought it back under the same name. Which is super weird. I've opened #2829 to suggest auto-deleting Docker-deployed workers if they're being shut down without draining.

Given the age of this issue, the fact that it's at least somewhat mitigated already, and we've got another issue proposing the next steps (which I've prioritized in the Operations backlog), I'm gonna close this one out. Thanks for the detailed report, @mbjelac!