Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
'base resource type not found' errors starting with 3.9.1+ #2081
Starting with release 3.9.1, running concourse via http://concourse-ci.org/docker-repository.html, after loading a pipeline, we see all resources turn orange with the error base resource type not found.
Digging through the logs, we see:
concourse-db: (looks the same in 3.9.0)
If we edit the docker-compose to point to v3.9.0 (or 3.8.0), these symptoms go away. This is confirmed on three diff deployed servers we have so far running concourse.
We have 300+ servers deployed running concourse that get rebuilt to the newest version of concourse every couple of months, so this will become a big issue for us soon and we need to know if we should update the server build template to peg to 3.9.0 to avoid widespread issues.
This is an example of how the resource definition looks for one of the git resources failing:
resources: - name: HelloWorld type: git webhook_token: <redacted> source: uri: git@<redacted>.git branch: master private_key: ((github-private-key-value)) check_every: 24h
This is a sanitized example of the
version: '3' services: concourse-db: image: postgres:9.6 restart: unless-stopped environment: POSTGRES_DB: concourse POSTGRES_USER: concourse POSTGRES_PASSWORD: <redacted> PGDATA: /database concourse-web: image: concourse/concourse links: [concourse-db] command: web depends_on: [concourse-db] ports: ["80:8080"] volumes: ["./keys/web:/concourse-keys"] restart: unless-stopped # required so that it retries until concourse-db comes up environment: CONCOURSE_EXTERNAL_URL: "<redacted>" CONCOURSE_GITHUB_AUTH_CLIENT_ID: "<redacted>" CONCOURSE_GITHUB_AUTH_CLIENT_SECRET: "<redacted>" CONCOURSE_GITHUB_AUTH_ORGANIZATION: "<redacted>" CONCOURSE_GITHUB_AUTH_AUTH_URL: "https://<redacted>/login/oauth/authorize" CONCOURSE_GITHUB_AUTH_TOKEN_URL: "https://<redacted>/login/oauth/access_token" CONCOURSE_GITHUB_AUTH_API_URL: "https://<redacted>/api/v3/" CONCOURSE_PUBLICLY_VIEWABLE: "true" CONCOURSE_POSTGRES_HOST: concourse-db CONCOURSE_POSTGRES_USER: concourse CONCOURSE_POSTGRES_PASSWORD: <redacted> CONCOURSE_POSTGRES_DATABASE: concourse CONCOURSE_ENCRYPTION_KEY: "<redacted>" CONCOURSE_VAULT_URL: "<redacted>" CONCOURSE_VAULT_AUTH_BACKEND: "approle" CONCOURSE_VAULT_AUTH_PARAM: "<redacted>" concourse-worker: image: concourse/concourse privileged: true links: [concourse-web] depends_on: [concourse-web] command: worker restart: unless-stopped volumes: ["./keys/worker:/concourse-keys", "/opt/concourse/:/opt/concourse/"] environment: CONCOURSE_TSA_HOST: concourse-web CONCOURSE_WORK_DIR: "/opt/concourse/" https_proxy: "<redacted>" http_proxy: "<redacted>" no_proxy: "<redacted>"
After trying different things, we have observed another repeatable pattern with this issue:
Exploring creative ways to 'delay' the concourse-worker startup in the docker-compose but no simple path forward for this so far (the
The other thing we've observed is that when starting up a fresh concourse deployment via the
Indeed, if we simply
In other words:
Did something change w/respect to how the worker operates since 3.9.0 that could explain this change in behavior?
We can alter our server bootstrapping script to introduce the sleep and associated restart of the worker, but this approach seems like a hack masking a real problem. In addition, it puts additional burden on the users of the server to remember to restart the worker any time they need to blow away and restart the concourse stack.