Upgrade/Downgrade integration tests fails when base resource type stays consistent

[aoldershaw]

Summary

When the version of the mock base resource type doesn't change between the latest image and the dev image under test, the upgrade and downgrade tests fail with the following errors:

find or create container on worker ops-worker: volume '1311fba6-13ea-4d8b-7a18-746232f265b0' disappeared from worker 'ops-worker'

Steps to reproduce

$ docker pull concourse/dev
$ docker pull concourse/concourse
$ docker-compose build
$ go test -v ./integration/ops -run TestDowngrade

Expected results

Test succeeds

Actual results

Test fails with the error above

Additional context

Based on the logs, it appears as though it's failing trying to FindOrCreateVolumeForBaseResourceType:

web_1     | {"timestamp":"2021-08-09T17:19:18.225118400Z","level":"info","source":"atc","message":"atc.tracker.notify.run.check-step.find-or-create-volume-for-base-resource-type.created-volume-not-found","data":{"build":"check","build_id":9,"container":"8079a51a-02bc-4cf9-50d5-221ce9b5b213","pipeline":"test","resource":"mockery","session":"24.3.2.3.1","step-name":"mockery","team":"main","volume":"1311fba6-13ea-4d8b-7a18-746232f265b0"}}

However, although the /worker-state directory gets wiped when we recreate the worker container, the DB still thinks the volume is created.

Note that this only happens when the version for the mock base resource type stays consistent since otherwise the old worker_base_resource_type would be invalidated, as would the existing volume in the DB.

Also note that we only started to experience this now as a result of 795dccb

Triaging info

• Concourse version: 7.4.0 dev
• Did this used to work? yes

[aoldershaw]

It seems to fail on upgrade as well as just downgrade, though it will pass at least ~half the time on upgrade