fetching the image did not produce a volume

[acarsercan]

Summary

Getting error

Pulling troykinsella/concourse-fly-resource@sha256:ad6e997c29fde4e50a455918cea8e46e32ba061ec5570d01ca09ab01e5f3d2e6 (attempt 3 of 3)...

Failed to pull image troykinsella/concourse-fly-resource@sha256:ad6e997c29fde4e50a455918cea8e46e32ba061ec5570d01ca09ab01e5f3d2e6.

fetching the image did not produce a volume

Steps to reproduce

Trying different resource tag
example

- name: fly
  source:
    repository: troykinsella/concourse-fly-resource
    tag: latest
  type: docker-image

Expected results

The step to download the new image as tagged

Actual results

Error above

Additional context

Triaging info

• Concourse version: 6.3.0
• Browser (if applicable): firefox
• Did this used to work? yes in 5.8.0

Workaround

Renamed the resource
example

- name: fly
```
```
- name: flyy
```

[analytically]

@jamieklassen same issue I was seeing with 6.3.0

[analytically]

After downgrading to 6.2.0, and destroying/creating the pipeline (exact same config) it works again.

[analytically]

Just downgrading doesn't work, something must be stuck in the DB.

[xtreme-sameer-vohra]

Hey @acarsercan
I'll try to reproduce the issue shortly on my end.

[xtreme-sameer-vohra]

Hey @acarsercan , @analytically
I wasn't able to reproduce the issue following these steps

1. Ran v6.3.0 of Concourse using Docker with 1 web and 1 worker
2. Set pipeline

resources:
- name: fly
  source:
    repository: troykinsella/concourse-fly-resource
    tag: latest
  type: docker-image

jobs:
- name: test
  plan:
  - get: fly

3. Ran 2 builds and they both were able to fetch the image

Did I miss any steps ?

[jamieklassen]

From what I can tell and my reasoning below, it seems like @acarsercan must have had a pipeline where the fly resource was being used as a resource_type, not as a raw docker-image resource. Do you know what I mean? I think it makes sense since the image being referred to (troykinsella/concourse-fly-resource) is a resource type implementation

[jamieklassen]

Furthermore it's notable that the particular digest in his logs doesn't seem to exist on docker hub, i.e. I get

$ docker pull troykinsella/concourse-fly-resource@sha256:ad6e997c29fde4e50a455918cea8e46e32ba061ec5570d01ca09ab01e5f3d2e6
Error response from daemon: manifest for troykinsella/concourse-fly-resource@sha256:ad6e997c29fde4e50a455918cea8e46e32ba061ec5570d01ca09ab01e5f3d2e6 not found: manifest unknown: manifest unknown

[jamieklassen]

so then the more pressing question would be -- why did the underlying resource ever discover that digest? did it actually exist on dockerhub at some point and get removed?

[acarsercan]

Unfortunately my route to the problem wasn't the same and I'm not sure if its reproducible, but here it goes anyways

Existing setup

Concourse 5.8
Existing pipeline using the fly resource

Upgrade to 6.3 using helm
Fly step no longer succeeds and errors with:

Failed to pull image troykinsella/concourse-fly-resource@sha256:ad6e997c29fde4e50a455918cea8e46e32ba061ec5570d01ca09ab01e5f3d2e6.

fetching the image did not produce a volume

We have only seen this with the fly resource

Unfortunately I don't have a debug output, so happy for this to be closed until I can get more details

[shyamz-22]

@xtreme-sameer-vohra We are facing this issue too in 6.3.0 for different resources and multiple teams have reported this. What can we do to provide more insights on this? At the moment it is tough for us to reproduce. when it does happen can you point out where we should have a look and which logs will be helpful for you to fix this issue?

[jamieklassen]

From the analysis I just did below, I think you could at least monitor how often this is happening by looking for logs where the message includes both init-image and failed-to-fetch-resource.

[jamieklassen]

@shyamz-22 can you analyze the builds where you have encountered this error? Does it consistently appear on get and put steps for custom resource types based on docker-image or task steps where the image_resource is of type docker-image?

[jamieklassen]

Converted to a discussion following https://blog.concourse-ci.org/nobody-wants-a-stale-bot/#an-update-on-triage until we can figure out a reproducible. I can contribute a bit to the conversation -- the message fetching the image did not produce a volume appears in only one place in the concourse v6.3.0 codebase: the atc/worker/image package:

concourse/atc/worker/image/image_resource_fetcher.go

Line 27 in c4261d9

var ErrImageGetDidNotProduceVolume = errors.New("fetching the image did not produce a volume")

The linked error only appears in one place, which is where the image resource fetcher didn't error, but didn't produce a volume:

concourse/atc/worker/image/image_resource_fetcher.go

Line 205 in c4261d9

return nil, nil, nil, ErrImageGetDidNotProduceVolume

Reading through the unit tests it seems like this is intended to be a catch-all error case, so it sorta makes sense that it's hard to reproduce and reason about:

• Image when no version is specified when initializing the Check resource works when check returns a version when saving the version in the database succeeds when fetching resource succeeds when the resource has a volume when the resource still does not have a volume for some reason [It] returns an appropriate error
• Image when a version is specified when saving the version in the database succeeds when fetching resource succeeds when the resource has a volume when the resource still does not have a volume for some reason [It] returns an appropriate error

[jamieklassen]

And I just worked through a bunch of code paths while reading the code, and came upon this one. It seems like a smoking gun:

concourse/atc/worker/fetch_source.go

Lines 173 to 179 in c4261d9

	// TODO: Is this compatible with previous behaviour of returning a nil when error type is NOT ErrResourceScriptFailed
	if failErr, ok := err.(runtime.ErrResourceScriptFailed); ok {
	return GetResult{
	ExitStatus: failErr.ExitStatus,
	}, nil, nil
	}

[jamieklassen]

This is consistent with @acarsercan's logs, which include the message from the docker-image resource itself saying 'Failed to pull image ...'.

All this to say that the problem, to me, seems related to the /opt/resource/in script failing on the base resource type. In @acarsercan's case, at least, this is the docker-image resource.

[jamieklassen]

@acarsercan can you give a complete pipeline where this problem was happening?

[acarsercan]

Sure...early apologies in if I was doing wrong

resource_types:
- name: flyy
  type: docker-image
  source:
    repository: troykinsella/concourse-fly-resource
    tag: latest

resources:
- name:test
  source:
    debug: false
    password: ((concourse_local_password))
    target: local
    team: sre
    url: https://xxxx
    username: concourse
  type: flyy

jobs:
- name: promote-to-test
  on_success:
    do:
      - params:
          options: trigger-job -j sre-test/integration
        put: test
  plan:
  - in_parallel:
      steps:
...

[jamieklassen]

In your original post you mention

Trying different resource tag

What did this pipeline look like before? Was the only change renaming the resource type to flyy, or was there a different tag before as well?

[acarsercan]

Same pipeline but different name to fly resource

[jamieklassen]

Hearing about @analytically encountering this on a v6.2.0->v6.3.0 upgrade, I have to wonder if this change to the checking behaviour of the docker-image resource could be related:

concourse/docker-image-resource@d3b914b

since:

$ docker run --rm --entrypoint cat concourse/concourse:6.2.0 /usr/local/concourse/resource-types/docker-image/resource_metadata.json
{
  "type": "docker-image",
  "version": "1.3.1",
  "privileged": true,
  "unique_version_history": false
}
$ docker run --rm --entrypoint cat concourse/concourse:6.3.0 /usr/local/concourse/resource-types/docker-image/resource_metadata.json
{
  "type": "docker-image",
  "version": "1.3.3",
  "privileged": true,
  "unique_version_history": false
}

[jamieklassen]

@acarsercan @analytically the only working theory I have at this point is that an image with SHA ad6e997c29fde4e50a455918cea8e46e32ba061ec5570d01ca09ab01e5f3d2e6 once existed on dockerhub but was removed -- I imagine concourse had it cached, but somehow because the source format of the docker-image resource changed due to the upgrade it had to re-pull. And maybe there's a bug somewhere in the code around resource types where they might not be re-checking frequently enough so that it is never noticed that the version no longer exists. Or maybe the check script for the docker-image resource doesn't notify when versions disappear...

[acarsercan]

I just got the same error for different resource, here is what I did

I changed

- name: slack-notifier
  source:
    repository: mockersf/concourse-slack-notifier
  type: docker-image

To

- name: slack-notifier
  source:
    repository: arbourd/concourse-slack-alert-resource
  type: docker-image

Guessing same reason, docker image was there and isn't now?

[jamieklassen]

Just noticed we've hit this issue with custom resources in our own cluster, even though the resource is a registry-image: https://ci.concourse-ci.org/teams/main/pipelines/release-5.5.x/jobs/bosh-check-props/builds/280

[jamieklassen]

@acarsercan @analytically @shyamz-22 we've developed a pretty consistent theory about these use cases which I'm currently writing up as a bug. In the meantime if you find yourself stuck in one of these state where a custom resource's puts are consistently attempting to run on invalid images, you should be able to get unstuck by running fly check-resource-type on the custom type in question. This feels like a more idiomatic workaround than modifying the resource's name/config.

EDIT: the bug - #5872

[analytically]

fly check-resource-type indeed resolved it for me, thanks Jamie!