problem to pull fluent-bit:2.0.11 or higher (including 2.1.7) on old docker version.

[litvos]

Bug Report

Describe the bug
RHEL7 is distributed with an old docker 1.13.1 with their own backported patches.
I noticed that pulling fluent-bit starting from 2.0.11 including 2.1.7 I get an error
Jul 25 09:45:01 ocitest dockerd-current: time="2023-07-25T09:45:01.541975674Z" level=error msg="Attempting next endpoint for pull after error: manifest unknown: OCI index found, but Accept header does not support OCI indexes"
Jul 25 09:45:02 ocitest dockerd-current: time="2023-07-25T09:45:02.496253937Z" level=error msg="Not continuing with pull after error: Error: image fluent/fluent-bit:2.1.7 not found"

To Reproduce
[root@ocitest user]# docker pull fluent/fluent-bit:2.1.7
Trying to pull repository docker.io/fluent/fluent-bit ...
manifest for docker.io/fluent/fluent-bit:2.1.7 not found

• Example log message if applicable:

Attempting next endpoint for pull after error: manifest unknown: OCI index found, but Accept header does not support OCI indexes

• Steps to reproduce the problem:
  docker pull fluent/fluent-bit:2.1.7 on RHEL/CentOS7 with docker 1.13.1
  Expected behavior
  Pull is successful

Screenshots

Your Environment

• Version used: fluent bit 2.1.7
• Configuration: Docker 1.13.1 on RHEL7
• Environment name and version (e.g. Kubernetes? What version?): RHEL7 VM with docker 1.13.1
• Server type and version: x86
• Operating System and version: RHEL 7.9
• Filters and plugins:

Additional context

Last version which is pullable is 2.0.10. From 2.0.11 I get an error.
@patrick-stephens noticed following change:
2.1.0 includes the switch to the more recent build-push-action that turn on attestation:

fluent-bit/.github/workflows/call-build-images.yaml

Line 142 in 341ddbc

uses: docker/build-push-action@v4

Assumption is that old docker on RHEL7 expect images in the old format and is not supporting OCI properly moby/moby#39727

[patrick-stephens]

It seems to be an issue with the responses now no longer providing the old format which may be related to default including provenance now in the docker CI action.

TOKEN=$(curl "[https://auth.docker.io/token?service=registry.docker.io&scope=repository:fluent/fluent-bit:pull"|](https://auth.docker.io/token?service=registry.docker.io&scope=repository:fluent/fluent-bit:pull%22|) jq -r .token)
curl -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/fluent/fluent-bit/manifests/2.1.0
curl -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/fluent/fluent-bit/manifests/2.0.10

We need to verify if disabling the provenance option enables the legacy format.

[patrick-stephens]

Confirmed this is indeed down to the new provenance enabled setting, disabling it means images are provided in the legacy format as well.

Tested with a simple repo: https://github.com/patrick-stephens/provenance_test

Image with provenance enabled:

$ curl -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/calyptiaci/provenance_test/manifests/sha-3373fc1
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"OCI index found, but accept header does not support OCI indexes"}]}

Image with it disabled:

$ curl -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/calyptiaci/provenance_test/manifests/sha-e874453
{
   "schemaVersion": 1,
   "name": "calyptiaci/provenance_test",
   "tag": "sha-e874453",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },

[patrick-stephens]

@litvos can you confirm if you can pull docker.io/fluent/fluent-bit:2.1.8-amd64?

This is just the linux/amd64 2.1.8 image re-tagged and pushed but this seems to resolve the issue seen with curl.

I've got a fix for future builds in the PR but want to see if this works for you.

I've also pushed fluent/fluent-bit:2.1.8-arm64 for ARM64.

[litvos]

looks good. thank you @patrick-stephens!