Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v2.0.x containerd docker remote bug: sometimes gets "cannot reuse body, request must be retried" #122

Closed
thockin opened this issue Jun 24, 2021 · 12 comments · Fixed by #159
Closed

v2.0.x containerd docker remote bug: sometimes gets "cannot reuse body, request must be retried" #122

thockin opened this issue Jun 24, 2021 · 12 comments · Fixed by #159

Comments

@thockin
Copy link

thockin commented Jun 24, 2021 

manifest-tool                                         \
    --username=oauth2accesstoken                      \
    --password=$(gcloud auth print-access-token)     \
    push from-args                                    \
    --platforms "$platforms"                         \
    --template gcr.io/k8s-staging-git-sync/git-sync:v3.3.3__OS_ARCH \
    --target gcr.io/k8s-staging-git-sync/git-sync:v3.3.3
FATA[0004] Error pushing manifest list/index to registry: sha256:343fd4cd9582fc02b98d6641aa29356d5986ab1a31de5c5b1d0080815e083cbe: failed commit on ref "manifest-sha256:fcb67f390ed38b64a54c7c26bb72c3494d6c269c7389add0448d0dde8b510d58": cannot reuse body, request must be retried

Syncing to v1.0.3 works, but since this doesn't use go modules, that's a PITA :)
@estesp
Copy link
Owner

estesp commented Jun 24, 2021

Hmm, strange. I tested v2.0.0-beta.0 with GCR, but my normal test image uses repos not tags for differentiating the platforms. I'll have to test with tags in case that changes behavior in some way.

Just tried again to make sure:

$ ./manifest-tool --version
./manifest-tool version 2.0.0-beta.0 (commit: ff5aefe0f43b89efc21373c5514ef98a62c63e8a)
[long push from-args command]
.... response.status="201 Created" size=966 url="https://gcr.io/v2/[REDACTED]/alpine/manifests/latest"
Digest: sha256:493661ff8ba145bbe641dc56c6da3071b7761dbb22cd42fdf15b31b6f170eaa8 966

@estesp
Copy link
Owner

estesp commented Sep 15, 2021

Any chance you've been able to try this again? Especially now that maybe you are using v2 :) I have never been able to reproduce an issue pushing to GCR.

@thockin
Copy link
Author

thockin commented Sep 15, 2021 via email

@estesp
Copy link
Owner

estesp commented Feb 14, 2022

Just released v2.0.0 and it seems to be working fine with GCR in my tests. Going to close this for now but feel free to re-open if you are able to reproduce.

@estesp estesp closed this as completed Feb 14, 2022
@thockin
Copy link
Author

thockin commented Mar 11, 2022

I'm baaaaaack

platforms=$(echo linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/s390x | sed 's/ /,/g');  \
manifest-tool                                         \
    --username=oauth2accesstoken                      \
    --password=$(gcloud auth print-access-token)     \
    push from-args                                    \
    --platforms "$platforms"                         \
    --template gcr.io/k8s-staging-git-sync/git-sync:v3.5.0__OS_ARCH \
    --target gcr.io/k8s-staging-git-sync/git-sync:v3.5.0
FATA[0004] Error pushing manifest list/index to registry: sha256:98821a216b62b6ee5dccdc2682c87f85d64e905d61872d0ab19073f9c4463623: failed commit on ref "manifest-sha256:6850b0792b249e2562a2b622e1938cb35747174030aef27d5274319811d9f150": cannot reuse body, request must be retried

I have no idea where to even start

@thockin thockin mentioned this issue Mar 11, 2022
Closed
@thockin
Copy link
Author

thockin commented Mar 11, 2022

FWIW manifest-tool 1.0.3 still works:

platforms=$(echo linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/s390x | sed 's/ /,/g');  \
manifest-tool                                         \
    --username=oauth2accesstoken                      \
    --password=$(gcloud auth print-access-token)     \
    push from-args                                    \
    --platforms "$platforms"                         \
    --template gcr.io/k8s-staging-git-sync/git-sync:v3.5.0__OS_ARCH \
    --target gcr.io/k8s-staging-git-sync/git-sync:v3.5.0
Digest: sha256:d16f5b2bca94cdbb4e40b256bfe639450a6f0577dbd8b3fcaf126a2261822fcd 1665

@estesp
Copy link
Owner

estesp commented Mar 11, 2022

You may not be alone; looks like this might be a codepath my testing just isn't hitting (and we have a recent report in containerd proper as well); also #156 (comment)

In the #156 issue, I see a 404 response that my tries with Quay aren't getting, and that potentially is trying to reuse the body on the response asking for auth? Need to dig deeper now that it doesn't seem to be a random one-off.

@estesp estesp reopened this Mar 11, 2022
@estesp estesp changed the title HEAD is broken v2.0.x containerd docker remote bug: sometimes gets "cannot reuse body, request must be retried" Mar 11, 2022
@estesp estesp mentioned this issue Mar 11, 2022
Closed
@thockin
Copy link
Author

thockin commented Mar 11, 2022 via email

@estesp
Copy link
Owner

estesp commented Mar 14, 2022

Thanks; just found a reproduce scenario and found that, aside from the fact containerd is working on fixing the "body reuse" problem, there is a different flow of registry interaction when the source repos are != target repo (e.g. use "names" not "tags" to differentiate OS/arch).

When all source images are in the same repo namespace as the target manifest list, that flow ends in a 404 existence check on the final target name:tag which appears to cause a 401 Unauthorized, which leads to the retry with same body error in the auth flow. Digging into that now; I think it's related to registry token scopes (which are set at the library/auth level, not in my code, but need to find out why it's causing different behavior)

@estesp estesp mentioned this issue Mar 15, 2022
Merged
@estesp
Copy link
Owner

estesp commented Mar 15, 2022

Not sure if you have the chance to test the changes in #159, but I'm 99% convinced this will workaround the issue until containerd has a solution to the req/resp retry with a 401. I've verified it fixes the same issue hit on Quay. Interestingly it requires that you have a public repo for all the sources; if your repo is private or any of the sources come from a private repo, then you don't hit the bug as the 401 auth challenge comes during a "fetch" which means you don't get the body reuse error.

@thockin
Copy link
Author

thockin commented Mar 15, 2022 via email

@thockin
Copy link
Author

thockin commented Oct 11, 2022 via email

@rrjjvv rrjjvv mentioned this issue Jan 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Workaround for body reuse/401 auth errors estesp/manifest-tool
2 participants
@estesp @thockin