Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k8s.gcr.io/pause:3.1 images does not match gcr.io/google-containers/pause:3.1 image #63057

Closed
mkumatag opened this issue Apr 24, 2018 · 14 comments
Closed

k8s.gcr.io/pause:3.1 images does not match gcr.io/google-containers/pause:3.1 image #63057

mkumatag opened this issue Apr 24, 2018 · 14 comments
Assignees
@zmerlynn @thockin
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/release Categorizes an issue or PR as relevant to SIG Release.

Comments

@mkumatag
Copy link
Member

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
k8s.gcr.io/pause:3.1 images does not match gcr.io/google-containers/pause:3.1 image

$ manifest-tool inspect k8s.gcr.io/pause:3.1
k8s.gcr.io/pause:3.1: manifest type: application/vnd.docker.distribution.manifest.v2+json
      Digest: sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
Architecture: amd64
          OS: linux
    # Layers: 1
      layer 1: digest = sha256:67ddbfb20a22d7c0ea0df568069e7ffc42378467402d04f28ecfa244e78c5eb8

~ ⌚ 10:24:31
$ manifest-tool inspect gcr.io/google-containers/pause:3.1
Name:   gcr.io/google-containers/pause:3.1 (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
 * Contains 5 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
1  Mfst Length: 527
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant:
1           - Feature:
1     # Layers: 1
         layer 1: digest = sha256:67ddbfb20a22d7c0ea0df568069e7ffc42378467402d04f28ecfa244e78c5eb8

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:c84b0a3a07b628bc4d62e5047d0f8dff80f7c00979e1e28a821a033ecda8fe53
2  Mfst Length: 527
2     Platform:
2           -      OS: linux
2           - OS Vers:
2           - OS Feat: []
2           -    Arch: arm
2           - Variant:
2           - Feature:
2     # Layers: 1
         layer 1: digest = sha256:7fe7c69563d350959054c17c0ab3ed8b43e7b93b2056cbfb956e20f12c20f6cb

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:f365626a556e58189fc21d099fc64603db0f440bff07f77c740989515c544a39
3  Mfst Length: 527
3     Platform:
3           -      OS: linux
3           - OS Vers:
3           - OS Feat: []
3           -    Arch: arm64
3           - Variant:
3           - Feature:
3     # Layers: 1
         layer 1: digest = sha256:71bc2f389bec9e26f8f4ffa1a7ccde852a2322e13f20944b68be00ee7bcf31b8

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:bcf9771c0b505e68c65440474179592ffdfa98790eb54ffbf129969c5e429990
4  Mfst Length: 527
4     Platform:
4           -      OS: linux
4           - OS Vers:
4           - OS Feat: []
4           -    Arch: ppc64le
4           - Variant:
4           - Feature:
4     # Layers: 1
         layer 1: digest = sha256:597c4e8bfd5eb55ef7a8c6dcbfbad9ee6ac9392c2f983c2b5498384ee2a7b16a

5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:882a20ee0df7399a445285361d38b711c299ca093af978217112c73803546d5e
5  Mfst Length: 527
5     Platform:
5           -      OS: linux
5           - OS Vers:
5           - OS Feat: []
5           -    Arch: s390x
5           - Variant:
5           - Feature:
5     # Layers: 1
         layer 1: digest = sha256:cbd367e766809719cacec04bab7f65980ef98791c98c258085876cbdc8472a02

This seems like k8s.gcr.io/pause:3.1 is not mapped properly to gcr.io/google-containers/pause:3.1 because of this pods are failing to come up. So this need to be fixed.
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. kind/bug Categorizes issue or PR as related to a bug. labels Apr 24, 2018
@mkumatag
Copy link
Member Author

/sig cluster-lifecycle

@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 24, 2018
@mkumatag
Copy link
Member Author

/cc @luxas @ixdy @thockin

@mkumatag mkumatag mentioned this issue Apr 24, 2018
Closed
@nitkon nitkon mentioned this issue Apr 24, 2018
Closed
@dims
Copy link
Member

dims commented Apr 24, 2018

cc @verb @yujuhong

Do we know who uploaded the containers to k8s.gcr.io/pause:3.1 ?

@dims
Copy link
Member

dims commented Apr 24, 2018

I think i know what happened... who every uploaded the pause container to k8s.gcr.io did not use all-push (https://github.com/kubernetes/kubernetes/blob/master/build/pause/Makefile#L61) and probably used push. So just the amd64 image got uploaded.

Hoping someone like @BenTheElder or @krzyzacy can push the other images

/sig testing

@k8s-ci-robot k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Apr 24, 2018
@BenTheElder
Copy link
Member

/cc @jpbetz @ixdy
/sig release
/remove-sig testing

@k8s-ci-robot k8s-ci-robot added sig/release Categorizes an issue or PR as relevant to SIG Release. and removed sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Apr 24, 2018
@ixdy
Copy link
Member

ixdy commented Apr 24, 2018

cc @listx

@ixdy
Copy link
Member

ixdy commented Apr 24, 2018

My understanding is that k8s.gcr.io is (currently) basically just an alias for gcr.io/google-containers, though I'm not entirely sure how it's implemented. I wonder if that redirect isn't working correctly with manifest lists?

@ixdy
Copy link
Member

ixdy commented Apr 24, 2018

/assign @thockin
who I think is the most knowledgeable about the k8s.gcr.io setup

@ixdy
Copy link
Member

ixdy commented Apr 24, 2018

I think this might be a bug in the process we use to replicate images to the regional gcr.io mirrors.
Compare https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/pause with e.g. https://console.cloud.google.com/gcr/images/google-containers/EU/pause.

@ixdy
Copy link
Member

ixdy commented Apr 24, 2018

/assign @zmerlynn

@ixdy
Copy link
Member

ixdy commented Apr 25, 2018

Thanks to @zmerlynn this should be fixed now.

$ manifest-tool inspect k8s.gcr.io/pause:3.1
Name:   k8s.gcr.io/pause:3.1 (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
 * Contains 5 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
1  Mfst Length: 527
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant:
1           - Feature:                            
1     # Layers: 1                                 
         layer 1: digest = sha256:67ddbfb20a22d7c0ea0df568069e7ffc42378467402d04f28ecfa244e78c5eb8
                                                  
2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:c84b0a3a07b628bc4d62e5047d0f8dff80f7c00979e1e28a821a033ecda8fe53
2  Mfst Length: 527                               
2     Platform:                                   
2           -      OS: linux                      
2           - OS Vers:                            
2           - OS Feat: []                         
2           -    Arch: arm                        
2           - Variant:                            
2           - Feature:                            
2     # Layers: 1                                 
         layer 1: digest = sha256:7fe7c69563d350959054c17c0ab3ed8b43e7b93b2056cbfb956e20f12c20f6cb
                                                  
3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:f365626a556e58189fc21d099fc64603db0f440bff07f77c740989515c544a39
3  Mfst Length: 527                               
3     Platform:                                   
3           -      OS: linux                      
3           - OS Vers:                            
3           - OS Feat: []                         
3           -    Arch: arm64                      
3           - Variant:                            
3           - Feature:                            
3     # Layers: 1                                 
         layer 1: digest = sha256:71bc2f389bec9e26f8f4ffa1a7ccde852a2322e13f20944b68be00ee7bcf31b8
                                                  
4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:bcf9771c0b505e68c65440474179592ffdfa98790eb54ffbf129969c5e429990
4  Mfst Length: 527                               
4     Platform:                                   
4           -      OS: linux                      
4           - OS Vers:                            
4           - OS Feat: []                         
4           -    Arch: ppc64le                    
4           - Variant:                            
4           - Feature:                            
4     # Layers: 1                                 
         layer 1: digest = sha256:597c4e8bfd5eb55ef7a8c6dcbfbad9ee6ac9392c2f983c2b5498384ee2a7b16a
                                                  
5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:882a20ee0df7399a445285361d38b711c299ca093af978217112c73803546d5e
5  Mfst Length: 527                               
5     Platform:                                   
5           -      OS: linux                      
5           - OS Vers:                            
5           - OS Feat: []                         
5           -    Arch: s390x                      
5           - Variant:                            
5           - Feature:                            
5     # Layers: 1                                 
         layer 1: digest = sha256:cbd367e766809719cacec04bab7f65980ef98791c98c258085876cbdc8472a02

@ixdy ixdy closed this as completed Apr 25, 2018
@nitkon nitkon mentioned this issue Apr 26, 2018
Closed
@ArchiFleKs
Copy link

I still get this issue when trying to join a cluster on my raspberry pi3 with kubeadm.

I'm on archlinux arm and using containerd

mai 26 09:07:45 raiden containerd[410]: time="2018-05-26T09:07:45+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-proxy-r8mtg,Uid:42c9a762-60b3-11e8-883a-0025900eb312,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image "k8s.gcr.io/pause:3.1": failed to pull image "k8s.gcr.io/pause:3.1": failed to get image information: failed to get image diffIDs: manifest sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea: not found"

@ArchiFleKs ArchiFleKs mentioned this issue May 28, 2018
Merged
@ixdy
Copy link
Member

ixdy commented May 29, 2018

Does containerd support manifest lists?

I'm not sure what's going on with that error message, because that manifest (sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea) does exist on gcr.io.

@BenTheElder
Copy link
Member

BenTheElder commented May 29, 2018 •
edited

@Random-Liu ^
edit: I'd expect it to, since docker is now built on top of containerd, but I'm not positive, so CCing Lantao

@Kaylleur Kaylleur mentioned this issue Oct 21, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zmerlynn zmerlynn

@thockin thockin

Labels
kind/bug Categorizes issue or PR as related to a bug. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/release Categorizes an issue or PR as relevant to SIG Release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@dims @BenTheElder @ArchiFleKs @zmerlynn @thockin @ixdy @mkumatag @k8s-ci-robot