docker build using a pull secret Building from a template should create a docker build that pulls using a secret run it

[stevekuznetsov]

Extended.[builds][pullsecret][Conformance] docker build using a pull secret Building from a template should create a docker build that pulls using a secret run it

Seen here in this job

/data/src/github.com/openshift/origin/test/extended/builds/docker_pullsecret.go:48
Expected
    <bool>: false
to be true
/data/src/github.com/openshift/origin/test/extended/builds/docker_pullsecret.go:41

[bparees]

looks like a registry issue:

2016-11-22T22:59:12.736978000Z Successfully built 84ff1257879c
2016-11-22T22:59:12.760573000Z Pushing image 172.30.153.38:5000/extended-test-docker-build-pullsecret-jt7vn-saoxh/image1:latest ...
2016-11-22T22:59:17.392481000Z Registry server Address: 
2016-11-22T22:59:17.392786000Z Registry server User Name: serviceaccount
2016-11-22T22:59:17.393058000Z Registry server Email: serviceaccount@example.org
2016-11-22T22:59:17.393318000Z Registry server Password: <<non-empty>>
2016-11-22T22:59:17.393586000Z error: build error: Failed to push image: Error: Status 404 trying to push repository extended-test-docker-build-pullsecret-jt7vn-saoxh/image1: "404 page not found\n"

note that we dump some registry pod info here and the filesystem looks pretty full, but i don't see that causing a 404:

Nov 22 17:59:24.393: INFO: Running 'oc exec --config=/tmp/extended-test-docker-build-pullsecret-jt7vn-saoxh-user.kubeconfig --namespace=extended-test-docker-build-pullsecret-jt7vn-saoxh -n default docker-registry-1-z017o df --config /tmp/openshift/test-extended/core/openshift.local.config/master/admin.kubeconfig'


 df from registry pod: 
Filesystem                                                                                         1K-blocks     Used Available Use% Mounted on
/dev/mapper/docker-202:2-33888628-33b56421827c5be7467b9b65760e25093e3533d79cf8230ab8d0242a5ecd7803  10474496   411548  10062948   4% /
tmpfs                                                                                                8002656        0   8002656   0% /dev
tmpfs                                                                                                8002656        0   8002656   0% /sys/fs/cgroup
/dev/mapper/docker--vg-openshift--xfs--vol--dir                                                      5212160    33076   5179084   1% /registry
/dev/xvda2                                                                                          26202092 16463532   9738560  63% /run/secrets
shm                                                                                                    65536        0     65536   0% /dev/shm
tmpfs                                                                                                8002656       16   8002640   1% /run/secrets/kubernetes.io/serviceaccount

[miminar]

Running the extended test alone in a loop, I cannot reproduce it locally. Trying whole conformance suite now.

[miminar]

The conformance suite takes like forever on my end. I didn't manage to reproduce in a first run taking 1 and half hour. The second is running twice that time without hitting the test in question.

I'll rather focus on collecting more info in jenkins.

[miminar]

Since I failed to reproduce locally, I fixed an extended cleanup to let us collect openshift and registry log in the jenkins. Please review #12037

The referenced job terminated with:

[INFO] Stacked plot for log subset "memory" written to /tmp/openshift/test-extended/core/logs/memory.pdf
/data/src/github.com/openshift/origin/test/extended/setup.sh: line 45: cleanup_openshift: command not found

[miminar]

Is this really a P1? So far no luck hitting this in my previous 2 attempts in jenkins.

[stevekuznetsov]

@bparees @danmcp do we have some tangible guidelines for prioritizing this?

[danmcp]

@stevekuznetsov If it's not happening frequently (> 2%) then it can move to a P2.

[miminar]

I failed to reproduce in jenkins in 10 out of 10 runs. I'm postponing investigation until someone hits this.

[eparis]

Seems like I've hit this twice today...
https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/216/ is one of them...

[smarterclayton]

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_conformance_gce/127/testReport/junit/(root)/Extended/_builds__pullsecret__Conformance__docker_build_using_a_pull_secret_Building_from_a_template_should_create_a_docker_build_that_pulls_using_a_secret_run_it/

[miminar]

I wish there were registry logs available for gce runs.

[smarterclayton]

Pull requests welcome! :)

[miminar]

Here you go: #13557 merged. Next time somebody hits this, we'll have more information.

[csrwng]

New failure https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/629/

[bparees]

most recent flake didn't include the logs, so i can't determine if it was really a recurrence of this issue or not. Otherwise i've not seen/heard evidence that this is still an issue. Closing and someone can open a new flake w/ logs if it's still happening.