[oc cluster] - Unable to push to internal registry

[tnozicka]

At the end of BuildConfig, pushing to internal registry fails with:

error: build error: Failed to push image: Received unexpected HTTP status: 500 Internal Server Error

Version

$ oc version
oc v1.3.0-alpha.1-208-g0effc3c-dirty
kubernetes v1.3.0-alpha.1-331-g0522e63

Steps To Reproduce

1. oc cluster up
2. oc new-project jenkins-slaves
3. oc create -f https://github.com/openshift/origin/raw/master/examples/jenkins/master-slave/jenkins-slave-template.json
4. oc process jenkins-slave-builder -v 'IMAGE_NAME=docker.io/fabric8/maven-builder,IMAGE_STREAM_NAME=maven,SLAVE_REPO_URL=https://github.com/openshift/origin.git,SLAVE_REPO_CONTEXTDIR=examples/jenkins/master-slave/slave/,SLAVE_REPO_REF=master' | oc create -f -
5. oc logs -f maven-jenkins-slave-1-build

Current Result

error: build error: Failed to push image: Received unexpected HTTP status: 500 Internal Server Error

Expected Result

silent 200

Additional Information

Investigating registry logs I have found User \"system:serviceaccount:default:registry\" cannot list limitranges in project \"jenkins-slaves\"" err.message="unknown error"

time="2016-06-09T07:20:07.671503204Z" level=error msg="response completed with error" err.code=UNKNOWN err.detail="User \"system:serviceaccount:default:registry\" cannot list limitranges in project \"jenkins-slaves\"" err.message="unknown error" go.version=go1.6 http.request.host="172.30.14.17:5000" http.request.id=a7159c1f-0a1b-4d8f-b422-fa3b799a7d7b http.request.method=PUT http.request.remoteaddr="10.34.129.146:56342" http.request.uri="/v2/jenkins-slaves/maven-jenkins-slave/blobs/uploads/dd3aacc9-16bf-4377-b7cf-3490842561ce?_state=eBZOkX3qQFyJ4C8jrL2kKH5pIVXAEv_k0MW82bxwQSt7Ik5hbWUiOiJqZW5raW5zLXNsYXZlcy9tYXZlbi1qZW5raW5zLXNsYXZlIiwiVVVJRCI6ImRkM2FhY2M5LTE2YmYtNDM3Ny1iN2NmLTM0OTA4NDI1NjFjZSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTYtMDYtMDlUMDc6MjA6MDdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.9.1 go/go1.5.3 kernel/4.5.5-201.fc23.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=16.852853ms http.response.status=500 http.response.written=329 instance.id=707499e1-7271-4ce4-8556-b80651a217a6 vars.name="jenkins-slaves/maven-jenkins-slave" vars.uuid=dd3aacc9-16bf-4377-b7cf-3490842561ce

[tnozicka]

btw. It works in ADB.

[csrwng]

@tnozicka I could not reproduce locally. Can you make sure you have the latest openshift/origin* images? (either manually pull them or delete them from your local docker).

We currently don't have a way to tell cluster up to pull the latest for you. I'll work on adding a flag that will do that.

[tnozicka]

@csrwng Yes, I had docker.io/openshift/origin:latest 5 days old and registry about 7 hours old. I have deleted all the images, but I got stuck with DNS issue, not resolving github.com from inside pods. I am not sure if it might be bug in the newer images or somewhere else, but I can resolve it from host system just fine. I will look into it tomorrow once I get back to work if it's still an issue.

I always thought oc cluster up would update the images. I would consider adopting it as default behaviour and used flag to disable the pull.

[csrwng]

@bparees

[bparees]

@tnozicka @csrwng is there still progress to be made on this?

[tnozicka]

@bparees Not sure. Last time I have updated docker images and tried the same I run into DNS issues not resolving github.com so it did not get to the push phase. It started DNS on 8053 as always, but might have some collision with ADB. Since then I have not yet have time to investigate further but I will try is again after summit.

[tnozicka]

It is working with latest images right now so this was most probably caused by not having all latest images updated.

(I have not seen the DNS issue mentioned above again after using newest oc.)

@csrwng Do we have an issue for adding the option to pull all images in oc cluster? (Preferably to pull them by default and having --no-pull=[false] option to disable it.)

[csrwng]

@tnozicka we don't have one, but it's definitely something we need to address. Please open one.