The authorization server encountered an unexpected condition

[iocanel]

I am using a custom Jenkins image (based on openshift jenkins) inside openshift dedicated.

When I try to access jenkins, I am prompted to log in using openshift and then I get the error below.

{"error":"server_error","error_description":"The authorization server encountered an unexpected condition that prevented it from fulfilling the request.","state":"MzIxYWJjNGMtNzcyNi00"}

Steps to reproduce:

git clone https://github.com/redhat-ipaas/ipaas-ci
cd ipaas-ci
oc create -f jenkins-ephemeral
oc process redhat-ipaas-ci ROUTE_HOSTNAME=jenkins-$(oc project -q).b6ff.rh-idev.openshiftapps.com KUBERNETES_NAMESPACE=$(oc project -q) | oc create -f -

You might need to change the ROUTE_HOSTNAME.
You might also need to manually reenable the plugin, as I intend to disable it until the issue is resolved.

[gabemontero]

That log means an error occurred in the oauth code in the master server.

There are a couple of potential root causes here:

1. We've seen similar issues with openshift online in the past; a couple of times there has been bugs on the openshift api master server side (don't know if all those fixes are in dedicated yet); other times, there have been set up issues wrt auth in the online env that needed to be addressed.

We'd minimally need to have someone like @enj (who did the associated oauth changes for the jenkins log in) to look at the master logs. Most likely, opening an issue against dedicated would be needed to drive this home.

2. Of course if you don't see the issue with when running the openshift jenkins image inside openshift dedicated, we'd need to understand the differences in the images.

Among other things from what you posted, there is some config in the jenkins ephemeral template that could be of interest:

    {
      "kind": "ServiceAccount",
        "apiVersion": "v1",
        "metadata": {
            "name": "${JENKINS_SERVICE_NAME}",
            "annotations": {
		"serviceaccounts.openshift.io/oauth-redirectreference.jenkins": "{\"kind\":\"OAuthRedirectReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"Route\",\"name\":\"${JENKINS_SERVICE_NAME}\"}}"
            }
        }
    },

The service account annotation needs to be pointing to whatever route you are using for the oauth redirects to work. Based on your oc process invocation, I couldn't say for certain if there were some changes to your Route setup or not.

My feel is you need to go down the path I noted in 1), but certainly we work through 2) in parallel.

I'll keep this issue open for now until 1) and/or 2) get sufficient traction, or if something else unfolds.

@bparees FYI in case this needs attention next week while I'm out.

[iocanel]

We've also hit the issue using the openshift jenkins image.

I'll have a look at the ServiceAccount and see if it helps.

[enj]

At minimum I would need the output of oc process, logs from the master and SA details to help debug this.

[gabemontero]

Yeah certainly give the ServiceAccount thread a go, but based on the openshift jenkins image info, this is sounding more like one of the recent online bugzillas, and we'll need what @enj noted. I'll see if I can dig up some links for reference.

[gabemontero]

The bugs that most match to this are https://bugzilla.redhat.com/show_bug.cgi?id=1399022 and https://bugzilla.redhat.com/show_bug.cgi?id=1413863 ... those bugs lead to my adding some debug in the plugin that lines up with the message @iocanel posted in the description. The claim in the bug was that a move to the 3.4 jenkins image fixed the issue, but that never made sense to me. I think the env recycling as part of importing the new images may have lead to things getting resolved.

Bottom line - we need that data @enj noted to see if there is a master side bug or env like set up issue in dedicated.

One of the actual fixes I was remembering was https://bugzilla.redhat.com/show_bug.cgi?id=1421629
but at first blush that manifested a bit differently, so probably is unrelated.

[gabemontero]

Any progress on getting the master logs for @enj to look at?

[syndesisci]

FWIW, On Monday our cluster was also updated to 3.4 and I stopped seeing the issue in one of our projects. Need to check again if this also applies to the rest.

[gabemontero]

Technically speaking, openshift master must be at 3.4 for this plugin to work. All the changes made by @enj on the master side went into 3.4.

I'm going to close this out then. If this is seen on a 3.4 cluster, the master logs will be needed, and a issue/bugzilla against dedicated oauth should be opened for minimally that initial triage.

[zonArt]

Hi,

Sorry for coming back on this closed issue but it looks like I got the same kind of issue running openshift 1.5.1. I got the "Login with OpenShift" button but when I click, got the same error. I used customized image jenkins-2-centos7. Please find attached the logs from oc process -f <my template> and the definition of jenkins ServiceAcount.

Thank you in advance
oc_process.txt
SA.txt

[gabemontero]

hi @zonArt - the SA and route look OK.

There are a couple of other known gotchas. After an upgrade for example, you need to reconcile cluster roles ... see https://docs.openshift.org/latest/install_config/upgrading/manual_upgrades.html#updating-policy-definitions

Have you all already handled this?

If not, at 1.5.1, we really need your master logs when the error occurs to really see what is occurring on the master side, which is where this error comes from. @enj can most likely sort it out from those.

If the master logs will take some time to get, a list of events from the namespace might shed some light, as would the jenkins pod logs.

[zonArt]

Whoa, what a quick reply and on an already closed issue, thanks. I actually ran the reconcile cluster roles as we upgraded from 1.3 to 1.4 to 1.5 so that was not the issue. I think you can just ignore my comment, I probably missed something as I made the template from scratch and retrieved some information on the provided jenkins template provided by openshift.

After noticing if failed I decided to go the other way around and starting from the "official" template with some modification to adapt my setup (such as definition of the image to pull and adding a needed securityContext) and then everything worked as expected.

I'll now dig a little further to check what are the thing I missed from the openshift template, I got mine in yaml and the one provided is json so some adaptation are needed. Anyway thanks again for your help

[gabemontero]

glad you are at least up and running with the official template @zonArt

if you figure out what the difference was let us know

[zonArt]

You actually won't believe it, I'm a little ashamed, it was a typo in the Route definition (at least this is my guess as this is the only thing which really was different from the official template, but you'll probably confirm/infirm if it has an incidence or not):

        "metadata": {
            "annontations": {
                "template.openshift.io/expose-uri": "http://{.spec.host}{.spec.path}"
            },

Noticed the extra "n" in annontations ?

[gabemontero]

yep that is it ... if it is any consolation, I missed it too when looking at your oc process text :-)