While watching a job, we get a " You are not authorized to view the details of this pipeline" error

[aeijdenberg]

Bug Report

After updating from 5.0.1 to 5.1.0 we've observed that when using the UI to watch a running job, then after waiting a period of time (one user reported 30 seconds, I just observed 180 seconds) s the UI shows a 401 Unauthorized - You are not authorized to view the details of this pipeline error message.

Steps to Reproduce

View a running job and watch it.

Expected Results

To watch it uninterrupted by error messages.

Actual Results

Error message appears after 180 seconds (though we'd had varying reports of this time length).

Additional Context

We login to our Concourse with UAA.
We have an HAProxy in front of our Concourse.

Version Info

• Concourse version: 5.1.0 - this previously worked fine with 5.0.1.
• Deployment type (BOSH/Docker/binary): bosh
• Infrastructure/IaaS: AWS
• Browser (if applicable): Chrome
• Did this used to work? Yes

[aeijdenberg]

cc @gordcorp

[aeijdenberg]

I'll note a few more observations - the time appears a little random. This morning I watched a job and it gave me the error after 2.5 minutes, then after page refresh, 1.45.

Not that I'd expect it to make a difference, but quite some time ago we moved our Concourse instance off the public internet, and only accessible via a SOCKS5 proxy. At about that time we also noticed that sometimes when watching a job, the output streaming from the server (if not changing frequently) would appear to stall and we wouldn't get any updates. A page refresh would fix.

I'm wondering if the now "401 Unauthorized" message is perhaps a different manifestation of an issue that previously failed a bit more silently?

[aeijdenberg]

Hmm, at the same time the error appears on screen, Chrome console shows:

GET https://concourse.example.com/api/v1/builds/2113698/events net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)
... then, a little while later, after the error has already appeared ...
GET https://concourse.example.com/api/v1/builds/2113698/events 504 (Gateway Time-out)

[biodigitalfish]

We're getting the same error

[nterry]

Same here...

[aeijdenberg]

@vito for visibility - looks like a bunch of us are seeing this issue since the v5.1.0 release last week.

[cirocosta]

cc @pivotal-jwinters

[jamieklassen]

I'm marking this as web-ui, ans I suspect it is related to changes in the
build events code -- I'm not certain, but I believe that back when we used an
effect module (a feature that was removed in Elm 0.19) there was a slightly
more resilient behaviour. I think the browser would attempt to re-open the
stream of build events on an error. Probably worth investigating, anyway.

[aeijdenberg]

I wonder if it's related to the HAProxy we have in front. I note it has the following timeouts:

    timeout connect         5000ms
    timeout client          30000ms
    timeout server          30000ms
    timeout tunnel          3600000ms
    timeout http-keep-alive 500ms
    timeout http-request    5000ms
    timeout queue           30000ms

I wonder if 30 seconds of inactivity (e.g. long running job with no new log lines in that period) is causing HAProxy to kill the connection?

I wonder if we could convince HAProxy that this connection is a tunnel (and thus apply the longer timeout): https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-timeout%20tunnel

[romangithub1024]

we are hitting the same problem.

[farukhkhan123]

we have noticed the same problem viewing logs of a running pipeline job...timing of the error is fluctuates b/w 1-2 mins

[aeijdenberg]

By way of update, we changed the timeout server 30000ms setting in our HAProxy to be 1 hour, and that seems to have worked around the issue we were seeing.

[avanier]

Yeah, we're getting the same issue over here.

Our webs runs in Kubernetes behind an Amazon ELB Classic.

cc @Typositoire

[nterry]

By way of update, we changed the timeout server 30000ms setting in our HAProxy to be 1 hour, and that seems to have worked around the issue we were seeing.

Great, how do we do that. Is there a new docker image in the works to address this?

[aeijdenberg]

In our case our HAProxy config was fairly custom to begin with, so no shared Docker image to update.

My feeling is that a more correct fix would be make the stream look more like a tunnel (e.g. websocket is hinted at in the HAProxy documentation, which by default attracts a much higher timeout) - and/or put some kind of heartbeat in so that large periods of expected inactivity don't drop the connection... (and/or try reconnecting if it drops).

[connorpwilliams]

Adding on to this, same here, bumped from 4.2.2 to 5.1.0, when looking at a specific build in a job, after some time of being idle the screen changes to "401 unauthorized" - I haven't poked into it much to see when the time occurs. A simple page refresh goes back to watching the build for another set of time. Appears to only happen while idle.

As per request below:
Firefox Quantum - 66.0.3 (64-bit)
(Also happened on Chrome but I am unable to access that machine so I don't have specific version)

[vito]

Saw this myself a couple times just now when viewing PR builds. The console logs showed ERR_NETWORK_CHANGED in my case. Which is odd because I'm on ethernet, but maybe this was happening before too and the front-end code was just more resilient to it so I didn't notice? 🤔 Haven't seen it happen since I've started paying more attention to it...

We don't have any HAProxy and for me the error happened pretty much immediately after opening the page, so my scenario sounds slightly different. If anyone can get this down to a reliable repro case that'd help a lot! 👍

Also everyone please report your exact browser version just in case there's been some behavior change in recent Chrome versions or something.

[avanier]

I'm reporting from :

Mozilla Firefox 66.0.3

[vito]

@pivotal-jamie-klassen Looks like it might be this bit of code?:

concourse/web/elm/src/Build/Build.elm

Lines 357 to 365 in 6261dbd

	newModel =
	if
	List.map .data envelopes
	|> List.member STModels.NetworkError
	then
	{ model | authorized = False }
	else
	model

Maybe it needs to be made conditional on eventSourceOpened like it is here?:

concourse/web/elm/src/Build/Build.elm

Lines 380 to 388 in 6261dbd

	( if eventSourceOpened then
	-- connection could have dropped out of the blue;
	-- just let the browser handle reconnecting
	model
	else
	-- assume request was rejected because auth is required;
	-- no way to really tell
	{ model | authorized = False }

[biodigitalfish]

Chrome Version 74.0.3729.131 (Official Build) (64-bit)

[Typositoire]

@vito In our case I'm always getting net::ERR_INCOMPLETE_CHUNKED_ENCODING in the console when this happens.

And it happens when trying to get [...]/events (eventSource)

[Maghil]

getting this error both in chrome and edge both v98
also the pipeline has set-pipeline: self job in it and also a lot of them