Avatar images are broken on GHE Server running in private mode

[artiomle]

Pictures that needs to be retrieved from Github instance (both reviewable and github are self-hosted in this case) are blocked due to CORS.

F12 (chrome - console error)
Cross-Origin Read Blocking (CORB) blocked cross-origin response https://github.xxxxxxxx.com/login?return_to=https%3A%2F%2Fgithub.xmcyber.com%2Favatars%2Fu%2F75%3Fs%3D60%26 with MIME type text/html. See https://www.chromestatus.com/feature/5629709824032768 for more details.

[pkaminski]

Yep, sorry about that -- we're aware of this issue and trying to find a resolution with GitHub support. I believe the root cause is actually the lack of SameSite annotations on GHE's session cookies, which forces the avatar fetch into an authentication flow that then runs into CORB restrictions.

[dhbradshaw]

I submitted a discription of this issue to GHE. Hopefully we'll hear back soon.

[artiomle]

Any news regarding this issue (we are using 2.22 {latest} GitHub version)
All avatars are broken, and it's a bit confusing for large dev teams

[pkaminski]

Doug is out this week but I don't believe we ever heard back from GitHub. On the other hand, we haven't heard from any other customers about this issue either, so I'm guessing it's either due to some oddity in your configuration or perhaps a recent regression. I'll put in some time this week to repro and find a workaround, if possible.

[pkaminski]

I spent a few hours working the issue but failed to make progress. Basically, at some point GitHub Enterprise started to set session cookies to SameSite: Lax. (I didn't bother bisecting to figure out exactly when they started doing so since Chrome will now default cookies to this setting anyway, and I don't think they were ever explicitly SameSite: None.) Those cookies will hence not be included in image requests from cross-origin sites like Reviewable, and with GHE's private mode on it won't serve anything other than the login page without an active session. (Note that even when it gets redirected to the login page, Chrome remembers that this is not a top-level context so it still won't serve SameSite: Lax cookies. And of course, it refuses to read the HTML login page in lieu of the requested image -- that's the CORB part.)

I was hoping I could work around the issue by proxying the avatars through Reviewable's server but I can't find a way for the server to fetch avatars at all! There's no separate API for fetching avatar image data, and the usual API authorization header doesn't work on avatar image URLs. GitHub's session cookies are of course set as HttpOnly so there's no way to snag a copy for the server to use.

We'll push GitHub support for answers, but at this point the only workaround I can think of is to turn off private mode in GHE. The issue seems pretty fundamental -- do you have other GitHub apps installed that are successfully displaying avatars?

[dhbradshaw]

My initial submission seems to be lost in the ether. I've submitted again with through the same channel ( https://enterprise.githubsupport.com/hc/en-us/requests/new ) and also again through one that will hopefully work better ( https://support.github.com/ticket/personal/0/867191 ).

[pkaminski]

Here's the response from GitHub:

This is a known limitation of GitHub Enterprise Server; I've added your feedback to the internal issue we're tracking it under. I can't make any guarantees on a release or timeline, but keep an eye on our changelog for updates!

Looks like all you can do is put pressure on your GitHub Enterprise rep to get this issue fixed from their side. 😢 I'll mark this issue as blocked in the meantime.

[hasanirogers]

I'm running into the same issue. Any updates on this?

[artiomle]

The only thing that solved the problem for me is having both github enterprise and reviewable on same parent domain.For example reviewable.domain.congithub.domain.com In this case github is not causing the CORS problem.If you can get them on same domain, for any reason, I don't see how this issue can be solved.On 8 Mar 2021 9:42, Hasani Rogers <notifications@github.com> wrote: I'm running into the same issue. Any updates on this? —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

[apottere]

It seems likely that this is the same GHE issue as https://issues.jenkins.io/browse/JENKINS-61506. We've been having trouble with GHE avatars loading as Jenkins folder icons for a long time now, in the console there's a CORB error from a redirect to the GHE login page.

[amit1rrr]

@pkaminski Were you able to resolve this? Any official guidance from GitHub?

• Will disabling Private Mode on GHE help?
• Is putting reviewable & GHE on the same parent domain the only solution here? Does that work in all cases?

[pkaminski]

I never got this completely straightened out, and received no official guidance from GitHub beyond the default "known limitation" blurb above. I'm pretty sure that disabling private mode will help, or at least everyone who has complained about this had private mode enabled. I also believe that there's some workaround to do with domains since there are customers running in private mode who don't encounter the issue, but I don't have enough data points to discern a definite pattern.

[pkaminski]

Another customer reports that having GHE and Reviewable on the same domain resolves the issue in Chrome, but not in Safari, FWIW.

[skyzyx]

I don't use this software, but I came across this thread while trying to debug the same issue in a different project. For future users of search engines who find this thread, here's something I've noticed.

1. If your GHE login page is WWW-facing, you need to have Private Mode enabled. You don't want the whole world to see your source code.

2. If your GHE instance is running entirely behind a firewall (e.g., where only employees can access it), then you should probably be safe disabling Private Mode. Check with your IT/Ops team first before taking my word for it.

3. The core issue is that a valid authentication cookie needs to be sent in the request to GHE in order for GHE to respond with the binary image data instead of the login page HTML. Most services can't generate this cookie on behalf of users themselves.

4. I've also noticed that these images cannot be fetched using a bearer token and a PAT either (like API calls). If they worked like the API, you could generate a PAT to give to the service, and the service could make those calls on your behalf to fetch the images. But it doesn't seem to work that way.

5. Having both services on the same root domain allows them to share the cookies with one another. This is why the shared domain solution works, but different domains don't.

[pkaminski]

We implemented a fallback mode with procedurally generated avatars when we detect that avatars aren't loading. It doesn't look like GitHub is in any hurry to fix the underlying problem so this is likely as good a fix as we'll get in the foreseeable future. ☹️ Closing.