x/pkgsite: stdlib source links are very slow to load (cs.opensource.google)

[benhoyt]

As I posted on golang-nuts, the source links for the standard library source code on pkg.go.dev are very slow -- at least for me down here (in Christchurch, New Zealand).

As shown in the attached Chromium dev tools images (see screenshots below), cs.opensource.google links take about 10 seconds to show the source code with a cold cache, and about 5 seconds with a warm cache. This is very slow in absolute terms, and almost enough time for the TL;GU ("too slow; give up") response to kick in. :-) It's also very slow in relative terms: see the attached screenshot showing GitHub.com taking 2 seconds to load the same source code.

Diving into the screenshots in a bit more detail:

1. The cs.opensource.google page timings: as you can (barely) see from the tiny screenshots, it goes through three loading states: a blank page with "loading" shows after about 3s, then the frame with sidebar shows after about 6s, but the source code itself isn't visible till about 10s.
2. The github.com page timings: a single loading state, and the source code is visible after about 2s.

I have a reasonably fast internet connection: 40ms ping time to google.com, wherever that's coming from, and about 50Mbits/s up and down. Even relatively big sites with lots of pictures, like nytimes.com, load in about 2-3 seconds with a cold cache. So while I'm sure latencies are bigger down here, the internet in New Zealand is not slow in general -- this is very much a cs.opensource.google thing. Anecdotally, a friend in Australia got timings like 5-7 seconds for cs.opensource.google. And Dan Kortschak, author of gonum, replied on golang-nuts that "This has been my experience too (in .au). It is quite frustrating." -- though he didn't give timings.

I haven't dived deeply, but my hunch is that the slowness for cs.opensource.google is a combination of the latency here, and the amount of JavaScript that has to be run and then subsequent requests loaded before the content shows up. Whereas the GitHub is a much simpler, server-side rendered page with the source code right in the first HTTP response.

I realize it is probably fast for folks in the U.S. and Europe (I have timings from folks in Germany and Sweden on the order of 2 seconds), so most people on the Go team probably don't notice this. But this is a significant performance regression (and therefore usability regression) for those of us on this side of the world, because the old source code viewer on golang.org was fast. I can't access it now, but I remember it was as fast as GitHub.com, or faster.

In terms of resolution: I would love it if the source links switched back to the old, fast viewer. Or to GitHub.com, which is almost as fast. I'm sure Google switched to cs.opensource.google for a reason, of course (I know it has quite a lot of interactive features), but speeding that up seems like it'd be a whole lot more work than just switching provider.

For what it's worth, I'm on Ubuntu Linux 20.04, running Chromium 94.0. However, Firefox takes equally long to load the cs.opensource.google pages.

[kortschak]

In .au I see ~15 seconds for load of that page cold, ~10 warm.

[seebs]

In Minnesota, presumably not quite as far from the Google servers, I still see noticably over 2-3 seconds for the pages to show up, and cold/warm has virtually no impact on any part of this, but wow is there a lot going on in there. I also would in practice prefer a simpler page with significantly less overhead.

[kortschak]

For me, it's now quicker and preferable for me to go to the trouble of opening an editor and viewing the source locally (manually following a link provided elsewhere).

[ericlagergren]

Thank you for filing this issue. I’m in Seattle on gigabit fiber and I’ve also found the change to cs.opensource.google to be unbearably slow.

Go is so simple that I rarely need the full feature suite of Code Search, particularly when looking at stdlib docs.

Additionally I’ve found that CS does not play nicely with ad blockers (it causes the dark mode text color to be the same as the background color, among other things) whereas the original documentation pages had no such issue.

CS also does not display well on mobile, whereas the original documentation was half decent.

IMO, the change is a regression, even though Code Search is an otherwise wonderful tool.

[jba]

We've contacted the Code Search team and they're looking into it.

[damienmg]

Hi all,

Sorry that you experience such latency. There is various issue regarding location of our servers for people in Oceania, but even 2s sounds like a lot to me who is in europe with a 100Mps DSL line, I get better performance.

There is several fact that could explains that that we are actively working on fixing. To see if there is something obvious that we are missing, that would be awesome if one of you could send me a HAR file of the performance profile (the little download arrow in the performance measure)

[benhoyt]

@damienmg Sure, thanks. Attaching the performance profile. It's a JSON file that it downloaded (is HAR a JSON-based format?) -- let me know if you need something else.

Profile-20211006T095948.zip

Note that this was from someone else's place with different internet, so the request is a little bit faster, but not much (looks like we're seeing source code after about 8.5s rather than 10s on mine).

[damienmg]

Thanks yes this is what I needed.

For people in Oceania we are going to add new servers there. We though we could just add them to the config but this is a bit more complicated. If you know broccoli man, you know what I mean :). But it should be coming in a few weeks max.

[benhoyt]

Sounds good, thanks @damienmg! Looking forward to that.

[damienmg]

Thanks there is nothing exceptional in this trace, the time is spend waiting for RPC.

There is 2 issues there:

• Slow network --> will be addressed by bringing the new nodes
• Slow git backend --> this has been an outstanding issues for quarters but we are getting closer to a full resolution (work by the git team at Google). I don't believe this one is so present here.

Anyway we are keeping on working on improving performance of Code Search, this is a tool use by all developer at Google and we are happy that it can help developer outside of Google.

[damienmg]

We have now server in South Asia, hopefully this should bring down latency for everybody in APAC.

[kortschak]

That is much better.

[benhoyt]

Wow, that's a big improvement. Now it's taking 2-3 seconds normally (compared to 5-6 before) and showing ~5s till source is visible in Chromium's performance tools (compared to ~10s before). So it seems about twice as fast, and 2-3 seconds isn't enough for my "give up" response to kick in. :-)

So thank you! Still not the fastest website in the world, but a vast improvement and feels actually usable from here now.

[benhoyt]

In case you want an updated profile to compare against:
Profile-20211007T231620.zip

[benhoyt]

Up to you, but I'd be happy to close this issue now, unless you want it open to track any further work.

[damienmg]

The profile still show very slow RPCs (1.6s for fetching the content :/), would it be possible to download network tab HAR file? Maybe send it to me privately to avoid sharing potentially confidential information.

[benhoyt]

@damienmg Thanks. I just sent it to your d...@inria.fr email address (found from a GitHub commit). Let me know if I should sent it somewhere else. My email address is on my website.

[damienmg]

That's my previous work email, my current email is ***@***.***

…

On Thu, Oct 7, 2021, 1:22 PM Ben Hoyt ***@***.***> wrote: @damienmg <https://github.com/damienmg> Thanks. I just sent it to your ***@***.*** email address (found from a GitHub commit). Let me know if I should sent it somewhere else. My email address is on my website <https://benhoyt.com/>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4MO755E2U47FM6VGB6NX3UFV7G3ANCNFSM5FH6X2FA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[benhoyt]

Sorry, GitHub seems to be redacting the parts of your email address to ***. If you send an email to my address (linked from my website above) I'll reply with the file.

[damienmg]

I just sent an email.

…

On Thu, Oct 7, 2021, 1:33 PM Ben Hoyt ***@***.***> wrote: Sorry, GitHub seems to be redacting the parts of your email address to ***. If you send an email to my address (linked from my website above) I'll reply with the file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4MO7ZAXRBUVEIU3ZJNKHDUFWARPANCNFSM5FH6X2FA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[benhoyt]

Thanks for your work and digging on this. It's still not going to win any Fastest Website Down Under awards, but it's moved from unusably slow to merely sluggish. I'll close this.