Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: stdlib source links are very slow to load (cs.opensource.google) #48752

Closed
benhoyt opened this issue Oct 3, 2021 · 21 comments
Closed

x/pkgsite: stdlib source links are very slow to load (cs.opensource.google) #48752

benhoyt opened this issue Oct 3, 2021 · 21 comments

Comments

@benhoyt
Copy link
Contributor

@benhoyt benhoyt commented Oct 3, 2021

As I posted on golang-nuts, the source links for the standard library source code on pkg.go.dev are very slow -- at least for me down here (in Christchurch, New Zealand).

As shown in the attached Chromium dev tools images (see screenshots below), cs.opensource.google links take about 10 seconds to show the source code with a cold cache, and about 5 seconds with a warm cache. This is very slow in absolute terms, and almost enough time for the TL;GU ("too slow; give up") response to kick in. :-) It's also very slow in relative terms: see the attached screenshot showing GitHub.com taking 2 seconds to load the same source code.

Diving into the screenshots in a bit more detail:

  1. The cs.opensource.google page timings: as you can (barely) see from the tiny screenshots, it goes through three loading states: a blank page with "loading" shows after about 3s, then the frame with sidebar shows after about 6s, but the source code itself isn't visible till about 10s.
  2. The github.com page timings: a single loading state, and the source code is visible after about 2s.

I have a reasonably fast internet connection: 40ms ping time to google.com, wherever that's coming from, and about 50Mbits/s up and down. Even relatively big sites with lots of pictures, like nytimes.com, load in about 2-3 seconds with a cold cache. So while I'm sure latencies are bigger down here, the internet in New Zealand is not slow in general -- this is very much a cs.opensource.google thing. Anecdotally, a friend in Australia got timings like 5-7 seconds for cs.opensource.google. And Dan Kortschak, author of gonum, replied on golang-nuts that "This has been my experience too (in .au). It is quite frustrating." -- though he didn't give timings.

I haven't dived deeply, but my hunch is that the slowness for cs.opensource.google is a combination of the latency here, and the amount of JavaScript that has to be run and then subsequent requests loaded before the content shows up. Whereas the GitHub is a much simpler, server-side rendered page with the source code right in the first HTTP response.

I realize it is probably fast for folks in the U.S. and Europe (I have timings from folks in Germany and Sweden on the order of 2 seconds), so most people on the Go team probably don't notice this. But this is a significant performance regression (and therefore usability regression) for those of us on this side of the world, because the old source code viewer on golang.org was fast. I can't access it now, but I remember it was as fast as GitHub.com, or faster.

In terms of resolution: I would love it if the source links switched back to the old, fast viewer. Or to GitHub.com, which is almost as fast. I'm sure Google switched to cs.opensource.google for a reason, of course (I know it has quite a lot of interactive features), but speeding that up seems like it'd be a whole lot more work than just switching provider.

For what it's worth, I'm on Ubuntu Linux 20.04, running Chromium 94.0. However, Firefox takes equally long to load the cs.opensource.google pages.

cs-opensource-google

github

@gopherbot gopherbot added this to the Unreleased milestone Oct 3, 2021
@kortschak
Copy link
Contributor

@kortschak kortschak commented Oct 3, 2021

In .au I see ~15 seconds for load of that page cold, ~10 warm.

@seebs
Copy link
Contributor

@seebs seebs commented Oct 3, 2021

In Minnesota, presumably not quite as far from the Google servers, I still see noticably over 2-3 seconds for the pages to show up, and cold/warm has virtually no impact on any part of this, but wow is there a lot going on in there. I also would in practice prefer a simpler page with significantly less overhead.

@kortschak
Copy link
Contributor

@kortschak kortschak commented Oct 3, 2021

For me, it's now quicker and preferable for me to go to the trouble of opening an editor and viewing the source locally (manually following a link provided elsewhere).

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Oct 4, 2021

Thank you for filing this issue. I’m in Seattle on gigabit fiber and I’ve also found the change to cs.opensource.google to be unbearably slow.

Go is so simple that I rarely need the full feature suite of Code Search, particularly when looking at stdlib docs.

Additionally I’ve found that CS does not play nicely with ad blockers (it causes the dark mode text color to be the same as the background color, among other things) whereas the original documentation pages had no such issue.

CS also does not display well on mobile, whereas the original documentation was half decent.

IMO, the change is a regression, even though Code Search is an otherwise wonderful tool.

@jamalc jamalc removed this from the Unreleased milestone Oct 4, 2021
@jamalc jamalc added this to the pkgsite/unplanned milestone Oct 4, 2021
@jamalc jamalc assigned jamalc and julieqiu and unassigned jamalc Oct 4, 2021
@julieqiu julieqiu assigned jba and unassigned julieqiu Oct 4, 2021
@jba
Copy link
Contributor

@jba jba commented Oct 4, 2021

We've contacted the Code Search team and they're looking into it.

@damienmg
Copy link

@damienmg damienmg commented Oct 5, 2021

Hi all,

Sorry that you experience such latency. There is various issue regarding location of our servers for people in Oceania, but even 2s sounds like a lot to me who is in europe with a 100Mps DSL line, I get better performance.

There is several fact that could explains that that we are actively working on fixing. To see if there is something obvious that we are missing, that would be awesome if one of you could send me a HAR file of the performance profile (the little download arrow in the performance measure)

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 5, 2021

@damienmg Sure, thanks. Attaching the performance profile. It's a JSON file that it downloaded (is HAR a JSON-based format?) -- let me know if you need something else.

Profile-20211006T095948.zip

Note that this was from someone else's place with different internet, so the request is a little bit faster, but not much (looks like we're seeing source code after about 8.5s rather than 10s on mine).

@damienmg
Copy link

@damienmg damienmg commented Oct 6, 2021

Thanks yes this is what I needed.

For people in Oceania we are going to add new servers there. We though we could just add them to the config but this is a bit more complicated. If you know broccoli man, you know what I mean :). But it should be coming in a few weeks max.

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 6, 2021

Sounds good, thanks @damienmg! Looking forward to that.

@damienmg
Copy link

@damienmg damienmg commented Oct 6, 2021

Thanks there is nothing exceptional in this trace, the time is spend waiting for RPC.

There is 2 issues there:

  • Slow network --> will be addressed by bringing the new nodes
  • Slow git backend --> this has been an outstanding issues for quarters but we are getting closer to a full resolution (work by the git team at Google). I don't believe this one is so present here.

Anyway we are keeping on working on improving performance of Code Search, this is a tool use by all developer at Google and we are happy that it can help developer outside of Google.

@damienmg
Copy link

@damienmg damienmg commented Oct 7, 2021

We have now server in South Asia, hopefully this should bring down latency for everybody in APAC.

@kortschak
Copy link
Contributor

@kortschak kortschak commented Oct 7, 2021

That is much better.

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 7, 2021

Wow, that's a big improvement. Now it's taking 2-3 seconds normally (compared to 5-6 before) and showing ~5s till source is visible in Chromium's performance tools (compared to ~10s before). So it seems about twice as fast, and 2-3 seconds isn't enough for my "give up" response to kick in. :-)

So thank you! Still not the fastest website in the world, but a vast improvement and feels actually usable from here now.

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 7, 2021

In case you want an updated profile to compare against:
Profile-20211007T231620.zip

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 7, 2021

Up to you, but I'd be happy to close this issue now, unless you want it open to track any further work.

@damienmg
Copy link

@damienmg damienmg commented Oct 7, 2021

The profile still show very slow RPCs (1.6s for fetching the content :/), would it be possible to download network tab HAR file? Maybe send it to me privately to avoid sharing potentially confidential information.

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 7, 2021

@damienmg Thanks. I just sent it to your d...@inria.fr email address (found from a GitHub commit). Let me know if I should sent it somewhere else. My email address is on my website.

@damienmg
Copy link

@damienmg damienmg commented Oct 7, 2021

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 7, 2021

Sorry, GitHub seems to be redacting the parts of your email address to ***. If you send an email to my address (linked from my website above) I'll reply with the file.

@damienmg
Copy link

@damienmg damienmg commented Oct 7, 2021

@benhoyt
Copy link
Contributor Author

@benhoyt benhoyt commented Oct 14, 2021

Thanks for your work and digging on this. It's still not going to win any Fastest Website Down Under awards, but it's moved from unusably slow to merely sluggish. I'll close this.

@benhoyt benhoyt closed this Oct 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants