Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

Closed
andybons opened this issue Dec 13, 2018 · 70 comments
Closed
Assignees
Labels
Milestone

Comments

@andybons
Copy link
Member

@andybons andybons commented Dec 13, 2018

With the website slated for a refresh, we should move the static content out of the main go source tree (https://go.googlesource.com/go/+/master/doc/) and into its own repository (web? website?).

This will reduce binary and source code download sizes (#27151) and allow us to apply changes unrelated to our release schedule without having to cherry-pick them to a release branch. Hopefully we can also implement continuous delivery as part of this as well.

We also won’t have to worry about bloating the main go source tree as we add more assets and content for the refresh.

We still only want to display package documentation and source files from a GOROOT pinned to the current stable release (and not show content only relevant for future releases like release notes). So there are elements that still require logic that ties content to a specific Go version. What we consider fluid vs tied to a release is to be determined and can be fleshed out here.

/cc @bradfitz @ianlancetaylor @julieqiu @katiehockman @dmitshur @FiloSottile @cnoellekb

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 13, 2018

"website" SGTM. I like the repo name golang/website.

Inside that I'd like something like:

  • README.md
  • codereview.cfg
  • content/ directory
  • cmd/ directory
  • cmd/goweb or cmd/golangweb or cmd/golang-org-server (name TBD)

We could start with mirroring stuff over from the existing golang/go repo (and any necessary code from x/tools/cmd/godoc) but without modifying either for now. And then serve the new one on something like https://new.golang.org for a bit and when we're happy then switch over and start deleting code & assets from the old locations.

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

If they're the same, then do we develop it in x/tools or in x/website? I kinda like having the code next to the assets but I could go either way and haven't thought about it much.

@andybons
Copy link
Member Author

@andybons andybons commented Dec 13, 2018

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

My preference is for x/tools/cmd/godoc to become a binary for viewing local $GOPATH/etc documentation but not anything else (which would be more in line with its name anyhow). A common go-source-to-html rendering library could be pulled out (if it’s not already general enough in x/tools/godoc) and used in places like godoc.org, x/tools/cmd/godoc, and the new website. This way most of the logic specific to new.golang.org is right next to the content.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Dec 13, 2018

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

Yes, we need to figure that out. At this time, I strongly suspect that we’ll want to make the new cmd/golang-org-server be strictly the website and not a tool users run locally, other than to contribute to the golang.org website development.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 13, 2018

Pulling the render-godoc-as-HTML handler out and using it on both places SGTM. That can happen first probably.

And making x/tools/cmd/godoc be super minimal also SGTM.

@cnoellekb
Copy link

@cnoellekb cnoellekb commented Dec 20, 2018

We discussed 4 main milestones for moving the website:

  1. Separate out rendering logic (converting code -> HTML) from x/tools/cmd/godoc. There should be no behavior change at this point.
  2. Creating website binary and serving templates (the static content) from go/doc to x/website
  3. Using the common rendering library in x/website.
  4. Reduce x/tools/cmd/godoc to be the minimal version (since the website can now be run out of x/website).

Right now I'm focusing on the second step because I was able to get a prototype up and running. I copied tools/cmd/godoc, tools/godoc, and go/doc into a local directory that will eventually become the root for the main website. The plan is to just copy everything for now to maintain x/tools functionality. Can someone help me set up an x/website repo?

Also, I was looking into how the rest of the tools directory would be affected by removing any godoc files and I found that x/tools/cmd/present/play.go uses the tools/godoc/static package. Does anyone think this may be an issue?

Lastly, is the final version of godoc going to need an app engine configuration? If not, then tools/internal/memcache can be moved to the x/website repository.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 21, 2018

@dmitshur, @andybons, where are the notes from last time we created a new repo? I'd rather not do it myself. It'd be nice if we can have somebody else run through the steps from the docs. (gerrit, github, maintner, gitmirror updating, etc)

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Dec 21, 2018

where are the notes from last time we created a new repo?

@bradfitz The steps are documented at golang.org/wiki/CreatingSubRepository.

It'd be nice if we can have somebody else run through the steps from the docs.

Yep, that's a good idea. If anyone on the team wants to go through the process to learn it, I'm happy to assist and fill in any gaps in docs. Note that most of the bottlenecks will be having the right permissions to all components involved. I can also do it myself.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 2, 2019

I created https://github.com/golang/website and https://go.googlesource.com/website (Gerrit).

@cnoellekb, can you send CLs for the maintner and dashboard and gitmirror bits? And gopherbot if needed.

@cnoellekb
Copy link

@cnoellekb cnoellekb commented Jan 3, 2019

Sure, I'll take care of it!

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 3, 2019

Change https://golang.org/cl/156217 mentions this issue: maintner/maintnerd, cmd/gerritbot, cmd/gitmirror: add golang/website to

gopherbot pushed a commit to golang/build that referenced this issue Jan 3, 2019
…to whitelist

The website now has its own subrepo with gerrit mirroring set up.

Updates golang/go#29206

Change-Id: I6fb6fb62dfd50b48d2f78db2503641c521600ae7
Reviewed-on: https://go-review.googlesource.com/c/156217
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 4, 2019

Change https://golang.org/cl/156337 mentions this issue: cmd/godoc: add x/website redirect

gopherbot pushed a commit to golang/tools that referenced this issue Jan 4, 2019
Add a redirect for the recently created x/website subrepository.

It's not yet included at https://golang.org/pkg/#subrepo because it's
in development. Once development reaches the point that x/website
is the canonical location of the golang.org server, we can consider
including x/website at https://golang.org/pkg/#subrepo (just like
x/blog, x/tour, x/build are already included there).

Updates golang/go#29206

Change-Id: I6889c1f5e40f11abca944b217a7354f76c08c8eb
Reviewed-on: https://go-review.googlesource.com/c/156337
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 4, 2019

Change https://golang.org/cl/156321 mentions this issue: x/website: add code and static files for website

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 4, 2019

Change https://golang.org/cl/156326 mentions this issue: x/website: add code and static files for website

gopherbot pushed a commit to golang/website that referenced this issue Jan 8, 2019
All of the code and static assets that the website uses to run have been
copied to this repo. There was also a few lines of code added telling
the website where the doc directory, favicon.ico and robots.txt are.

go repo change-id: Ife6443c32673b38000b90dd2efb2985db37ab773
x/tools repo change-id: Ia979a8b06d1b4db47d25ffdfdf925ba8a0ac67de

Real new code additions:
- main.go
    * lines 89-95 added getFullPath method
    * lines 217-222 mapped paths to doc/, favicon.ico, robots.txt in vfs
- appinit.go
    * lines 147-153 added getFullPath method
    * lines 80-84 mapped paths to doc/, favicon.ico in vfs

Several files were copied from x/tools and go so paths (and corresponding
import paths) were changed as follows:

"x/tools/cmd/godoc/" --> "x/website/cmd/golangorg/"
"x/tools/godoc/static/" --> "x/website/content/static/"
"x/tools/godoc/" (without godoc/static/) --> "x/website/cmd/golangorg/godoc/"
"x/tools/internal/memcache" --> "x/website/internal/memcache"
"go/doc/" --> "x/website/content/doc/"
"go/favicon.ico" --> "x/website/favicon.ico"
"go/robots.txt" --> "x/website/robots.txt"

Updates golang/go#29206

Change-Id: I53985fc027f73e60c6946038f85133acf1ecb08c
Reviewed-on: https://go-review.googlesource.com/c/156321
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@cnoellekb
Copy link

@cnoellekb cnoellekb commented Jan 9, 2019

Some decisions that have been made over the last few days:

  1. I am currently prioritizing turning shared documentation-rendering logic between x/website and the godoc binary into a package, maybe it will live in a directory called "unstable" in x/tools?
  2. Deployment of the website will come later when godoc has been properly stripped down with all non-documentation stuff ready to be deleted from x/tools
  3. In the long term @dmitshur plans to expand the package to support rendering documentation for modules
@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jan 9, 2019

  1. I am currently prioritizing turning shared documentation-rendering logic between x/website and the godoc binary into a package, maybe it will live in a directory called "unstable" in x/tools?

This sounds good. Perhaps under x/tools/godoc/unstable since it will be for go documentation.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 9, 2019

I wouldn't bother with the /unstable suffix: just put a prominent note in the package comment explaining that it's unstable.

Actually, better still, could we start a new repo with the path golang.org/x/internal? Then the Go tool's visibility rules would enforce that it is only used within other golang.org/x modules.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jan 9, 2019

We already have the golang.org/x/tools/godoc package, which is documented as:

Package godoc is a work-in-progress (2013-07-17) package to begin splitting up the godoc binary into multiple pieces.

This package comment will evolve over time as this package splits into smaller pieces.

I think this existing package is a good candidate to move code that we want to share between x/tools/cmd/godoc (for viewing documentation) and x/website/cmd/golangorg (for serving golang.org) in the short term.

I like the idea of x/internal in general, because if it existed, we could use it in other instances to do similar refactors between subrepos. However, I think that should be a separate proposal. Also, golang.org/x/internal would preclude golang.org/dl from being able to use it, so maybe it should be golang.org/internal.

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 25, 2019

Change https://golang.org/cl/159737 mentions this issue: x/website: remove godoc package/make module aware

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 28, 2019

Change https://golang.org/cl/159897 mentions this issue: website: use modules

gopherbot pushed a commit to golang/website that referenced this issue Dec 11, 2019
The Release History, Pre-Go 1 Release History, and Weekly Snapshot
History pages are copied from Go master as of 2019/12/10 (commit
golang/go@931fe39, CL 210717).

Updates golang/go#36075
Updates golang/go#29206

Change-Id: I24d24d6346405df2d3d657ea2b4aa049182d5141
Reviewed-on: https://go-review.googlesource.com/c/website/+/210797
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit that referenced this issue Dec 11, 2019
These pages were moved to the x/website repo in CL 210797 (commit
golang/website@9aef1ee).
Remove the old copies in this repo since they're no longer used.

Updates #36075
Updates #29206

Change-Id: I6e3ffaebd92fa753cb5f3b21e4238edfb7f5f0e8
Reviewed-on: https://go-review.googlesource.com/c/go/+/210798
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur dmitshur self-assigned this Dec 13, 2019
@gopherbot
Copy link

@gopherbot gopherbot commented Dec 13, 2019

Change https://golang.org/cl/211300 mentions this issue: content/static/doc: copy Go Security Policy page from Go repo

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 13, 2019

Change https://golang.org/cl/211301 mentions this issue: doc: remove Go Security Policy page (moved to x/website)

gopherbot pushed a commit to golang/website that referenced this issue Dec 16, 2019
The Go Security Policy page is copied from Go master as of 2019/12/13
(commit golang/go@0497f91, CL 211177).

This page not closely tied to specific Go releases, and thus is better
suited to live in the x/website repo.

Updates golang/go#29206

Change-Id: Ib23fa6ccd11218d5da231cca685f62e610d5c6c9
Reviewed-on: https://go-review.googlesource.com/c/website/+/211300
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
gopherbot pushed a commit that referenced this issue Dec 16, 2019
This page has moved to the x/website repo in CL 211300 (commit
golang/website@3c8b7f9).
Remove the old copy in this repo since it's no longer used.

Updates #29206

Change-Id: I8b3396d9e42d1e7262a8cde9577962d33b215836
Reviewed-on: https://go-review.googlesource.com/c/go/+/211301
Reviewed-by: Filippo Valsorda <filippo@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 2, 2020

Change https://golang.org/cl/213157 mentions this issue: doc: remove root.html and conduct.html (moved to x/website)

gopherbot pushed a commit that referenced this issue Jan 3, 2020
root.html has been moved to the x/website repo in CL 180959
(commit golang/website@d83058c).

conduct.html has been moved to the x/website repo in CL 207437
(commit golang/website@99763cb).

There should be only one copy, otherwise it may lead to confusion,
or changes made in the wrong place. This CL removes the old copies
from this repo since they're no longer used.

Updates #29206

Change-Id: I41adfb2c34ed3d870fb7a671f48ccc8f90863feb
Reviewed-on: https://go-review.googlesource.com/c/go/+/213157
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 8, 2020

Change https://golang.org/cl/213937 mentions this issue: content/static: add gopath_code.html file and regenerate

gopherbot pushed a commit to golang/website that referenced this issue Jan 9, 2020
CL 199417 added a new doc/gopath_code.html file. This change adds
it to the list of files that are embedded in package static, and
regenerates the package.

A change described in https://golang.org/issue/29206#issuecomment-536099768
will make it no longer neccessary to do this step in the future.

Updates golang/go#28215
Updates golang/go#29206

Change-Id: Ia1bd125488e04b34f9b8e36de4397de32d77bcef
Reviewed-on: https://go-review.googlesource.com/c/website/+/213937
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 20, 2020

Change https://golang.org/cl/229081 mentions this issue: cmd/golangorg: generate release history page from internal/history

gopherbot pushed a commit to golang/website that referenced this issue Apr 21, 2020
Previously, the release history page was a raw HTML file that was
manually edited whenever new Go releases were made. This change
converts release history entries into a structured format in the
new internal/history package, and generates release history entries
from that format.

For now, only Go 1.9 and newer releases are converted, but the
structured format is flexible enough to represent all releases
going back to the original Go 1 release.

Various English grammar rules and special cases are preserved,
so that the release history entries appear in a consistent way.

New release history entries need only to be added to the internal/
history package, making it so that English grammar rules and HTML
tags don't need to go through human code review for each release.
Future work may involve constructing that list from data already
available in the Go issue tracker.

This change makes minimal contributions to reducing the dependence
of x/website on the x/tools/godoc rendering engine for displaying
pages other than Go package documentation. The x/tools/godoc code
is in another module and does not provide flexibility desired for
the general purpose website needs of x/website.

Fixes golang/go#38488.
For golang/go#37090.
For golang/go#29206.

Change-Id: I80864e4f218782e6e3b5fcd5a1d63f3699314c81
Reviewed-on: https://go-review.googlesource.com/c/website/+/229081
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 22, 2020

Change https://golang.org/cl/229483 mentions this issue: cmd/golangorg: generate major version list on Go project page

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 22, 2020

Change https://golang.org/cl/229482 mentions this issue: content/static/doc: copy The Go Project page (contrib.html) from Go repo

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 22, 2020

Change https://golang.org/cl/229485 mentions this issue: doc: remove The Go Project page (moved to x/website)

gopherbot pushed a commit to golang/website that referenced this issue Apr 24, 2020
The Go Project page is copied from Go master as of 2020/04/24
(commit golang/go@ee8972c, CL 209679).

This page not closely tied to specific Go releases, and thus is better
suited to live in the x/website repo.

It will be modified in CL 229483 to use the release history data from
the internal/history package, rather than using hand-written HTML
which needs to be updated manually after each major Go release.

For golang/go#29205.
For golang/go#29206.

Change-Id: I6a6fa734fb3d873da59d67d796616fc53d684610
Reviewed-on: https://go-review.googlesource.com/c/website/+/229482
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit that referenced this issue Apr 24, 2020
This page has moved to the x/website repo in CL 229482 (commit
golang/website@70f4ee8).
Remove the old copy in this repo since it's no longer used.

For #29206.

Change-Id: Ief093ed8c5dfec43e06d473e4282275f61da74a5
Reviewed-on: https://go-review.googlesource.com/c/go/+/229485
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Apr 24, 2020
This change builds on what was done in CL 229081, and uses the Go
release history data from internal/history package to generate the
list of major Go versions on the Go project page.

This way, this page doesn't need to be manually edited when major
Go releases are made.

For golang/go#38488.
For golang/go#29205.
For golang/go#29206.

Change-Id: Ie0b12707d828207173a54f0a1bc6a4ef69dcedef
Reviewed-on: https://go-review.googlesource.com/c/website/+/229483
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
xujianhai666 added a commit to xujianhai666/go-1 that referenced this issue May 21, 2020
This page has moved to the x/website repo in CL 229482 (commit
golang/website@70f4ee8).
Remove the old copy in this repo since it's no longer used.

For golang#29206.

Change-Id: Ief093ed8c5dfec43e06d473e4282275f61da74a5
Reviewed-on: https://go-review.googlesource.com/c/go/+/229485
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 16, 2020

Change https://golang.org/cl/243118 mentions this issue: internal/dl: add CORS support to JSON endpoint

gopherbot pushed a commit to golang/website that referenced this issue Jul 16, 2020
This change allows users to request golang.org/dl/?mode=json
via Cross-Origin Resource Sharing (CORS).

It also removes the golangorg build tag as it did not seem
necessary and adds tests for the “include” GET parameter.

Updates golang/go#29206
Fixes golang/go#40253

Change-Id: I5306a264c4ac2a6e6f49cfb53db01eef6b7f4473
Reviewed-on: https://go-review.googlesource.com/c/website/+/243118
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 28, 2020

I think this can be closed. The golang.org website and most of its content has been factored out into the x/website repository, and it's been in production for a long while. There's more that we can optionally deal with in the future, but there's nothing critical left.

I've factored out some simplification opportunities from #29206 (comment) into a separate issue #41102.

If there's anything more that needs to be done, let's open individual issues.

Thanks to everyone who contributed to this successful move!

@dmitshur dmitshur closed this Aug 28, 2020
@dmitshur dmitshur removed the Documentation label Aug 28, 2020
@dmitshur dmitshur modified the milestones: Backlog, Unreleased Aug 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.