x/pkgsite: local setup - tracking issue

[julieqiu]

Today, users can run pkgsite locally without setting up a local database by running:

go run cmd/frontend/main.go -direct_proxy -proxy_url=<your proxy URL>

See flags in cmd/frontend/main.go, and doc/frontend.md for docs.

Doing so allows users to view the package documentation, overview, imports, and licenses tabs. Search and other tabs on the package page are not supported in direct proxy mode.

However, there are cases when a user might want to run pkgsite on their local machine, and be able to view a private repository that is not available via a proxy.

Issues related to this topic:

• #40159: run locally to view documentation of a pending CL

This is an umbrella issue to discuss what features are needed for this to happen, so please comment on your use cases below. For discussions about hosting a private instance of pkgsite, including your own internal proxy, see #39827.

[achille-roussel]

Hello @julieqiu

If I can add color to the pain points of running locally, having to start a proxy isn't too much trouble, but the constraint of requiring https is a bigger barrier:

$ go run ./cmd/frontend -direct_proxy -proxy_url http://localhost:8888
config: {
    "AuthHeader": "",
    "ProxyURL": "https://proxy.golang.org",
    "IndexURL": "https://index.golang.org/index",
    "Port": "",
    "DebugPort": "",
    "ProjectID": "",
    "ServiceID": "",
    "VersionID": "",
    "ZoneID": "",
    "InstanceID": "",
    "LocationID": "us-central1",
    "QueueService": "",
    "GaeEnv": "",
    "GoogleTagManagerID": "",
    "AppMonitoredResource": {
        "type": "gae_app",
        "labels": {
            "module_id": "",
            "project_id": "",
            "version_id": "",
            "zone": ""
        }
    },
    "FallbackVersionLabel": "20200729t023455",
    "DBSecret": "",
    "DBUser": "postgres",
    "DBHost": "localhost",
    "DBPort": "5432",
    "DBName": "discovery-db",
    "DBSecondaryHost": "",
    "RedisCacheHost": "",
    "RedisCachePort": "6379",
    "RedisHAHost": "",
    "RedisHAPort": "6379",
    "UseProfiler": false,
    "Quota": {
        "QPS": 10,
        "Burst": 20,
        "MaxEntries": 1000,
        "RecordOnly": true,
        "AuthHeader": "",
        "AuthValues": null
    },
    "TeeproxyAuthValue": "",
    "TeeproxyForwardedHosts": null
}
2020/07/29 02:34:55 Error: proxy.New("http://localhost:8888"): scheme must be https (got http)
exit status 1

What do you think about relaxing this constraint?

[gopherbot]

Change https://golang.org/cl/245639 mentions this issue: internal/proxy: remove URL validation in New

[julieqiu]

@achille-roussel - done! Thanks for the feedback.

[MicahParks]

To run pkgsite in a docker container, it'd be good to customize what networking interface the program binds to.

Executables in docker containers can more easily publish their services on the host's networking interface if bound to 0.0.0.0. It would be possible to allow this configuration via a flag by changing these lines:

https://github.com/golang/pkgsite/blob/ef183200d192b20102bb1246a86ae11de2d0738e/cmd/frontend/main.go#L170
https://github.com/golang/pkgsite/blob/ef183200d192b20102bb1246a86ae11de2d0738e/cmd/frontend/main.go#L148

to use a variable that is assigned by a flag instead of the "localhost" string.

[achille-roussel]

We had to solve for this as well in the fork we maintain at https://github.com/segmentio/pkgsite:

diff --git a/cmd/frontend/main.go b/cmd/frontend/main.go
index 5773d8e..e81022b 100644
--- a/cmd/frontend/main.go
+++ b/cmd/frontend/main.go
@@ -42,6 +42,7 @@ var (
                "for direct proxy mode and frontend fetches")
        directProxy = flag.Bool("direct_proxy", false, "if set to true, uses the module proxy referred to by this URL "+
                "as a direct backend, bypassing the database")
+       httpAddr = flag.String("http", "localhost:8080", "address to listen for incoming requests on")
 )

 func main() {
@@ -167,7 +168,7 @@ func main() {
                middleware.Timeout(54*time.Second),
                middleware.Experiment(experimenter),
        )
-       addr := cfg.HostAddr("localhost:8080")
+       addr := cfg.HostAddr(*httpAddr)
        log.Infof(ctx, "Listening on addr %s", addr)
        log.Fatal(ctx, http.ListenAndServe(addr, mw(router)))
 }

Taking a closer look at the way the configuration is setup, it appears that if the PORT environment variable is set the server will be listening on all network interfaces (as the address will be :$PORT), so maybe that change was unnecessary:

// HostAddr returns the network on which to serve the primary HTTP service.
func (c *Config) HostAddr(dflt string) string {
	if c.Port != "" {
		return fmt.Sprintf(":%s", c.Port)
	}
	return dflt
}

[gopherbot]

Change https://golang.org/cl/260779 mentions this issue: cmd/frontend: add -local flag

[gopherbot]

Change https://golang.org/cl/290134 mentions this issue: cmd/pkgsite: add command

[gopherbot]

Change https://golang.org/cl/290135 mentions this issue: cmd/pkgsite: add -http flag

[akavel]

I was redirected here from an issue requesting a flag in godoc for showing internal packages (for use during internal/company development purposes, to make it possible to easily browse internal APIs) - #12092 (comment). Thus (per the request for use cases in the main comment above), that's a use case me and several other people expressed interest in. If godoc is seen as "in maintenance mode" (which was quite a surprise to me), it would be nice if the plan for the new replacement tool was outlined here (?) more clearly, or something. Currently, the situation & status may appear unclear to community members (like me).

[thediveo]

A major stumbling Block for us is the inferior Support of private modules and APIs in turn. In our past, a dev-system local godoc helped us, even if that required developers to check out the API repo. godoc already was problematic when viewed from a typical private GitHub and Gitlab use case, as you can't get any static pages to be hosted. gopkg seems to make the situation even worse, as it now requires more and more infrastructure and doesn't scale down.