Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: API for pkg.go.dev #36785

Open
rhcarvalho opened this issue Jan 26, 2020 · 48 comments
Open

x/pkgsite: API for pkg.go.dev #36785

rhcarvalho opened this issue Jan 26, 2020 · 48 comments
Labels
FeatureRequest NeedsInvestigation pkgsite

Comments

@rhcarvalho
Copy link
Contributor

@rhcarvalho rhcarvalho commented Jan 26, 2020

Prior to pkg.go.dev, godoc.org has had a JSON API that can be used to, among other things, discover importers of a given package.

Example: https://api.godoc.org/importers/golang.org/x/net/html

Given that pkg.go.dev does a much better job at tracking importers thanks to Go Modules and the Module Proxy, it would be nice if the community could get access to a public API similar to that of godoc.org.

@julieqiu julieqiu added NeedsInvestigation FeatureRequest labels Jan 28, 2020
@julieqiu
Copy link
Contributor

@julieqiu julieqiu commented Jan 29, 2020

@rhcarvalho - it would be helpful to get a sense of what your current use cases are for api.godoc.org, and feature requests are for an API for pkg.go.dev.

It sounds like getting the importers for a package is one of them. With pkg.go.dev being module aware, what specific information about importers would be useful to surface via an API?

For example:

  • importers for a specific version of a package
  • only importers of the latest version of a package
  • any importer for all versions of a package
  • something else?

Additionally, what other information would be useful to you to surface via an API?

/cc @tbpg who has also mentioned wanting an API for pkg.go.dev

@julieqiu julieqiu added WaitingForInfo NeedsInvestigation and removed NeedsInvestigation labels Jan 29, 2020
@rhcarvalho
Copy link
Contributor Author

@rhcarvalho rhcarvalho commented Jan 29, 2020

Without much thought, having an API to answer the more specific question "what are the importers of a specific version of a package" would make plausible to derive the answer to the other items in your list.

At the moment I consume the godoc.org API and scrape data from pkg.go.dev to answer the question "who uses my package". As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing.

Here are the other endpoints in the GoDoc API: https://github.com/golang/gddo/blob/7365cb292b8bfd13dfe514a18b207b9cc70e6ceb/gddo-server/main.go#L901-L904

  • /search: takes a q parameter. Can be useful to explore known packages/modules.
  • /packages: attempts to return all known packages, no pagination. Doesn't seem too usable as is.
  • /importers/: useful, hard to compute without central knowledge.
  • /imports/: dispensable, easy to compute offline with go list.

So if we need a more specific request, here it is:

api.go.dev MVP

  • /importers/{module-or-package}/@v/<version>: returns the list of importers of a given module/package at a given version. URL scheme could be tuned to match the Go Proxy specification (go help goproxy).
  • /search/: matches the functionality of https://pkg.go.dev/search?q=hello, returns JSON instead.

@gopherbot gopherbot added this to the Unreleased milestone Feb 6, 2020
@julieqiu julieqiu removed the WaitingForInfo label Feb 29, 2020
@tbpg
Copy link
Contributor

@tbpg tbpg commented Mar 18, 2020

Another endpoint idea (standalone or as part of a set of info returned for a module): available versions of a given module.

@myitcv
Copy link
Member

@myitcv myitcv commented Mar 18, 2020

@tbpg

Another endpoint idea (standalone or as part of a set of info returned for a module): available versions of a given module.

Shouldn't people use cmd/go for that? Not least because of the fall-through semantics of the go env GOPROXY` variable. Noting #37367

@julieqiu
Copy link
Contributor

@julieqiu julieqiu commented Mar 18, 2020

What information beyond https://proxy.golang.org/<module>/@v/list would be provided by that endpoint?

@tbpg
Copy link
Contributor

@tbpg tbpg commented Mar 18, 2020

Shouldn't people use cmd/go for that? Not least because of the fall-through semantics of the go env GOPROXY` variable. Noting #37367

There are some cases where cmd/go might not be available and a normal web API would be helpful. I think it's reasonable to assume pkg.go.dev will only return the modules/versions it knows about.

What information beyond https://proxy.golang.org/<module>/@v/list would be provided by that endpoint?

None. That works. :)

@myitcv
Copy link
Member

@myitcv myitcv commented Mar 18, 2020

There are some cases where cmd/go might not be available and a normal web API would be helpful

I'd be wary of encouraging tool authors to query pkg.go.dev/a proxy directly. Because it could well introduce skew compared to the answers from cmd/go.

@adamdecaf
Copy link
Contributor

@adamdecaf adamdecaf commented Mar 18, 2020

In my use-case I don't want to execute cmd/go binaries as I just want to grab some metadata (i.e. detected license of all dependencies) from a dozen modules and organize it on a webpage. If I need local clones of those modules it really complicates this internal tool.

@myitcv
Copy link
Member

@myitcv myitcv commented Apr 9, 2020

In #37952 I raised the question of whether module/package license file information could be surfaced in the output of cmd/go list. Following a conversation on yesterday's golang-tools call, we concluded that doing so would be a bad idea; reasons summarised in #37952 (comment).

@ianthehat instead suggested exposing the information via a pkg.go.dev API, leveraging the fact that the content and presentation on pkg.go.dev has already jumped through the relevant legal hoops.

This comment is therefore to explicitly request that we include license file information in the API. Thank you!

@tiegz
Copy link

@tiegz tiegz commented Jun 5, 2020

👋 Thanks for all the new Go tools like pkg.go.dev, they're super useful. Some input on this topic from the perspective of Libraries.io:

  • +1 to a /search/ endpoint, or maybe just adding pagination to this url so it's possible to get a list of all packages: https://index.golang.org/index?since=TIME&limit=LIMIT. Many popular package repositories offer a full listing like this, e.g. https://packagist.org/packages/list.json for Packagist

  • What information beyond https://proxy.golang.org//@v/list would be provided by that endpoint?

Currently to get the publish times for all versions, you have to make N+1 requests: one for @v/list and then @v/xxx.info for each version and grab the "Time" value for each one. Assuming it's difficult to change the Go proxy spec, an API endpoint that returns all versions with metadata would be really nice.

  • As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing.

Anyone know if there are plans to fix this?

@jba
Copy link
Contributor

@jba jba commented Jun 8, 2020

  • As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing.

Anyone know if there are plans to fix this?

No immediate plans. We currently gather that information from import statements in the code, so there is no version information attached. The go.mod file doesn't have all the version information we need.

So we understand it's an approximation and we want to fix it, but it's going to take some time.

@julieqiu julieqiu changed the title go.dev: API for pkg.go.dev x/pkgsite: API for pkg.go.dev Jun 15, 2020
@pombredanne
Copy link

@pombredanne pombredanne commented Jul 12, 2020

@julieqiu my use for an API would be to:

  1. search and list packages
  2. for a package, access all the information as available for a given package in each of the tabs, such as here https://pkg.go.dev/github.com/gin-gonic/gin?tab=licenses ... basically everything that is available as HTML in the frontend https://github.com/golang/pkgsite/tree/master/internal/frontend should be available in a JSON api

@pombredanne
Copy link

@pombredanne pombredanne commented Jul 12, 2020

@julieqiu oh and please do not retire http://api.godoc.org/packages unless there is an alternative!

@eclipseo
Copy link

@eclipseo eclipseo commented Aug 29, 2020

it would be helpful to get a sense of what your current use cases are for api.godoc.org, and feature requests are for an API for pkg.go.dev.

As a downstream package maintainer for Fedora, I'm also interested in an API. We have our own tool, Anitya, to track package releases, but it was not designed to track GIT commits. And many Go packages still don't publish version. So any information about latest published commit, with info like date of the commit, new dependencies, would be very helpful. I would gather data from the API in Python and compare it to the latest version we have for a given package.

I'm also interested in getting the License info, so we could find recursively all the licenses used in a static binary.

@restuwahyu13
Copy link

@restuwahyu13 restuwahyu13 commented Jul 22, 2021

But is api.godoc.org the same as pkg.go.dev sir @jba ?

@fzipp
Copy link
Contributor

@fzipp fzipp commented Jul 22, 2021

@restuwahyu13 pkg.go.dev doesn't have an API yet, that's why this issue is open and not closed.

@shellscape
Copy link

@shellscape shellscape commented Oct 7, 2021

Stopping by to ask that any first version of an API provide some means to glean the source code / repo location of a package that uses a vanity path/url/name.

@guseggert
Copy link

@guseggert guseggert commented Oct 20, 2021

I have similar use cases as others, so I resorted to scraping the UI as well (🤢), and put it behind an API/CLI: https://github.com/guseggert/pkggodev-client

@sig-aarena
Copy link

@sig-aarena sig-aarena commented Jan 21, 2022

The /packages api endpoint, https://api.godoc.org/packages, is returning {"error":{"message":"Internal Server Error"}} .

Has this endpoint been migrated?

tsenart added a commit to sourcegraph/sourcegraph that referenced this issue Jan 28, 2022
This commit fixes our Go import badges on sourcegraph.com, which have
been broken since api.godoc.org was brought down. There's a [tracking
issue](golang/go#36785) to re-introduce an
API for pkg.go.dev, but it's not implemented yet.

Instead, we implement a best-effort alternative that relies on searching
`go.mod` files for the given repo name on sourcegraph.com.
Sometimes packages will have different canonical names than their
repositories, and we don't handle that case.

There are [many open-source projects that have a broken badge](https://sourcegraph.com/search?q=context:global+sourcegraph.com+-/badge.svg&patternType=regexp).
After this lands, they will recover.
tsenart added a commit to sourcegraph/sourcegraph that referenced this issue Jan 28, 2022
This commit fixes our Go import badges on sourcegraph.com, which have
been broken since api.godoc.org was brought down. There's a [tracking
issue](golang/go#36785) to re-introduce an
API for pkg.go.dev, but it's not implemented yet.

Instead, we implement a best-effort alternative that relies on searching
`go.mod` files for the given repo name on sourcegraph.com.
Sometimes packages will have different canonical names than their
repositories, and we don't handle that case.

There are [many open-source projects that have a broken badge](https://sourcegraph.com/search?q=context:global+sourcegraph.com+-/badge.svg&patternType=regexp).
After this lands, they will recover.

Co-authored-by: Erik Seliger <erikseliger@me.com>
Co-authored-by: Keegan Carruthers-Smith <keegan@sourcegraph.com>
@tills13
Copy link

@tills13 tills13 commented Jan 28, 2022

Seems like most api.godoc.org endpoints are erroring, currently?

e.g. https://api.godoc.org/packages, https://api.godoc.org/imports/github.com/goburrow/cache, etc.

I'm particularly interested in the imports endpoint. Has this moved somewhere else or is there a mirror I can use?

@pombredanne
Copy link

@pombredanne pombredanne commented Jan 28, 2022

This is starting to be really problematic. @jba could someone kindly look into this? This is really looking bad

@pombredanne
Copy link

@pombredanne pombredanne commented Jan 28, 2022

@julieqiu do you know what happened?

@jba
Copy link
Contributor

@jba jba commented Jan 31, 2022

Will get on this today.

@pombredanne
Copy link

@pombredanne pombredanne commented Jan 31, 2022

@jba you rock... that's very kind of you! 🙇

@pombredanne
Copy link

@pombredanne pombredanne commented Jan 31, 2022

Note that only alternative today would be to resort to extensive pseudo-random screen scraping of https://pkg.go.dev/search laced in with some extensive recursive hitting of https://index.golang.org/index to get a complete package picture. It can work if this is what is recommended, but it would feel a bit like it is 1999 all over again. ;)
If anything and if that's the way to go, then at least I could create a common service that does this for everyone?

davejrt pushed a commit to sourcegraph/sourcegraph that referenced this issue Jan 31, 2022
This commit fixes our Go import badges on sourcegraph.com, which have
been broken since api.godoc.org was brought down. There's a [tracking
issue](golang/go#36785) to re-introduce an
API for pkg.go.dev, but it's not implemented yet.

Instead, we implement a best-effort alternative that relies on searching
`go.mod` files for the given repo name on sourcegraph.com.
Sometimes packages will have different canonical names than their
repositories, and we don't handle that case.

There are [many open-source projects that have a broken badge](https://sourcegraph.com/search?q=context:global+sourcegraph.com+-/badge.svg&patternType=regexp).
After this lands, they will recover.

Co-authored-by: Erik Seliger <erikseliger@me.com>
Co-authored-by: Keegan Carruthers-Smith <keegan@sourcegraph.com>
@jba
Copy link
Contributor

@jba jba commented Feb 1, 2022

api.godoc.org should be back up.

@pombredanne

This comment has been hidden.

@pombredanne

This comment has been hidden.

davejrt pushed a commit to sourcegraph/sourcegraph that referenced this issue Feb 15, 2022
This commit fixes our Go import badges on sourcegraph.com, which have
been broken since api.godoc.org was brought down. There's a [tracking
issue](golang/go#36785) to re-introduce an
API for pkg.go.dev, but it's not implemented yet.

Instead, we implement a best-effort alternative that relies on searching
`go.mod` files for the given repo name on sourcegraph.com.
Sometimes packages will have different canonical names than their
repositories, and we don't handle that case.

There are [many open-source projects that have a broken badge](https://sourcegraph.com/search?q=context:global+sourcegraph.com+-/badge.svg&patternType=regexp).
After this lands, they will recover.

Co-authored-by: Erik Seliger <erikseliger@me.com>
Co-authored-by: Keegan Carruthers-Smith <keegan@sourcegraph.com>
@ZiViZiViZ

This comment was marked as off-topic.

@srenatus
Copy link
Contributor

@srenatus srenatus commented Feb 25, 2022

@ZiViZiViZ

This comment was marked as off-topic.

@tills13

This comment was marked as off-topic.

@ZiViZiViZ

This comment was marked as off-topic.

@Alphare

This comment was marked as off-topic.

@seankhliao
Copy link
Member

@seankhliao seankhliao commented Mar 3, 2022

please keep this issue on topic and refer questions to
https://github.com/golang/go/wiki/Questions

@shellscape
Copy link

@shellscape shellscape commented May 7, 2022

Rather than hiding what I'd consider a lot of really useful comments by folks (because I landed here for many of the same reasons they did, and there is extremely little available with regard to go package APIs) it would be infinitely more useful to explain to folks how to use the wiki for questions - or specify that tangential and related questions here are not welcome, and folks should seek the answers on the wiki. As it is now, that curt reply seems to imply that we should post questions on the wiki itself, which I doubt is what you're actually after.

The go team has been kicking this can for years, so users are naturally going to have a lot of questions. This reply #36785 (comment) specifically asks users what features they're after in an API - thus the issue is fair game for feature-related questions. Go packages are the very last major ecosystem package registry to lack a comprehensive API and that which doesn't make its data readily available. Because of this, people are naturally going to have a myriad of questions about trying to access that data. I'd humbly ask @seankhliao to be a bit more gracious in your moderation of this issue.

@golightlyb
Copy link
Contributor

@golightlyb golightlyb commented Jun 29, 2022

  • Use case: generating static documentation, or running a local documentation server, but wanting to be able to query (with javascript, at the time of the page view by a web browser) the "[Imported by: NNN]" number that appears if instead viewed on pkg.go.dev.

  • currently api.godoc.org/importers is returning {"error":{"message":"Internal Server Error"}}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest NeedsInvestigation pkgsite
Projects
None yet
Development

No branches or pull requests