-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/pkgsite: API for pkg.go.dev #36785
Comments
@rhcarvalho - it would be helpful to get a sense of what your current use cases are for api.godoc.org, and feature requests are for an API for pkg.go.dev. It sounds like getting the importers for a package is one of them. With pkg.go.dev being module aware, what specific information about importers would be useful to surface via an API? For example:
Additionally, what other information would be useful to you to surface via an API? /cc @tbpg who has also mentioned wanting an API for pkg.go.dev |
Without much thought, having an API to answer the more specific question "what are the importers of a specific version of a package" would make plausible to derive the answer to the other items in your list. At the moment I consume the godoc.org API and scrape data from pkg.go.dev to answer the question "who uses my package". As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing. Here are the other endpoints in the GoDoc API: https://github.com/golang/gddo/blob/7365cb292b8bfd13dfe514a18b207b9cc70e6ceb/gddo-server/main.go#L901-L904
So if we need a more specific request, here it is: api.go.dev MVP
|
Another endpoint idea (standalone or as part of a set of info returned for a module): available versions of a given module. |
What information beyond |
There are some cases where
None. That works. :) |
I'd be wary of encouraging tool authors to query pkg.go.dev/a proxy directly. Because it could well introduce skew compared to the answers from |
In my use-case I don't want to execute |
In #37952 I raised the question of whether module/package license file information could be surfaced in the output of @ianthehat instead suggested exposing the information via a pkg.go.dev API, leveraging the fact that the content and presentation on pkg.go.dev has already jumped through the relevant legal hoops. This comment is therefore to explicitly request that we include license file information in the API. Thank you! |
👋 Thanks for all the new Go tools like pkg.go.dev, they're super useful. Some input on this topic from the perspective of Libraries.io:
Currently to get the publish times for all versions, you have to make N+1 requests: one for
Anyone know if there are plans to fix this? |
No immediate plans. We currently gather that information from import statements in the code, so there is no version information attached. The So we understand it's an approximation and we want to fix it, but it's going to take some time. |
@julieqiu my use for an API would be to:
|
@julieqiu oh and please do not retire http://api.godoc.org/packages unless there is an alternative! |
As a downstream package maintainer for Fedora, I'm also interested in an API. We have our own tool, Anitya, to track package releases, but it was not designed to track GIT commits. And many Go packages still don't publish version. So any information about latest published commit, with info like date of the commit, new dependencies, would be very helpful. I would gather data from the API in Python and compare it to the latest version we have for a given package. I'm also interested in getting the License info, so we could find recursively all the licenses used in a static binary. |
and BTW my general use case is for https://github.com/nexB/scancode-toolkit and related projects to provide my users with license, origin and dependencies details. And for https://github.com/nexb/vulnerablecode where we provide vulnerabilities details. |
This commit fixes our Go import badges on sourcegraph.com, which have been broken since api.godoc.org was brought down. There's a [tracking issue](golang/go#36785) to re-introduce an API for pkg.go.dev, but it's not implemented yet. Instead, we implement a best-effort alternative that relies on searching `go.mod` files for the given repo name on sourcegraph.com. Sometimes packages will have different canonical names than their repositories, and we don't handle that case. There are [many open-source projects that have a broken badge](https://sourcegraph.com/search?q=context:global+sourcegraph.com+-/badge.svg&patternType=regexp). After this lands, they will recover. Co-authored-by: Erik Seliger <erikseliger@me.com> Co-authored-by: Keegan Carruthers-Smith <keegan@sourcegraph.com>
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
please keep this issue on topic and refer questions to |
Rather than hiding what I'd consider a lot of really useful comments by folks (because I landed here for many of the same reasons they did, and there is extremely little available with regard to go package APIs) it would be infinitely more useful to explain to folks how to use the wiki for questions - or specify that tangential and related questions here are not welcome, and folks should seek the answers on the wiki. As it is now, that curt reply seems to imply that we should post questions on the wiki itself, which I doubt is what you're actually after. The go team has been kicking this can for years, so users are naturally going to have a lot of questions. This reply #36785 (comment) specifically asks users what features they're after in an API - thus the issue is fair game for feature-related questions. Go packages are the very last major ecosystem package registry to lack a comprehensive API and that which doesn't make its data readily available. Because of this, people are naturally going to have a myriad of questions about trying to access that data. I'd humbly ask @seankhliao to be a bit more gracious in your moderation of this issue. |
|
For what it's worth, we have been running api.godoc.org as an unmonitored service (meaning we don't page people for it), approximately "best effort" although even that may be too generous. We discovered today that it has been down for 30+ days because the disk filled. Given that being down for a month was a non-issue, we've decided to leave it down. The godoc.org redirects will continue to run (approximately forever), but api.godoc.org will no longer be accessible. Or rather it will continue to be inaccessible. |
@rsc fair enough if this is the decision but the previous comment is mine, right at the start of that 30+ day period, saying it's down, and three people agreeing. If this is the decision, again, fine - but I submit that it shouldn't be based on people not complaining because to be fair I raised this pretty much as soon as it happened and it was never fixed and didn't have a response (fair enough- priorities) so anyone saying the same thing wouldn't be adding anything, and nobody knew they had a 30ish day deadline to object (I don't intend for this to sound harsh, just as a disagreement! 🙂) Edit: as an aside that might be helpful for people with an interest in this, Google's experimental Open Source Insights has an API. I wasn't aware of it until recently but it could very well resolve this entire conversation. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
I use the Dash doc viewer on my Mac which i find super useful for building a local index of all sorts of docs, Go included - Just found i couldn't install a new Go docset into it, presumably as it can no longer build an index from Would be nice to point the author towards an alternate index if this one isn't returning as the usefulness of Dash is greatly diminished for me without having rapid access to the docs of Go packages i use frequently. |
Dash 6.3.1 now relies on the web scraping. It works, but API would be better. |
FYI deps.dev launched a new API service. https://docs.deps.dev/api/v3alpha/ Its This does not cover the |
I was looking for an API to search for Go "commands" -- https://deps.dev/_/search/suggest?system=go&kind=package&q=axl (and then https://api.deps.dev/v3alpha/systems/go/packages/github.com%2Favamsi%2Faxl) kinda works (I don't mind that deps.dev/_/search is hacky and potentially unstable), but pkg.go.dev also annotates the commands as such (see https://pkg.go.dev/search?q=axl, for example), so it's easy to tell them apart from vanilla libraries. |
We are using that...but realized that for Go packages in particular, there is no publish date...which led me to this thread. My main real ask is that publish date be included for Go packages...otherwise my solution appears to be to call something like this for each package/version (https://proxy.golang.org/<module>/@v/v1.9.0.info). |
Rollback to previous scrape strategy as api.godoc.org is no longer available. See: golang/go#36785 (comment)
I also used api.godoc.org for my project https://github.com/dvob/go-project-usage/ |
I want to woke up this tread. https://docs.deps.dev/ that is referencing before does not contain any information about standard libraries packages (https://pkg.go.dev/std). Answering on why and who will use API: all researchers & data miners that needed to analyze open source packages (huge set of tools for code analyzes, code efficiency & vulnerabilities) (and it is only from my work-domain, I believe that there are more purposes) - we need information about package when we resolve vulnerabilities, when we investigate how languages growth, that dynamic on them, we need this information to track new releases & fixes & etc. Mostly all popular ecosystems for languages (maven, pypi, crates, conan, conda, etc) & operation systems like linux provide API to get this information - it could be DB dump, it could be REST API, github repo, anything, it could be huge csv files - any format will be better when scraping website. UPD: also, there is no ANY informations about packages (subpackages?) inside repo, for example, https://pkg.go.dev/vuln/GO-2021-0073 for Affected Package - github.com/git-lfs/git-lfs/lfsapi on deps.dev info only about 'root' packages like github.com/git-lfs/git-lfs and you cannot find any information that packages inside |
Prior to pkg.go.dev, godoc.org has had a JSON API that can be used to, among other things, discover importers of a given package.
Example: https://api.godoc.org/importers/golang.org/x/net/html
Given that pkg.go.dev does a much better job at tracking importers thanks to Go Modules and the Module Proxy, it would be nice if the community could get access to a public API similar to that of godoc.org.
The text was updated successfully, but these errors were encountered: