Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: handle raw latest versions #44437

Open
jba opened this issue Feb 19, 2021 · 11 comments
Open

x/pkgsite: handle raw latest versions #44437

jba opened this issue Feb 19, 2021 · 11 comments
Assignees
Labels

Comments

@jba
Copy link
Contributor

@jba jba commented Feb 19, 2021

Let's say that a module's raw latest version is its latest version before considering retractions or other modifications—or, in the case of pkgsite, whether the module is even considered "good," that is, worthy of being served. (Examples of "bad" modules are those with no .go files, and those whose go.mod's module path does not match the path that the proxy serves them from.)

The raw latest version is important, because its go.mod file defines whether the module is deprecated and which versions are retracted.

Pkgsite's notion of a module's raw latest version should match the go command's. Its database should store the raw latest version of every module it knows about to make it easy to get deprecation and retraction information.

This issue is a prerequisite for #41321 and #43265.

@jba jba added this to the pkgsite/unplanned milestone Feb 19, 2021
@jba jba self-assigned this Feb 19, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 22, 2021

Change https://golang.org/cl/295193 mentions this issue: internal/version: define Later and Latest

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/295195 mentions this issue: internal/discovery: add retraction fields

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/295196 mentions this issue: internal: types and functions for the raw latest version

gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 23, 2021
Define helper functions that implement a part of the go
command's definition of "latest": preferring release
to pre-release versions.

We will use Latest in subsequent CLs to implement
the full definition.

For golang/go#44437

Change-Id: I56f6448292d1ff373266da03d830e2882e6016c3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295193
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/295429 mentions this issue: internal/fetch: functions for fetching raw latest version

gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 23, 2021
Add fields to hold retraction information.

Also, switch deprecation to a pair of bool and string rather than a
pointer, because it's clearer.

For golang/go#44437

Change-Id: Ica771c5ec8e5c0abeec16f9d998339ef60b401a5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295195
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/295430 mentions this issue: internal/proxydatasource: implement deprecation and retractions

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/295449 mentions this issue: migrations: add raw_latest_versions table

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/295450 mentions this issue: internal/postgres: get and update raw_latest_versions

gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 23, 2021
Add RawLatestInfo, which describes the raw latest version of a module,
and provide functions to compute deprecation and retractions from it.

For golang/go#44437

Change-Id: I2f1aa03bf190b961642828fdc7b708e2ee47a20a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295196
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
Add RawLatestInfo, which uses the proxy to find the raw latest version
of a module and its associated go.mod file.

The fetchRawLatestVersion function implements the full definition of
"raw latest" as embodied in the current (go 1.16) go command.  In some
cases this requires looking for the presence of a go.mod file.
Although we could always download the zip from the proxy, the function
accepts a function argument that can avoid that. The worker will pass
a function that looks in the database.

For golang/go#44437

Change-Id: I84e19a4a7a10a87d5664217e0404bc795b3f9470
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295429
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
When fetching a module at a version, the proxy datasource uses
fetch.RawLatestInfo to get the go.mod file at the raw latest version
of the module, then uses internal.RawLatestInfo.PopulateModule to
determine whether the module version is deprecated or retracted.

Also, add some proxy test modules to facilitate testing.

For golang/go#41321
For golang/go#43265
For golang/go#44437

Change-Id: I312346d72f656e598ad170135046ef85da8e9b11
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295430
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
Add a table that will let us track the raw latest version
of each module, and store its go.mod file as well.

For golang/go#44437

Change-Id: I7cf9530a306a87d0c399b326ccffb7033e33461d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295449
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 24, 2021

Change https://golang.org/cl/295889 mentions this issue: internal/worker: call WithFetchDisabled sooner

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 24, 2021

Change https://golang.org/cl/295890 mentions this issue: internal/worker: update raw latest info on each fetch

gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
Add functions to get a row from the raw_latest_versions table,
and to update a row if the new version is later.

For golang/go#44437

Change-Id: I9f44b815b4eb42c6cd286f10f2e946e753c38989
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295450
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
Convert the proxy client to one which doesn't fetch at the beginning
of the request, so we don't have to pass a boolean argument through
several layers.

For golang/go#44437

Change-Id: I0495b5673cc399a1cdc16349d53ba4a9a4ecc380
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295889
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 24, 2021

Change https://golang.org/cl/295894 mentions this issue: internal/postgres: upsert path correctly

gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
Check for a row in the paths table before inserting.

For golang/go#44437

Change-Id: I763867c27b1d729311b7ec545c5cdab4e9b64036
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295894
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 24, 2021
Every time a module is fetched, update its raw latest information in
the database to the most recent value.

This is a simple way of making sure the DB has the most recent raw
latest info.  In the steady state it's unlikely to be wasteful, since
most of the time the new module version will be the latest, so the
update is needed.

For golang/go#44437

Change-Id: If64bca41b51fe63fb2aeb836d6ed98543090cd18
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295890
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 26, 2021

Change https://golang.org/cl/296814 mentions this issue: internal/worker: move raw-latest computation to FetchAndUpdateState

gopherbot pushed a commit to golang/pkgsite that referenced this issue Feb 26, 2021
To make sure it happens in our integration test.

For golang/go#44437

Change-Id: I0e921b067f6798d108880eae152edc5264168e74
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296814
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants