Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: migrate codebase to paths-based data model #39629

Open
julieqiu opened this issue Jun 17, 2020 · 22 comments
Open

x/pkgsite: migrate codebase to paths-based data model #39629

julieqiu opened this issue Jun 17, 2020 · 22 comments
Assignees
Labels
Milestone

Comments

@julieqiu
Copy link
Contributor

@julieqiu julieqiu commented Jun 17, 2020

As part of the redesign that we have planned for this year, we will be migrating our data model to use a paths table.

Structs and functions to be deprecated will be prefixed with "Legacy".

Related: #39621

TODOs:

  • populate new tables
    • update worker/postgres code to insert new data into new tables (paths, documentation, readmes, and package_imports)
  • add new postgres functions
    • #40032: add GetModuleInfo function to replace LegacyGetModuleInfo
    • update postgres.GetImports to read from package_imports table
    • add function to replace LegacyGetDirectory
    • add GetDirectory function to return data for a path
    • #40150: add a function to get the version history for a path
    • #40027: add GetLicenses function to return licenses for a given path
  • update frontend code:
    • add new servePackagePage code path, such that:
      • doc tab reads from documentation table
      • overview tab reads from readmes table
      • subdirectories tab reads from paths table
      • versions tab reads from paths table
      • imports tab reads from package_imports table
      • licenses tab reads from paths table
    • add new serveModulePage code path, such that:
      • overview tab reads from readmes table
      • packages tab reads from paths table
      • versions tab reads from paths table
      • licenses tab reads from paths table
    • add new serveDirectoryPage code path, such that:
      • overview tab reads from readmes table
      • subdirectories tab reads from paths table
      • versions tab exists, and reads from paths table
      • licenses tab reads from paths table
    • update latest badge to read from paths table
    • update search to read from paths table
  • turn on experiment flags
    • InsertDirectories
    • UseDirectories
    • UsePathInfo
    • UsePackageImports
  • replace use of Legacy functions and deprecate
    • LegacyGetDirectory
    • LegacyGetModuleLicenses
    • LegacyGetPackage
    • LegacyGetPackageLicenses
    • LegacyGetPackagesInModule
    • LegacyGetModuleInfo
  • delete old tables / columns
    • table packages (replaced by paths and documentation tables)
    • table imports (replaced by package_imports table)
    • columns modules.readme_file_path, modules.readme_contents (replaced by readmes table)
  • remove experiment flags
    • InsertDirectories
    • UseDirectories
    • UsePathInfo
    • UsePackageImports
@julieqiu julieqiu self-assigned this Jun 17, 2020
@gopherbot gopherbot added this to the Unreleased milestone Jun 17, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 1, 2020

Change https://golang.org/cl/240680 mentions this issue: internal/frontend: use paths table to check for existence

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 1, 2020
A new experiment is added, ExperimentUsePathInfoToCheckExistence, which
uses the paths table to check if a package/module/directory exists,
regardless of whether we are using the old or new data model to render
the pages.

This is a first step to use the new paths-based data model. It also
streamlines the flow for frontend fetches.

Updates golang/go#39629

Change-Id: I6178776264cbe71ffb3fb87b3409df902b33b2d4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240680
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 4, 2020

Change https://golang.org/cl/240938 mentions this issue: internal: rename GetImports to LegacyGetImports

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 7, 2020

Change https://golang.org/cl/241181 mentions this issue: internal/postgres: read from package_imports

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 7, 2020

Change https://golang.org/cl/241182 mentions this issue: internal/frontend: use GetPathInfo for search

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 7, 2020

Change https://golang.org/cl/241183 mentions this issue: internal/frontend: use GetPathInfo for latest version

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 7, 2020

Change https://golang.org/cl/241064 mentions this issue: internal/worker: replace GetModuleInfo with LegacyGetModuleInfo

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 7, 2020
GetImports now reads from the package_imports table when the
"use-package-imports" experiment is on.

Updates golang/go#39629

Change-Id: I80970eb4806c8ccf2c01e5dac6a04ff0978fd681
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241181
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 7, 2020
Tests in internal/worker now use GetModuleInfo instead of
LegacyGetModuleInfo.

Updates golang/go#39629

Change-Id: I524750809c3e2dcf795bb026636fb76aeba280db
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241064
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 7, 2020
The search endpoint now uses GetPathInfo to check if a package
exists, when the "use-path-info" experiment is on.

Updates golang/go#39629

Change-Id: I5c4b3c3250e06a33a647aac74f91849ceb4105ca
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241182
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 7, 2020
LatestVersion now uses GetPathInfo to check for the latest version of a
path, when the "use-path-info" experiment is on.

Updates golang/go#39629

Change-Id: I374f707cb06ccd8147925229eb48f2d08caf0a4f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241183
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 7, 2020

Change https://golang.org/cl/241320 mentions this issue: internal: delete insert-directories experiment

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 7, 2020
The insert-directories experiment flag is deleted, since we have already
inserted data for all modules into the paths, package_imports,
documentation, and readmes table, and been running that code path for a
while.

Updates golang/go#39629

Change-Id: I323850a462672c41ad0c67b6ab2b173bb32bf441
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241320
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 8, 2020

Change https://golang.org/cl/241397 mentions this issue: internal/frontend: add serveDetailsPage

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 9, 2020

Change https://golang.org/cl/241441 mentions this issue: internal/frontend: merge createPackage functions

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 9, 2020
serveDetailsPage is added, which replaces servePackagePageNew in serving
details pages when the "use-directories" experiment flag is on. Like
servePackagePageNew, serveDetailsPages supports the package/directory
views. It also supports the module view, so that we can stop using
LegacyModuleInfo and reading from modules.readme_file_path and
modules.readme_contents.

For golang/go#39629

Change-Id: I8e664bf1e9174a630db6b723c949ba1e9aa0dc9b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241397
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 9, 2020
The two functions for creating a frontend.Package, legacyCreatePackage
and createPackageNew, are merged into a single createPackage function.

createPackage uses the new PackageMeta type, as input. It does not take
any legacy structs as input and returns a frontend.Package.

In future CLs, PackageMeta will be used to LegacyDirectory, when
fetching data for the directories tab.

For golang/go#39629

Change-Id: I80ec5272c6f8e237f0752938a89c633e3a1b81f5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241441
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241839 mentions this issue: internal/frontend: update legacy and new terminology

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241859 mentions this issue: internal/frontend: add serveDirectoryPage

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241838 mentions this issue: internal/postgres: add GetPackagesInDirectory

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241898 mentions this issue: internal/postgres: move version code to separate file

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241899 mentions this issue: internal: add Legacy prefix to GetPsuedoVersions and GetTaggedVersions functions

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241900 mentions this issue: internal: remove New suffix from structs

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
…s functions

The GetPsuedoVersions* and GetTaggedVersions* functions read from the
packages table, and will be replaced by a single GetVersionsForPath
function.

These functions are now prefixed with "Legacy" to indicate that they
will be deprecated.

For golang/go#39629

Change-Id: I7f89f9890f135b5ddb363a51a9706e48d02594b5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241899
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
GetPackagesInDirectory is added, which uses the paths table to get all
of the packages within a given directory path. This will replace
LegacyGetDirectory and LegacyGetPackagesInModule in creating the
directories tab.

For golang/go#39629

Change-Id: I4b39690e838756421bee9b5ca389081e184761bf
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241838
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
The use of "legacy" and "new" is updated in the frontend package, so
that "legacy" is used as prefix to any function that will be removed once
the data migration is completed, or currently contains legacy types.
Otherwise, the function will remain after the data migration. The "new"
suffix is removed from these functions if it was present.

For golang/go#39629

Change-Id: Iae88fef8c53073442a8eeb0eeb54a3296448949e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241839
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
This change moves code related to fetching version history out of
details.go into its own file, versions.go. There are no code changes.

For golang/go#39629

Change-Id: I6e12e7829279a2116d3d455fd5a9d61be11a011f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241898
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
A serveDirectoryPage function is added, which serves pages for the
directory view using a versioned directory.

For golang/go#39629

Change-Id: I372981dfa7822d850aac59ffd9fea2aab8949baa
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241859
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
The New suffix is removed from PackageNew and DirectoryNew, since the
legacy methods are already prefixed with "Legacy".

For golang/go#39629

Change-Id: I8be32310c6b451035021663000aac6ca845ee6e0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241900
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/241958 mentions this issue: internal/frontend: merge effectiveName, packageHTMLTitle, and packageTitle functions

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 10, 2020
…Title functions

effectiveNameNew, packageHTMLTitleNew, and packageTitleNew are deleted.

packageHTMLTitle and packageTitle are updated to accept a pkgPath
and pkgName.

effectiveName, packageHTMLTitle, and packageTitle are used in place of
the "new" methods.

For golang/go#39629

Change-Id: Ia6d9a5d0e648381b721abc8e266e8c1b93645c00
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241958
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@julieqiu julieqiu changed the title x/pkgsite: migrate frontend code to paths-based data model x/pkgsite: migrate codebase to paths-based data model Jul 10, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 15, 2020

Change https://golang.org/cl/242581 mentions this issue: internal/frontend: rearrange functions in tabs.go

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 15, 2020

Change https://golang.org/cl/242580 mentions this issue: internal/frontend: add legacy prefix to fetch*VersionDetails

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 15, 2020

Change https://golang.org/cl/242778 mentions this issue: internal/frontend: use GetLicenses

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 15, 2020

Change https://golang.org/cl/242777 mentions this issue: internal/postgres: add support for stdlib

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 15, 2020
The "legacy" prefix is added to fetchPackageVersionDetails and
fetchModuleVersionDetails, since these will be replaced with a
fetchVersionDetails function that reads from the paths table.

For golang/go#39629

Change-Id: I059a26cb5998b831e9be725a055ac1ce9692fee4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242580
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 15, 2020
The functions in tab.go are rearranged so that ones with the "legacy"
prefix are moved to the bottom of the file.

For golang/go#39629

Change-Id: Ia0b1229e67eadc3dc7d26a8f4a02eab458e7de6b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242581
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 15, 2020
GetLicenses currently does not handle the std module or packages in the
stdlib. Support is added for those paths.

For golang/go#39629

Change-Id: I8e41cec7004111608c712b5d34f2ad07aede15b6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242777
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 16, 2020
GetLicenses is now used by the frontend to fetch license data when the
"use-directories" experiment is on.

For golang/go#39629

Change-Id: I7a4c358b9e4f0d0089c301a40703becda1d5514d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242778
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 16, 2020

Change https://golang.org/cl/242881 mentions this issue: internal/postgres: add a function to get the version history for a path

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 23, 2020
Adds a db method to get version history for a given path. It will
return pre-release and release versions of a package if tagged
versions exist or the first 10 pseudo versions.

In supporting getting version history based on path, this method
will replace the following legacy methods:
- LegacyGetTaggedVersionsForPackageSeries
- LegacyGetPsuedoVersionsForPackageSeries
- LegacyGetTaggedVersionsForModule
- LegacyGetPsuedoVersionsForModule

For golang/go#39629
For golang/go#40150

Change-Id: I251e80e5327e422579fd8c0854f1e93d1865d48d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242881
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 25, 2020

Change https://golang.org/cl/244608 mentions this issue: internal/stdlib: fix stdlib README for new data model

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jul 27, 2020
The overview tab for the stdlib package is currently displaying
src/README.vendor, instead of README.md, when the use-directories
experiment flag is on. This is because based on the way the std
module zip is constructed, both README.md and README.vendor are present
at the root for version v1.4.0-beta.1 and higher. The
fetch process will then choose README.vendor to include, since
it comes after README.md alphabetically.

The README.vendor file is no longer included in the std zip.

For golang/go#39629

Change-Id: I1c9d8fcd1776f7b58be08f5f1ec1dc1b735e736d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244608
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.