Skip to content

x/pkgsite: compute imported-by counts incrementally #81637

Closed
Closed
x/pkgsite: compute imported-by counts incrementally#81637
Assignees
jba
Labels
pkgsite
Milestone
Unreleased

Description

@jba
jba
opened on Sep 21, 2026

Background

The pkgsite worker computes the imported-by counts for all packages in a single computation, triggered by a Cloud Scheduler job. This has worked fine for years.

Problem

We've added counts for number of importing modules, and also there are many more packages than there used to be. So the request times out, exceeding the maximum Cloud Scheduler timeout of 30 minutes.

Solution

Compute the counts incrementally. Each Scheduler run will do as much as it can in the time budget.

It will be efficient to split the work up by package path. Both relevant tables, search_documents and imports_unique, are keyed by the package path, so we can efficiently scan any contiguous segment of the tables.

The main complication is storing progress. We really just need one value, the last package path reached previously. But we propose to store more information about each run: start and end time, number of packages processed, and error if any. This will give us insight into the runs so we can debug and optimize them. The new table, import_count_runs, won't take up significant space for a long time if ever, and we can always truncate it: at worst that will result in some duplicate work. (We could also periodically delete rows older than, say, a month.)

Alternative solutions

We could adjust the import counts as we insert and delete modules. This would spread the logic out in several places, and its correctness would not be obvious. Doing it all in one place is much clearer.

We could continue doing all the work at once, but with a detached context that doesn't get cancelled when the Cloud Scheduler job times out. This would be the smallest code change, but it has two problems:

  1. The scheduler jobs would always appear to succeed, even if the actual computation failed.
  2. The workers die periodically, so the work might not always finish, or might rarely finish. (For example, if the computation exceeds available memory, it would always fail and we wouldn't know if we didn't read the logs.)
  3. It doesn't scale. As the size of the search_documents table grows, the failures would happen more frequently.

Activity

  1. jba
    self-assigned this
    on Sep 21, 2026
  2. gopherbot
    added
    pkgsite
    on Sep 21, 2026
  3. gopherbot
    added this to the Unreleased milestone on Sep 21, 2026

  4. jba commented on Sep 21, 2026

    @jba
    jba
    on Sep 21, 2026
    ContributorAuthor

    /cc @ethanalee-work @hyangah

  5. ethanalee-work commented on Sep 22, 2026

    @ethanalee-work
    ethanalee-work
    on Sep 22, 2026
    Contributor

    Thanks for looking into this. I think it does make sense to ensure that each run does as much computation within the given time budget.

    I was wondering if you had considered an alternate approach that doesn't involve a new import_count_runs table and an alphabetical cursor.

    1. Handling packages that drop to 0 importers:
      If we scan imports_unique by package path, packages that lose all of their importers will never be encountered because rows are deleted from imports_unique when a module no longer imports them. As a result, their imported_by_count in search_documents would never be reset to 0.
    2. Cursor state management:
      Tracking last_package_path requires wrap-around handling (Z to A), crash recovery (if a run fails before persisting its cursor), handling overlaps/retries from Cloud Scheduler, and maintaining/cleaning up the import_count_runs table over time.
    3. Latency for new packages:
      A newly ingested package (e.g. at cloud.google.com/...) published while the cursor is at github.com/... would have to wait for the cursor to traverse the rest of the alphabet and wrap around before its initial count is computed.

    An alternative would involve no new table and just utilize the existing idx_search_documents_imported_by_module_count_updated_at index on search_documents

    In a loop bounded by the time budget we would:

    1. Pick the $N$ oldest packages (e.g. $N = 1{,}000$):

      SELECT package_path_id, package_path
FROM search_documents
ORDER BY imported_by_module_count_updated_at ASC NULLS FIRST
LIMIT $1
FOR UPDATE SKIP LOCKED;

    2. Fetch active importers for just those N packages: From imports_unique we could just retrieve the active importers for those packages.

    3. Compute counts and update: Update search_documents with the updated counts.

    This is just an idea, but we get the following benefits:

    1. It means that we would not need to perform any separate cursor management and maintain a separate table.
    2. There is immediate processing of new packages. NULLS FIRST ensures that newly ingested packages are processed on the next batch.

  6. jba commented on Sep 25, 2026

    @jba
    jba
    on Sep 25, 2026
    ContributorAuthor

    I basically did that before reading this. https://go.dev/cl/838325

    One difference: I only look at packages that haven't been updated in at least a week. That avoids constant writing to search_documents. (Assuming we get through everything in <1 week, which I think is feasible—the query is very fast. If not we could do it less frequently.) Most of the numbers change very slowly, so I think the delay is fine. I'm mindful of the write amplification from the updates to symbol_search_documents, and the fact that all those changes need to be replicated. (I'm not sure we'll be able to remove the symbol_search_documents columns after all, despite https://go.dev/cl/836405.)

    Doing nulls first is a good idea.

  7. gopherbot commented on Sep 25, 2026

    @gopherbot
    gopherbot
    on Sep 25, 2026
    Contributor

    Change https://go.dev/cl/838325 mentions this issue: internal/postgres: count module imports

  8. gopherbot commented on Sep 29, 2026

    @gopherbot
    gopherbot
    on Sep 29, 2026
    Contributor

    Change https://go.dev/cl/841245 mentions this issue: internal/postgres: update stale package importers

  9. gopherbot commented on Sep 29, 2026

    @gopherbot
    gopherbot
    on Sep 29, 2026
    Contributor

    Change https://go.dev/cl/836405 mentions this issue: internal/postgres: refine trigger for symbol search

  10. gopherbot commented on Sep 29, 2026

    @gopherbot
    gopherbot
    on Sep 29, 2026
    Contributor

    Change https://go.dev/cl/839545 mentions this issue: internal/postgres: use query for package importers

  11. gopherbot
    added 3 commits that reference this issue on Sep 30, 2026

    internal/postgres: count module imports

    6d94435

    internal/postgres: use query for package importers

    bae17e0

    internal/postgres: refine trigger for symbol search

    17c6cd8

  12. gopherbot commented on Oct 5, 2026

    @gopherbot
    gopherbot
    on Oct 5, 2026
    Contributor

    Change https://go.dev/cl/843545 mentions this issue: internal/postgres: simplify test-only import count func

  13. gopherbot
    added a commit that references this issue on Oct 5, 2026

    internal/postgres: update stale package importers

    affa6d3
  14. gopherbot
    closed this as completedin e8b22f1on Oct 6, 2026
  15. xushiwei
    added a commit that references this issue on Oct 9, 2026

    internal/postgres: simplify test-only import count func

    e8b22f1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

pkgsite

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions