feat: add rubygems ecosystem to bq-dataset-ingest (CM-1296)

[themarolt]

Summary

Adds RubyGems as a first-class ecosystem to the bq-dataset-ingest (osspckgs) pipeline, and folds in two operational improvements developed alongside it: resumable package_dependencies ingests and removal of the index drop/recreate step. RubyGems is manifest-sourced (deps.dev Scenario B) — packages/versions/repos/advisories come from the standard *Latest views; dependencies come from RubyGemsRequirementsLatest.

Changes

• RubyGems ecosystem — added RUBYGEMS to the systems filter, deps SQL (full + incremental via RubyGemsRequirements, RuntimeDependencies only), createVersionsLookup, triggerBootstrap CLI, and the monitor's known-ecosystems list.
• Resumable deps ingest — new getResumeExport activity + getIngestJobForResume DAL query let a partially-merged package_dependencies job resume by id, reusing its GCS export and chunk boundaries (skips the multi-hour BQ export on retry).
• Removed index drop/recreate — dropped the manage{PackageDeps,Versions}{Indexes,Constraints} activities and the drop→load→rebuild flow for all job kinds. full now merges against live indexes via ON CONFLICT DO NOTHING (idempotent). The day-long rebuild was only needed for the original NPM onboarding.
• Monitor — buffered redraw to eliminate flicker; shows the merging phase step during the merge loop.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

JIRA ticket

https://linuxfoundation.atlassian.net/browse/CM-1296

---

Note

High Risk
Touches multi-billion-row package_dependencies/versions load behavior (dropping the index-off fast path) and new resume semantics where wrong meta or file-count checks could skip or mis-merge data; resume paths fail loudly on invalid state.

Overview
RubyGems is wired through the osspckgs pipeline as a manifest-sourced ecosystem (like GO/NuGet): RUBYGEMS in default systems/CLI validation, RubyGemsRequirementsLatest / RubyGemsRequirements for full and incremental package_dependencies SQL (RuntimeDependencies only), and rubygems in the versions lookup allowlist and monitor.

Resumable deps ingest adds --resume-job <id> (deps-only, with bootstrap guards), getIngestJobForResume + getResumeExport, and workflow logic that reuses a prior job’s GCS parquet, restores ecosystems / fill from job meta, validates file counts, and restarts the chunk loop with overlap-safe merges. bqExportToGcs now persists meta:fill (and ecosystems on export) and re-stamps fill intent when reusing exports so resume picks the correct merge SQL (fill upsert vs DO NOTHING).

Ingest refactor: the four manage*Indexes / manage*Constraints activities are removed; versions and package_dependencies full/incremental paths always merge against live constraints using DISTINCT ON + ON CONFLICT. Deps merge timeouts go to 4h; monitor gets single-frame redraw and file-based total ETA.

^{Reviewed by Cursor Bugbot for commit 359a4cb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR extends the osspckgs bq-dataset-ingest pipeline to treat RubyGems as a first-class ecosystem, while also changing how large versions / package_dependencies full loads are merged (no longer dropping/rebuilding indexes & constraints) and adding a resume-by-job-id path for partially merged package_dependencies ingests.

Changes:

• Add RUBYGEMS to ecosystem/system filters, deps.dev dependency SQL (full + incremental), versions lookup creation, CLI bootstrap trigger, and the monitor’s known ecosystem list.
• Add resumable package_dependencies ingest support via --resume-job <id> (reuses a prior job’s GCS export and recorded progress to skip re-exporting from BigQuery).
• Remove the drop → load → rebuild index/FK workflow for versions and package_dependencies, using ON CONFLICT merges against live constraints instead; improve monitor rendering/ETA behavior.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/osspckgs/ingestJobs.ts	Adds DAL query to fetch resume-relevant ingest job fields (gcsPrefix + progress + merged rows).
services/apps/packages_worker/src/scripts/triggerBootstrap.ts	Adds RUBYGEMS to CLI and introduces --resume-job argument validation + propagation.
services/apps/packages_worker/src/scripts/monitorOsspckgs.ts	Buffered redraw to reduce flicker; improves status/ETA logic; adds rubygems to known ecosystems.
services/apps/packages_worker/src/deps-dev/workflows/ingestVersions.ts	Removes drop/rebuild flow; merges with ON CONFLICT DO NOTHING against live unique index.
services/apps/packages_worker/src/deps-dev/workflows/ingestDependencies.ts	Adds resume flow and removes drop/rebuild; increases merge timeout; sets step to merging.
services/apps/packages_worker/src/deps-dev/workflows/bootstrapOsspckgs.ts	Threads resumeJobId through bootstrap and skips watermark checks in resume mode.
services/apps/packages_worker/src/deps-dev/queries/systems.ts	Adds RUBYGEMS to default and valid system lists.
services/apps/packages_worker/src/deps-dev/queries/depsSql.ts	Adds RubyGems full+incremental dependency extraction branches.
services/apps/packages_worker/src/deps-dev/activities/manage.ts	Deletes index/constraint management activities (no longer used).
services/apps/packages_worker/src/deps-dev/activities/index.ts	Removes exports of deleted manage* activities; exports new getResumeExport.
services/apps/packages_worker/src/deps-dev/activities/getResumeExport.ts	Adds activity to fetch resume metadata for a prior ingest job.
services/apps/packages_worker/src/deps-dev/activities/createVersionsLookup.ts	Allows rubygems ecosystem in versions lookup creation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c805516. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.