fix: bq dataset ingest worker didnt start because of temporal lib issue (CM-1227)

[themarolt]

Summary

Fixes several bugs that prevented the bq-dataset-ingest worker from starting and running successfully in production. The worker was crash-looping on boot due to a webpack bundling error, and subsequent runs hit GCS permission errors, Temporal connection failures, and BQ byte-ceiling
misconfigurations.

Changes

• Fix webpack crash on boot: scorecardSql.ts imported SCORECARD_DATASET from config.ts, which instantiates BigQuery/Storage clients at module load — dragging the SDK into the Temporal workflow sandbox bundle and failing on node:util/node:zlib. Inlined the dataset string literal to
  match the existing pattern in deps-dev/queries/*.ts
• Fix Temporal connection in triggerBootstrap.ts: Script was using a plain non-TLS connection and reading the wrong namespace (CROWD_TEMPORAL_NAMESPACE = main CDP namespace instead of packages namespace). Replaced with getTemporalClient() + TEMPORAL_CONFIG() from @crowd/temporal
  which handles mTLS correctly
• Clean up packages_worker scripts: Removed redundant CROWD_TEMPORAL_NAMESPACE=$CROWD_PACKAGES_TEMPORAL_NAMESPACE inline overrides from all start:/dev: scripts — k8s manifests now override CROWD_TEMPORAL_NAMESPACE directly. Added trigger-bootstrap, trigger-bootstrap:local,
  monitor:osspckgs, monitor:osspckgs:local npm scripts. Fixed monitor:osspckgs to point to src/scripts/monitorOsspckgs.ts instead of deleted .mjs file
• BQ byte-ceiling env overrides: Added per-kind runtime override via BQ_DATASET_INGEST_<KIND>_MAX_BQ_GB env vars. Bumped package_dependencies ceiling 3000→10000 (incremental always scans ~2 full-day partitions, ~3.85TB by design), scorecard_repos 10→50, scorecard_checks 200→500
• Fix scorecard deadlock: mergeStagingToTable had maximumAttempts: 1 — a deadlock from a concurrent writer (github-repos-enricher) caused instant workflow failure. Bumped to 3 retries with 30s backoff. Added ORDER BY repo_url to the repos merge to enforce consistent lock order. Added
  skip-unchanged filter to reduce rows written
• Fix scorecard_checks dupe error: BQ source has duplicate (repo_url, check_name) rows → ON CONFLICT DO UPDATE command cannot affect row a second time. Fixed with DISTINCT ON (repo_url, check_name) in the INSERT SELECT
• Add @crowd/temporal dep to packages_worker
• Add src/deps-dev/README.md documenting all BQ ceiling overrides, env vars, and how to run/trigger

Type of change

• Bug fix
• Chore / dependency update
• Documentation

JIRA ticket

https://linuxfoundation.atlassian.net/browse/CM-1227

---

Note

Medium Risk
Touches production ingest (Temporal, BigQuery billing ceilings, and Postgres merge behavior); changes are targeted bug fixes but misconfigured ceilings or merge logic could still fail or over-bill runs.

Overview
Restores bq-dataset-ingest so the worker can start and complete scorecard/deps exports in prod.

Workflow bundle / boot: scorecardSql.ts no longer imports deps-dev/config (which pulled BigQuery/Storage into the Temporal workflow sandbox). OpenSSF tables are referenced with a literal `openssf.scorecardcron.scorecard-v2_latest` string instead.

Temporal ops: triggerBootstrap.ts uses @crowd/temporal (TEMPORAL_CONFIG, getTemporalClient) for mTLS and namespace config. npm start:/dev: scripts drop inline CROWD_PACKAGES_TEMPORAL_NAMESPACE overrides; new trigger-bootstrap and monitor:osspckgs scripts replace the old monitor .mjs path.

BQ cost guards: bqExportToGcs honors BQ_DATASET_INGEST_<KIND>[_<MODE>]_MAX_BQ_GB env overrides (documented in src/deps-dev/README.md). Default ceilings rise for package_dependencies (10TB), scorecard_repos, and scorecard_checks.

Scorecard Postgres merges: mergeStagingToTable retries increase (3× with backoff). Repos merge uses ORDER BY repo_url, skips unchanged rows, and checks merge uses DISTINCT ON (repo_url, check_name) to avoid duplicate-key failures from BQ dupes and deadlocks with concurrent writers.

^{Reviewed by Cursor Bugbot for commit 8a77284. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR addresses multiple production issues in the bq-dataset-ingest flow inside packages_worker, focusing on getting the worker to start reliably (Temporal client config / bundling) and to run successfully (BQ export ceilings and scorecard ingest robustness).

Changes:

• Updates triggerBootstrap to use the shared @crowd/temporal client/config instead of direct non-TLS Temporal connection wiring.
• Improves scorecard ingest robustness (retries on merge activity, reduced write volume, and de-duping of scorecard checks staging rows).
• Adds runtime-configurable BQ dry-run byte ceilings via env overrides and documents how to run/operate the pipeline.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
services/apps/packages_worker/src/scripts/triggerBootstrap.ts	Switches Temporal connection logic to @crowd/temporal client/config.
services/apps/packages_worker/src/scorecard/workflows/ingestScorecard.ts	Adds merge retries, reduces unnecessary updates, and de-dupes scorecard checks before upsert.
services/apps/packages_worker/src/scorecard/queries/scorecardSql.ts	Inlines the Scorecard BigQuery table reference to avoid importing config that drags SDKs into workflow bundling.
services/apps/packages_worker/src/deps-dev/workflows/ingestDependencies.ts	Raises the default BigQuery ceiling for package_dependencies.
services/apps/packages_worker/src/deps-dev/README.md	New operational/runbook-style documentation for the ingest pipeline and env overrides.
services/apps/packages_worker/src/deps-dev/activities/bqExportToGcs.ts	Adds per-kind env var overrides for the BQ dry-run ceiling.
services/apps/packages_worker/package.json	Cleans up scripts and adds helper scripts for triggering/monitoring.
pnpm-lock.yaml	Adds the workspace link for @crowd/temporal.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4c01afe. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file