chore: add created_at/updated_at to packages-db schemas for Tinybird sync by joanagmaia · Pull Request #4178 · linuxfoundation/crowd.dev

joanagmaia · 2026-06-08T16:11:49Z

Summary

Adds a migration (V1780600000__add_created_updated_at.sql) to add created_at / updated_at audit columns to all packages-db tables that need them, enabling Tinybird watermark-based sync.
Updates all INSERT/UPSERT queries across packages_worker and data-access-layer to populate the new columns.
Excludes tables that already have equivalent timestamp columns (e.g. package_name_history, audit logs, state-machine tables).

Notes

repos already has created_at (stores the GitHub repo creation date). last_synced_at is nullable (not set until the GitHub enricher runs), so updated_at is added as the non-nullable Tinybird watermark; last_synced_at is kept as a separate enrichment-freshness signal.
packages, versions, repo_docker: last_synced_at is NOT NULL DEFAULT NOW() and already serves as updated_at — only created_at is added.
package_repos: verified_at is NOT NULL DEFAULT NOW() and already serves as updated_at — only created_at is added.
Partitioned tables (versions, package_dependencies, downloads_daily, downloads_last_30d): PostgreSQL 12+ propagates new columns to all child partitions automatically.

🤖 Generated with Claude Code

Note

Medium Risk
Wide schema migration on large partitioned tables plus many write paths; Maven maintainer email handling changed alongside timestamp columns.

Overview
Adds created_at / updated_at on packages-db tables (via V1780600000__add_created_updated_at.sql) so Tinybird can watermark incremental sync, with documented exceptions where existing timestamps already play that role (last_synced_at, verified_at, log/state tables).

Ingestion and DAL now set the new columns on insert/upsert across deps.dev merges (packages, versions, deps, advisories, package_repos), npm/Maven/OSV paths, downloads, funding links, maintainers, and repo enrichment (updated_at on GitHub bulk updates and skip marks).

Maven maintainer upserts are aligned with npm by writing email instead of email_hash in upsertMaintainer / runMavenEnrichmentLoop (no schema change in this migration).

^{Reviewed by Cursor Bugbot for commit b435ad6. Bugbot is set up for automated code reviews on this repo. Configure here.}

…sync Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

github-actions · 2026-06-08T16:12:04Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

Copilot

Pull request overview

This PR adds created_at / updated_at audit timestamps across the packages-db schema so Tinybird can use timestamp watermarks for incremental sync, and updates the main write paths (packages_worker + DAL) to populate/bump these columns.

Changes:

Add Flyway migration V1780600000__add_created_updated_at.sql to introduce created_at / updated_at (and updated_at only for repos).
Update DAL INSERT/UPSERT queries to write created_at / updated_at and bump updated_at on conflict updates.
Update packages_worker deps.dev ingestion and repo enrichment SQL to set/bump updated_at.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
services/libs/data-access-layer/src/packages/versions.ts	Adds `created_at`/`updated_at` to npm versions upsert and bumps `updated_at` on conflict.
services/libs/data-access-layer/src/packages/repos.ts	Adds `created_at`/`updated_at` to `package_repos` upsert and bumps `updated_at` on conflict.
services/libs/data-access-layer/src/packages/packages.ts	Adds `created_at`/`updated_at` to npm package upsert and bumps `updated_at` on conflict.
services/libs/data-access-layer/src/packages/osv.ts	Adds `created_at`/`updated_at` to OSV advisory-related inserts/upserts and bumps `updated_at` on conflict.
services/libs/data-access-layer/src/packages/maintainers.ts	Adds `created_at`/`updated_at` to maintainers + package_maintainers writes and bumps `updated_at` on conflict.
services/libs/data-access-layer/src/packages/fundingLinks.ts	Adds `created_at`/`updated_at` to funding link insert path.
services/libs/data-access-layer/src/packages/downloadsLast30d.ts	Adds `created_at`/`updated_at` to last-30d downloads upsert and bumps `updated_at` on conflict.
services/libs/data-access-layer/src/packages/downloadsDaily.ts	Adds `created_at`/`updated_at` to daily downloads insert path.
services/apps/packages_worker/src/enricher/updateEnrichedRepos.ts	Bumps `repos.updated_at` when enrichment updates or when marking repos skipped.
services/apps/packages_worker/src/deps-dev/workflows/ingestVersions.ts	Adds `created_at`/`updated_at` to deps.dev versions merge inserts.
services/apps/packages_worker/src/deps-dev/workflows/ingestRepos.ts	Adds `updated_at` to deps.dev repos inserts; adds `created_at`/`updated_at` to `package_repos` inserts.
services/apps/packages_worker/src/deps-dev/workflows/ingestPackages.ts	Adds `created_at`/`updated_at` to deps.dev packages merge inserts.
services/apps/packages_worker/src/deps-dev/workflows/ingestDependencies.ts	Adds `created_at`/`updated_at` to deps.dev dependencies merge inserts.
services/apps/packages_worker/src/deps-dev/workflows/ingestAdvisories.ts	Adds `created_at`/`updated_at` to deps.dev advisories/advisory_packages/affected_ranges inserts.
services/apps/packages_worker/src/deps-dev/activities/rankPackagesUniverse.ts	Adds `created_at`/`updated_at` to `packages_universe` repopulation insert.
backend/src/osspckgs/migrations/V1780600000__add_created_updated_at.sql	Adds audit timestamps to packages-db tables for Tinybird watermarking.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+ALTER TABLE packages_universe
+  ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW(),
+  ADD COLUMN IF NOT EXISTS updated_at timestamptz NOT NULL DEFAULT NOW();
+
+ALTER TABLE packages
+  ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW(),
+  ADD COLUMN IF NOT EXISTS updated_at timestamptz NOT NULL DEFAULT NOW();


…ists Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

…ding deprecation) Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

+ALTER TABLE packages
+  ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW();
+
+ALTER TABLE package_funding_links
+  ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW(),
+  ADD COLUMN IF NOT EXISTS updated_at timestamptz NOT NULL DEFAULT NOW();


cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 04da2d3. Configure here.}

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

+ALTER TABLE packages
+  ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW();
+


Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

+ALTER TABLE repos
+  ADD COLUMN IF NOT EXISTS updated_at timestamptz NOT NULL DEFAULT NOW();
+
+ALTER TABLE packages
+  ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW();


         ON CONFLICT (ecosystem, username) DO UPDATE SET
           display_name = COALESCE(EXCLUDED.display_name, maintainers.display_name),
-           email        = COALESCE(EXCLUDED.email, maintainers.email)
+           email        = COALESCE(EXCLUDED.email, maintainers.email),
+           updated_at   = EXCLUDED.updated_at
         RETURNING display_name, email


      ON CONFLICT (ecosystem, username) DO UPDATE SET
        display_name = COALESCE(EXCLUDED.display_name, maintainers.display_name),
        url          = COALESCE(EXCLUDED.url,          maintainers.url),
-        email_hash   = COALESCE(EXCLUDED.email_hash,   maintainers.email_hash)
-      RETURNING id, display_name, url, email_hash
+        email        = COALESCE(EXCLUDED.email,        maintainers.email),
+        updated_at   = NOW()


Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

chore: add created_at/updated_at to packages-db schemas for Tinybird …

7c61585

…sync Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

Copilot AI review requested due to automatic review settings June 8, 2026 16:11

Copilot started reviewing on behalf of joanagmaia June 8, 2026 16:12 View session

joanagmaia requested review from epipav and themarolt June 8, 2026 16:13

joanagmaia assigned mbani01 and unassigned mbani01 Jun 8, 2026

joanagmaia requested review from mbani01 and ulemons June 8, 2026 16:13

cursor Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread services/libs/data-access-layer/src/packages/downloadsLast30d.ts

Comment thread services/apps/packages_worker/src/deps-dev/activities/rankPackagesUniverse.ts Outdated

Copilot AI reviewed Jun 8, 2026

View reviewed changes

chore: avoid redundant updated_at where last_synced_at/verified_at ex…

9da3538

…ists Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

mbani01 reviewed Jun 8, 2026

View reviewed changes

Comment thread backend/src/osspckgs/migrations/V1780600000__add_created_updated_at.sql Outdated

chore: skip packages_universe in created_at/updated_at migration (pen…

c4af1ac

…ding deprecation) Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

Copilot AI review requested due to automatic review settings June 8, 2026 16:42

Copilot started reviewing on behalf of joanagmaia June 8, 2026 16:42 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

cursor Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread services/libs/data-access-layer/src/packages/osv.ts

mbani01 previously approved these changes Jun 8, 2026

View reviewed changes

joanagmaia mentioned this pull request Jun 8, 2026

feat: add Tinybird datasources for packages-db tables (CM-1219) #4180

Open

Merge branch 'main' into chore/update-packages-schemas-for-tinybird

04da2d3

cursor Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread services/libs/data-access-layer/src/packages/maintainers.ts

Copilot AI review requested due to automatic review settings June 9, 2026 09:30

joanagmaia dismissed mbani01’s stale review via 80a7fd1 June 9, 2026 09:30

Copilot started reviewing on behalf of joanagmaia June 9, 2026 09:30 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

chore: fix osspckgs DAL to use email and populate created_at/updated_at

27f5aa2

Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

joanagmaia force-pushed the chore/update-packages-schemas-for-tinybird branch from 80a7fd1 to 27f5aa2 Compare June 9, 2026 10:23

joanagmaia requested a review from mbani01 June 9, 2026 10:32

chore: add updated_at to repos for non-nullable Tinybird watermark

78bc0eb

Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

Copilot AI review requested due to automatic review settings June 9, 2026 10:51

Copilot started reviewing on behalf of joanagmaia June 9, 2026 10:52 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

themarolt previously approved these changes Jun 9, 2026

View reviewed changes

fix: lint

b435ad6

Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

joanagmaia dismissed themarolt’s stale review via b435ad6 June 9, 2026 11:18

joanagmaia merged commit 661161e into main Jun 9, 2026
15 checks passed

joanagmaia deleted the chore/update-packages-schemas-for-tinybird branch June 9, 2026 11:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add created_at/updated_at to packages-db schemas for Tinybird sync#4178

chore: add created_at/updated_at to packages-db schemas for Tinybird sync#4178
joanagmaia merged 7 commits into
mainfrom
chore/update-packages-schemas-for-tinybird

joanagmaia commented Jun 8, 2026 •

edited by cursor Bot

Loading

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		ALTER TABLE packages
		ADD COLUMN IF NOT EXISTS created_at timestamptz NOT NULL DEFAULT NOW();

Conversation

joanagmaia commented Jun 8, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joanagmaia commented Jun 8, 2026 •

edited by cursor Bot

Loading