Skip to content

fix(data quality, db): migration to rename old data_quality_stats_table and create with new data_quality_stats table BED-8616#2946

Merged
sirisjo merged 5 commits into
mainfrom
BED-8616
Jul 1, 2026
Merged

fix(data quality, db): migration to rename old data_quality_stats_table and create with new data_quality_stats table BED-8616#2946
sirisjo merged 5 commits into
mainfrom
BED-8616

Conversation

@sirisjo

@sirisjo sirisjo commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Description

Upon adding this table in a previous release, we discovered that there exists an old data_quality_stats table in the test and production environments from a long time ago that is not tracked in any of our migration files. When running the migration in the test environment, the new table was not created because the data_quality_stats table already existed. We need to drop that old table first, and then we can recreate it with the new columns that we want.

Motivation and Context

Resolves BED-8616

Why is this change required? What problem does it solve?

How Has This Been Tested?

Ran the migration locally.

Screenshots (optional):

Types of changes

  • Database Migrations

Checklist:

Summary by CodeRabbit

  • Bug Fixes
    • Updated how data quality statistics are stored, including richer metric metadata and improved timestamp tracking to preserve metric history more reliably.
    • Improved migration safety by recreating the statistics storage cleanly and restoring the previous state if changes are rolled back.
    • Added support for an optional related “kind” reference and ensured efficient access to recent entries via a timestamp-based index.

@sirisjo sirisjo self-assigned this Jun 30, 2026
@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 81606c0d-538c-445b-aaca-395c2f11e342

📥 Commits

Reviewing files that changed from the base of the PR and between 37e5e98 and 6130de7.

📒 Files selected for processing (1)
  • cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql
🚧 Files skipped from review as they are similar to previous changes (1)
  • cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql

📝 Walkthrough

Walkthrough

Adds a Goose SQL migration that renames and recreates data_quality_stats with updated schema and index changes, plus a Down step that restores the previous table names.

Changes

Data Quality Stats Table Migration

Layer / File(s) Summary
Rename existing artifacts
cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql
Renames any existing data_quality_stats table, its id sequence, and related indexes to _old names in the Up step.
Create updated table
cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql
Creates data_quality_stats with the updated columns, defaulted metric_value, optional kind_id foreign key, timestamp fields, and a created_at index.
Restore prior names on Down
cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql
Drops data_quality_stats in Down and renames data_quality_stats_old back to the original table, sequence, and index names.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is specific and clearly matches the migration-focused change, including the related ticket.
Description check ✅ Passed The description follows the template and includes the change summary, motivation, testing, issue, and checklist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch BED-8616

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sirisjo sirisjo changed the title (feat, database): migration to delete old data_quality_stats_table and replace with new data_quality_stats table BED-8616 (fix): migration to delete old data_quality_stats_table and replace with new data_quality_stats table BED-8616 Jun 30, 2026
@sirisjo sirisjo changed the title (fix): migration to delete old data_quality_stats_table and replace with new data_quality_stats table BED-8616 fix(data quality, db): migration to delete old data_quality_stats_table and replace with new data_quality_stats table BED-8616 Jun 30, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats_BED-8616.sql`:
- Around line 1-40: The migration filename does not follow the required
snake_case convention because it uses an uppercase hyphenated ticket suffix.
Rename the migration to match the existing pattern used by the v9
data_quality_stats migrations, keeping the timestamp and version prefix intact
while changing the descriptive name to snake_case (for example, use a lowercase
ticket suffix like in 20260622132700_v9_add_data_quality_stats_bed_8616.sql).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 69007cf9-e4ed-4d12-944a-3bc9606b66bc

📥 Commits

Reviewing files that changed from the base of the PR and between f0a35b2 and 560eb28.

📒 Files selected for processing (1)
  • cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats_BED-8616.sql

-- +goose Up

-- Drop the old data_quality_stats table that is not currently used anywhere but still exists in prod
DROP TABLE IF EXISTS data_quality_stats;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we instead rename/archive the table instead of deleting it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed to data_quality_stats_old 🫡

@sirisjo sirisjo requested a review from elikmiller June 30, 2026 22:18
@sirisjo sirisjo changed the title fix(data quality, db): migration to delete old data_quality_stats_table and replace with new data_quality_stats table BED-8616 fix(data quality, db): migration to rename old data_quality_stats_table and create with new data_quality_stats table BED-8616 Jun 30, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql`:
- Around line 19-20: The migration currently renames data_quality_stats but
leaves the existing sequence and primary-key index behind on the old table,
which can conflict with the new table creation. Update the migration around the
ALTER TABLE rename and the subsequent CREATE TABLE for data_quality_stats so the
old sequence and PK index are also renamed/dropped, or define explicit names for
the new table’s serial sequence and primary key index. Use the existing
migration block and the new data_quality_stats table definition as the anchor
points when making the change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: e70308de-ef98-43dc-be9d-64c71e5da544

📥 Commits

Reviewing files that changed from the base of the PR and between a0297a1 and 37e5e98.

📒 Files selected for processing (1)
  • cmd/api/src/database/migration/migrations/20260630135900_v9_drop_and_recreate_data_quality_stats.sql

@mvlipka mvlipka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the indexes are correct and that we don't have an associated aggregation table in test
Great catch!

@sirisjo sirisjo merged commit aa6e331 into main Jul 1, 2026
13 checks passed
@sirisjo sirisjo deleted the BED-8616 branch July 1, 2026 16:45
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants