Skip to content

refactor: drop legacy metrics database schema#442

Merged
silent-cipher merged 3 commits intoFilOzone:refactor/cleanup-metrics-jobsfrom
Chaitu-Tatipamula:refactor/drop-metrics-schema
Apr 17, 2026
Merged

refactor: drop legacy metrics database schema#442
silent-cipher merged 3 commits intoFilOzone:refactor/cleanup-metrics-jobsfrom
Chaitu-Tatipamula:refactor/drop-metrics-schema

Conversation

@Chaitu-Tatipamula
Copy link
Copy Markdown
Collaborator

Summary

Adds a database migration to drop the legacy metrics schema objects and removes the corresponding TypeORM entity files and module registrations. This completes step 9 of #275.

Depends on #441.

Changes

[NEW] Migration 1762000000000-DropMetricsSchema

Drops the following database objects in dependency order:

  1. refresh_sp_performance_last_week() and refresh_sp_performance_all_time() functions
  2. sp_performance_last_week and sp_performance_all_time materialized views
  3. metrics_daily table (with CASCADE)
  4. metrics_daily_metric_type_enum and metrics_daily_service_type_enum types

⚠️ This migration is irreversible. Restore from backup if rollback is needed.

Deleted entity files

  • metrics-daily.entity.ts
  • sp-performance-all-time.entity.ts
  • sp-performance-last-week.entity.ts

Modified

  • database.module.ts — Removed MetricsDaily, SpPerformanceAllTime, SpPerformanceLastWeek from entity registrations, forFeature, providers, and exports
  • types.ts — Removed MetricType enum (only consumer was the deleted entity)

Verification

  • pnpm typecheck
  • pnpm test ✅ (249 tests)
  • pnpm check:ci

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Removes the legacy “metrics” persistence layer by dropping the underlying Postgres objects and deleting the corresponding NestJS/TypeORM code and configuration that referenced them, completing step 9 of #275.

Changes:

  • Added a TypeORM migration to drop legacy metrics functions, materialized views, table, and enum types.
  • Removed legacy metrics module/services/controllers/DTOs/utils and TypeORM entities; unregistered them from the DB/app modules.
  • Removed metrics-related pg-boss queues/scheduling/config (METRICS_PER_HOUR) and updated related docs/manifests.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
kustomize/overlays/local/backend-configmap-local.yaml Removes METRICS_PER_HOUR from local overlay config.
docs/jobs.md Updates “critical env vars” list to drop METRICS_PER_HOUR.
docs/infra.md Removes metrics worker/env references from infra guidance.
docs/environment-variables.md Removes METRICS_PER_HOUR from jobs env var docs.
apps/backend/src/metrics/utils/time-window-parser.ts Deletes legacy metrics time-window parsing utility.
apps/backend/src/metrics/utils/time-window-parser.test.ts Deletes tests for the removed metrics utility.
apps/backend/src/metrics/services/network-stats.service.ts Deletes legacy metrics network stats service.
apps/backend/src/metrics/services/metrics-scheduler.service.ts Deletes legacy metrics aggregation/refresh service.
apps/backend/src/metrics/services/failed-retrievals.service.ts Deletes legacy failed retrievals metrics service.
apps/backend/src/metrics/services/failed-deals.service.ts Deletes legacy failed deals metrics service.
apps/backend/src/metrics/services/daily-metrics.service.ts Deletes legacy daily metrics query service.
apps/backend/src/metrics/metrics.module.ts Deletes legacy metrics HTTP module wiring.
apps/backend/src/metrics/metrics-worker.module.ts Deletes legacy metrics worker-only module wiring.
apps/backend/src/metrics/dto/provider-performance.dto.ts Deletes legacy provider performance DTOs.
apps/backend/src/metrics/dto/network-stats.dto.ts Deletes legacy network stats DTOs.
apps/backend/src/metrics/dto/failed-retrievals.dto.ts Deletes legacy failed retrievals DTOs.
apps/backend/src/metrics/dto/failed-deals.dto.ts Deletes legacy failed deals DTOs.
apps/backend/src/metrics/dto/daily-metrics.dto.ts Deletes legacy daily metrics DTOs.
apps/backend/src/metrics/controllers/network-stats.controller.ts Deletes legacy metrics network controller endpoints.
apps/backend/src/metrics/controllers/failed-retrievals.controller.ts Deletes legacy metrics failed-retrievals endpoints.
apps/backend/src/metrics/controllers/failed-deals.controller.ts Deletes legacy metrics failed-deals endpoints.
apps/backend/src/metrics/controllers/daily-metrics.controller.ts Deletes legacy metrics daily endpoints.
apps/backend/src/jobs/repositories/job-schedule.repository.ts Removes metrics queue mapping from pg-boss job state aggregation queries.
apps/backend/src/jobs/jobs.service.ts Removes metrics queue workers/schedules and adds legacy schedule skipping.
apps/backend/src/jobs/jobs.service.spec.ts Updates tests to reflect removal of metrics queues and constructor deps.
apps/backend/src/jobs/jobs.module.ts Removes metrics module imports/wiring from JobsModule.
apps/backend/src/jobs/job-queues.ts Removes metrics.run and metrics.cleanup queue constants.
apps/backend/src/database/types.ts Removes legacy MetricType enum (no longer needed).
apps/backend/src/database/migrations/1762000000000-DropMetricsSchema.ts Adds migration that drops legacy metrics DB objects.
apps/backend/src/database/entities/sp-performance-last-week.entity.ts Deletes legacy materialized-view entity.
apps/backend/src/database/entities/sp-performance-all-time.entity.ts Deletes legacy materialized-view entity.
apps/backend/src/database/entities/metrics-daily.entity.ts Deletes legacy metrics_daily entity.
apps/backend/src/database/entities/job-schedule-state.entity.ts Keeps legacy job types in the JobType union for existing DB rows.
apps/backend/src/database/database.module.ts Removes deleted metrics entities from TypeORM registrations/exports.
apps/backend/src/config/app.config.ts Removes METRICS_PER_HOUR validation/config loading.
apps/backend/src/app.module.ts Removes MetricsModule from app wiring.
apps/backend/README.md Removes METRICS_PER_HOUR from backend env var documentation.
apps/backend/.env.example Removes METRICS_PER_HOUR from example env file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apps/backend/src/jobs/jobs.service.ts
Comment thread docs/jobs.md
Comment thread docs/environment-variables.md
@BigLep BigLep moved this from 📌 Triage to 🔎 Awaiting review in FOC Apr 12, 2026
@BigLep BigLep requested a review from silent-cipher April 12, 2026 21:00
@Chaitu-Tatipamula Chaitu-Tatipamula force-pushed the refactor/drop-metrics-schema branch from bf9511d to 83261fd Compare April 14, 2026 06:23
Drop metrics_daily table, sp_performance materialized views,
their refresh functions, and related enums. Remove corresponding
entity files and database module registrations.
@Chaitu-Tatipamula Chaitu-Tatipamula force-pushed the refactor/drop-metrics-schema branch from 83261fd to b8d06a9 Compare April 14, 2026 07:20
Copy link
Copy Markdown
Collaborator

@silent-cipher silent-cipher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update the e2e migration tests to include the new migration DropMetricsSchema1762000000000?

@github-project-automation github-project-automation Bot moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Apr 14, 2026
@Chaitu-Tatipamula Chaitu-Tatipamula force-pushed the refactor/drop-metrics-schema branch 5 times, most recently from d85dbb3 to cb125b5 Compare April 15, 2026 19:37
@Chaitu-Tatipamula
Copy link
Copy Markdown
Collaborator Author

Could you update the e2e migration tests to include the new migration DropMetricsSchema1762000000000?

so i was looking into this and there is a subtle issue with including DropMetricsSchema in the existing e2e roundtrip test.
the problem is DropMetricsSchema is irreversible. the existing roundtrip test undoes 7 migrations from the end to get back to before RemoveSpReceivedRetrieveRequest, then re-applies them. But several of those undone migrations (RemoveIpniRetrievedColumns, RemoveSpReceivedRetrieveRequest) have down() methods that ALTER TABLE metrics_daily, which no longer exists after DropMetricsSchema irreversibly drops it.
maybe we can use isolated databases or at worst adding metrics_daily existence guards to every migration that references it wdyt?

@Chaitu-Tatipamula Chaitu-Tatipamula force-pushed the refactor/drop-metrics-schema branch from cb125b5 to 31b6167 Compare April 15, 2026 20:03
@silent-cipher silent-cipher changed the base branch from main to refactor/cleanup-metrics-jobs April 17, 2026 18:56
@silent-cipher silent-cipher merged commit 492a785 into FilOzone:refactor/cleanup-metrics-jobs Apr 17, 2026
7 of 8 checks passed
@github-project-automation github-project-automation Bot moved this from ⌨️ In Progress to 🎉 Done in FOC Apr 17, 2026
silent-cipher added a commit that referenced this pull request Apr 21, 2026
* refactor: remove metrics job

* chore: address copilot comments

* chore: remove metrics job references from docs and tests

* chore: address copilot comments

* chore: add e2e migration test

* fix: correct migrations table name

* refactor: drop legacy metrics database schema (#442)

* refactor: remove legacy metrics API endpoints and module wiring

* refactor: add migration to drop legacy metrics schema

Drop metrics_daily table, sp_performance materialized views,
their refresh functions, and related enums. Remove corresponding
entity files and database module registrations.

* test: add e2e test for DropMetricsSchema migration

* fix: test e2e migrations end state

* chore: address pr review

* chore: address pr comments

---------

Co-authored-by: durga chaitanya <107246959+Chaitu-Tatipamula@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🎉 Done

Development

Successfully merging this pull request may close these issues.

5 participants