feat: MediaDB Media and Title Persistence

[BossRighteous]

Refactors MediaDB indexing to not truncate existing records.
To achieve this the ScanState is repopulated with Media and Titles on each loop.
Pulling this data from DB isn't free so indexing was refactored to handle all 3 launcher types per system.

Also added a DevErr system which should have no practical application other than making indexing verification easier between the 3 types.

So far tested as migration, fresh DB, and full/partial DB scans.
Re-found records self heal on next scan.
Rename some files and folders and get to scanning/browsing!

We should discuss purposeful truncation and orphan cleanup mechanisms and where they might live in the app/core.

Next stop: scrapers 🚂

Summary by CodeRabbit

• New Features

  • Persistent missing-media tracking that excludes missing items from searches and browse results.
  • Selective cache refresh to speed partial system indexing.
  • New "Developer Error Test" system with test launchers.

• Bug Fixes

  • Improved media title/tag reconciliation during reindexing.
  • Daemon restart now waits for the specific process to exit and is more idempotent.

• Tests

  • Extensive new/updated tests for scanning, missing-media flags, caching, indexing, and launchers.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds persistent missing-media tracking and reconciliation, a new DevErr test system and launchers, selective cache rebuilds for partial indexing, many DB/API additions for title/tag/missing-flag maintenance, and refactors the mediascanner into a persistent per-system indexing driver with accompanying tests and migrations.

Changes

Cohort / File(s)	Summary
System Definition & Asset pkg/assets/systems/DevErr.json, pkg/database/systemdefs/systemdefs.go	Add new DevErr system asset file and exported SystemDevErr constant; register it in Systems.
Platforms / Launchers pkg/platforms/shared/deverr/deverr.go, pkg/platforms/shared/deverr/deverr_test.go, pkg/platforms/shared/esde/systemmap.go, pkg/platforms/windows/platform.go	Add deverr launchers (system-specific and any-system variants), tests, map deverr folder to DevErr system, and include launchers in Windows platform launcher list.
DB Schema Migrations pkg/database/mediadb/migrations/20260424144847_add_is_missing.sql, pkg/database/mediadb/migrations/20260424190000_add_missing_runtime_indexes.sql	Add non-null Media.IsMissing (DEFAULT 0) and ensure indexes for Media.Path and Media.ParentDir.
Core DB Types & Interface pkg/database/database.go	Add Media.IsMissing, new MediaTagLink type, extend ScanState with per-system maps (MediaTitleIDs, MediaTagIDs, MissingMedia), and add multiple MediaDBI methods for title/tag reconciliation, missing-flag bulk ops, and selective cache refreshes.
MediaDB Implementation pkg/database/mediadb/mediadb.go, pkg/database/mediadb/slug_search_cache.go	Scope cache invalidation to systems, add selective slug-cache refresh/merge, and new public methods: UpdateMediaTitle, DeleteMediaTags, DeleteMediaTag, BulkSetMediaMissing, ResetMissingFlags, GetMediaTagsBySystemID.
SQL Helpers & Queries pkg/database/mediadb/sql_media.go, pkg/database/mediadb/sql_media_tags.go, pkg/database/mediadb/sql_search.go, pkg/database/mediadb/sql_browse.go, pkg/database/mediadb/sql_browse_cache.go, pkg/database/mediadb/sql_helpers.go, pkg/database/mediadb/sql_cache.go, pkg/database/mediadb/sql_tags.go, pkg/database/mediadb/sql_missing_flags_test.go	Add SQL helpers to update media title and delete media-tag links; add bulk-set/reset IsMissing; add per-system media-tag query; modify many queries to exclude missing media (IsMissing = 0); add tests for missing-flag helpers.
Slug Search Cache pkg/database/mediadb/slug_search_cache.go	Introduce partial-cache support with coverage tracking, fragment build/merge ops, CanServeSystems, and incremental RefreshSlugSearchCacheForSystems.
Batch Inserter pkg/database/mediadb/batch_inserter.go, pkg/database/mediadb/batch_inserter_test.go	Cache prepared statements keyed by exact rowCount with LRU eviction (preserve full-batch entry), adjust Flush/Close behavior, and add tests for bounded caching and eviction.
Mediascanner: Pipeline & Driver pkg/database/mediascanner/indexing_pipeline.go, pkg/database/mediascanner/mediascanner.go, pkg/database/mediascanner/indexing_pipeline_test.go, pkg/database/mediascanner/persistence_regression_test.go, pkg/database/mediascanner/mediascanner_test.go, pkg/database/mediascanner/mediascanner_bench_test.go	Refactor to persistent per-system single-pass indexing: seed/persist scan state, reconcile titles/tags in AddMediaPath, maintain per-system MissingMedia and do per-system bulk resets/sets, avoid mid-system flushes, selective cache refreshes for touched systems, and add/update tests/benchmarks.
Media-Tag Ops pkg/database/mediadb/sql_media_tags.go	Add function to delete a single media-tag link and to query all media-tag links for a given system (returns ordered MediaTagLink rows).
MediaDB Tests & Integration pkg/database/mediadb/mediadb_integration_test.go	Add integration tests asserting secondary-index presence/recreation and selective slug/tag cache behavior with tagged fixtures.
Testing Mocks & Helpers pkg/testing/helpers/db_mocks.go	Extend MockMediaDBI with UpdateMediaTitle, DeleteMediaTags/DeleteMediaTag, BulkSetMediaMissing, ResetMissingFlags, GetMediaTagsBySystemID, PopulateSystemTagsCacheForSystems, RefreshSlugSearchCacheForSystems and add default .Maybe() expectations.
Daemon Restart pkg/service/daemon/daemon.go, pkg/service/daemon/daemon_test.go	Make Restart PID-specific by waiting for the exact old PID to exit (waitForPIDExit, pidRunning) and add tests for PID-wait semantics and timeout messages.
API & Tests Adjustments pkg/api/methods/media_cancel_test.go, pkg/api/methods/methods_test.go	Update test mocks to explicitly stub Platform.Launchers(*config.Instance) and include expectations for tag/slug cache/index calls in tests.
Scan Behavior Test Sync pkg/service/scan_behavior_test.go	Add synchronization point in a test to wait for token state between game launches.

Sequence Diagram(s)

sequenceDiagram
    participant Scanner
    participant ScanState as ScanState
    participant MediaDB
    participant SQLite

    rect rgba(100,150,200,0.5)
    Note over Scanner,SQLite: Resume / Seed Phase
    Scanner->>MediaDB: PopulatePersistentScanStateForSystem(systemID)
    MediaDB->>SQLite: SELECT Media & Titles for system
    SQLite-->>MediaDB: Rows (IDs, titles, tags)
    MediaDB->>ScanState: Seed MediaTitleIDs, MediaTagIDs, MissingMedia
    end

    rect rgba(150,100,200,0.5)
    Note over Scanner,MediaDB: Per-file processing
    Scanner->>ScanState: AddMediaPath(path, title, tags)
    alt media exists
        ScanState->>ScanState: remove DBID from MissingMedia
        ScanState->>MediaDB: UpdateMediaTitle / reconcile tags (delete/insert)
        MediaDB->>SQLite: UPDATE/Delete/INSERT queries
    else new media
        Scanner->>MediaDB: INSERT Media/Title/Tags
        MediaDB->>SQLite: INSERT queries
    end
    end

    rect rgba(200,150,100,0.5)
    Note over Scanner,SQLite: Finalize missing flags
    Scanner->>MediaDB: ResetMissingFlags(systemDBID)
    MediaDB->>SQLite: UPDATE Media SET IsMissing=0 WHERE SystemDBID=?
    Scanner->>MediaDB: BulkSetMediaMissing(remainingDBIDs)
    MediaDB->>SQLite: UPDATE Media SET IsMissing=1 WHERE DBID IN (...)
    end

    rect rgba(100,200,150,0.5)
    Note over Scanner,MediaDB: Selective cache refresh
    Scanner->>MediaDB: PopulateSystemTagsCacheForSystems(systems)
    MediaDB->>SQLite: SELECT tags for touched systems
    MediaDB->>MediaDB: Merge partial tag/slug cache into global caches
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

• feat(database): filename parser, tag counts, title strategies, and scanner improvements #718 — Overlaps mediascanner refactor and media/tag reconciliation logic touched here.
• feat(database): tag taxonomy update, storage padding, and alias canonicalization #713 — Related changes in tag SQL and tag-cache building; intersects sql_tags.go / sql_cache.go.
• fix(mediascanner): fix media index hang on resume from interrupted run #705 — Related UNIQUE-constraint/dependency-flush classification changes in AddMediaPath and tests.

Poem

🐰 I hopped through indexes, tags, and seams,
Marked missing tracks in midnight dreams.
I stitched titles back where they belong,
Reconciled each tag with a cheerful song.
Now DevErrs test, and caches hum along.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 31.54% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: MediaDB Media and Title Persistence' accurately describes the main architectural change: refactoring MediaDB to persist media and titles instead of truncating records.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sentry]

Codecov Report

❌ Patch coverage is 67.90254% with 303 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/database/mediadb/mediadb.go	54.60%	62 Missing and 7 partials ⚠️
pkg/database/mediascanner/mediascanner.go	69.00%	41 Missing and 21 partials ⚠️
pkg/database/mediadb/slug_search_cache.go	75.92%	28 Missing and 24 partials ⚠️
pkg/database/mediascanner/indexing_pipeline.go	75.69%	25 Missing and 10 partials ⚠️
pkg/database/mediadb/sql_media_tags.go	0.00%	31 Missing ⚠️
pkg/database/mediadb/batch_inserter.go	56.66%	19 Missing and 7 partials ⚠️
pkg/database/mediadb/sql_media.go	61.36%	15 Missing and 2 partials ⚠️
pkg/service/daemon/daemon.go	72.72%	3 Missing and 3 partials ⚠️
pkg/platforms/shared/deverr/deverr.go	87.50%	4 Missing ⚠️
pkg/database/mediadb/sql_search.go	95.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[coderabbitai]

♻️ Duplicate comments (1)

pkg/testing/helpers/db_mocks.go (1)

1291-1305: ⚠️ Potential issue | 🟠 Major

Track missing-flag writes in the mock, too.

BulkSetMediaMissing and ResetMissingFlags mutate DB state, but they still skip trackDatabaseOperation(). That leaves OperationsOutsideTxn blind to the new missing-media write path, so transaction-safety regressions can slip through mock-based tests.

💡 Minimal fix

 func (m *MockMediaDBI) BulkSetMediaMissing(dbids map[int]struct{}) error {
+	m.trackDatabaseOperation() // Track if called outside transaction
 	args := m.Called(dbids)
 	if err := args.Error(0); err != nil {
 		return fmt.Errorf("mock operation failed: %w", err)
 	}
 	return nil
 }
 
 func (m *MockMediaDBI) ResetMissingFlags(systemDBIDs []int) error {
+	m.trackDatabaseOperation() // Track if called outside transaction
 	args := m.Called(systemDBIDs)
 	if err := args.Error(0); err != nil {
 		return fmt.Errorf("mock operation failed: %w", err)
 	}
 	return nil
 }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/testing/helpers/db_mocks.go` around lines 1291 - 1305, The mock methods
BulkSetMediaMissing and ResetMissingFlags perform DB-mutating actions but don't
call trackDatabaseOperation, so OperationsOutsideTxn can't detect these writes;
update both MockMediaDBI.BulkSetMediaMissing and MockMediaDBI.ResetMissingFlags
to invoke trackDatabaseOperation (or the mock's equivalent tracking helper) when
the method runs (before or after the m.Called check) so the missing-media write
is recorded for OperationsOutsideTxn checks.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@pkg/testing/helpers/db_mocks.go`:
- Around line 1291-1305: The mock methods BulkSetMediaMissing and
ResetMissingFlags perform DB-mutating actions but don't call
trackDatabaseOperation, so OperationsOutsideTxn can't detect these writes;
update both MockMediaDBI.BulkSetMediaMissing and MockMediaDBI.ResetMissingFlags
to invoke trackDatabaseOperation (or the mock's equivalent tracking helper) when
the method runs (before or after the m.Called check) so the missing-media write
is recorded for OperationsOutsideTxn checks.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ab6153f7-419d-45e9-b18e-9fd4f1f76976

📥 Commits

Reviewing files that changed from the base of the PR and between bc0853d and 7c8d1c5.

📒 Files selected for processing (6)

• pkg/api/methods/methods_test.go
• pkg/database/mediadb/mediadb.go
• pkg/database/mediascanner/indexing_pipeline.go
• pkg/database/mediascanner/mediascanner.go
• pkg/service/daemon/daemon_test.go
• pkg/testing/helpers/db_mocks.go

✅ Files skipped from review due to trivial changes (1)

• pkg/service/daemon/daemon_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/api/methods/methods_test.go