[Part 1] RFC: Book File Format / Issue Abstraction

[zachyale]

RFC: Book File Format / Issue Abstraction

Problem statement

Grimmory currently overloads the book_file model to represent both "this book is available in this format" and "this exact file or folder exists on disk." That works for single-file ebooks, documents, and comics, but it breaks down when a representation needs more than one file.

The clearest example today is multi-file audiobooks. A multi-file audiobook is collapsed into one book_file row with is_folder_based = true, while individual audio files are rediscovered from the filesystem whenever the system needs track-level data. That means Grimmory cannot reliably persist track order, per-file metadata, include/exclude state, per-file deletion/download state, or repeatable M4B/metadata-embedding inputs.

Audiobooks are the immediate reason to do this, but the abstraction should stay general. Grimmory may later track sidecar Readium manifest JSON, guided navigation packages, generated outputs (such as transcriptions), or other format-specific files. Those are easier to manage if Grimmory distinguishes a book's logical representation from the files that belong to it.

This RFC proposes splitting the current book_file abstraction into two layers:

• A logical format/issue layer: one owned representation of a book in a specific format or physical copy.
• A concrete file layer: one row per filesystem object associated with that representation.

This should not be a full rewrite of Grimmory's book model. The goal is to make the existing "this book has this format" concept explicit, then allow that format/issue to own multiple tracked files. This keeps the door open for a future Work / Edition model, but does not scope that broader work.

Goals

• Add a small logical model that encapsulates all files for a given type/format/issue of a given book
• Add a child file model so each filesystem file associated with that format/issue is tracked in the database
• Preserve the concept that a book can have multiple available formats
• Stop treating folder-based content as opaque rows with transient runtime files
• Provide stable DB identities for multi-file representations
• Enable follow-up features such as audiobook file/track management, chapter editing, M4B creation, and metadata embedding
• Keep current book, reader, download, progress, and metadata APIs compatible (if possible) during rollout
• Keep the model compatible with a future Work / Edition abstraction

Non-goals

• This RFC does not implement rich audiobook functionality. It lays the foundation for efficient multi-file operations and management
• This RFC does not complete the broader Work / Edition refactor
• This RFC does not redesign metadata matching, metadata editing, or reader UIs
• This RFC does not change upload/download permissions or introduce new user-visible audiobook tools

Proposed solution

Replace the current "one book_file row equals one file or folder" assumption with a model where a logical representation can own one or more concrete files.

This should be a pragmatic migration, not a deep remodel. The first pass should keep the current Book -> available formats mental model and insert one level beneath it:

Book
  Format / Issue
    Files

That means the first implementation should avoid pulling in the full Work / Edition hierarchy. It should also avoid making every concrete file look like a top-level format. A multi-file audiobook should be one audiobook format/issue with many file rows, not many audiobook formats.

The final names should be decided during implementation, but the conceptual model could look like this:

Book
  BookIssue / BookFormat / BookManifestation
    BookIssueFile / MediaFile

Examples:

A Promised Land
  AUDIOBOOK issue
    001.mp3
    002.mp3
    003.mp3

A Promised Land
  AUDIOBOOK issue
    A Promised Land.m4b

Dune
  EPUB issue
    Dune.epub

Watchmen
  CBX issue
    Watchmen.cbz

Physical copy
  PHYSICAL issue
    no filesystem files

The existing book_file table can either be renamed/reframed as the issue/format table, or replaced by a new table with a compatibility view/mapper. The important requirement is that the logical representation is no longer forced to be the same row as the concrete file. The simplest viable shape is:

• one row for the book's format/issue, such as "this book has an AUDIOBOOK representation"
• one or more child rows for the files that belong to that format/issue, such as 001.mp3, 002.mp3, and 003.mp3

What are the high level architectural changes?

• Introduce a logical issue/format abstraction under BookEntity
• Introduce a child file abstraction with one row per tracked filesystem object
• Update scan, watcher, upload, bookdrop, delete, move/organize, download, and reader services to resolve a book's issue/format first, then operate on its file rows where needed
• Keep compatibility mappers where possible for existing API shapes that expose "book files"
• Keep existing format-level behaviour wherever possible; only move data when the distinction between the format/issue and the concrete file matters
• Move concrete-file metadata to the file layer when it describes one filesystem object
• Make folder-based audiobook ingestion create one audiobook issue with many audio file rows instead of one opaque folder row

What are the high level data model changes?

Add or refactor toward a logical issue/format table with fields such as:

• id
• book_id
• book_type / format
• display_name
• is_primary
• is_physical
• added_on
• description
• format-level technical fields where appropriate, such as total duration, chapter count, codec summary, archive type, fixed-layout flag, and format-level hashes if needed

Add a concrete file table using the file-specific fields that currently live on book_file, such as path, filename, size, hashes, technical metadata, and missing/deleted state.

For the first migration, existing single-file book_file rows should become one issue/format row with one child file row. Existing folder-based audiobook rows should become one audiobook issue/format row with one child file row per audio file in the folder. That should be the primary focus; deeper metadata-specific improvements can remain follow-up work.

What are the main changes to the UI?

This RFC should mostly preserve existing UI behaviour. It should add or prepare small compatibility changes:

• Existing format/file lists should still show one entry for each available format
• Format detail views should be able to expand to show the concrete files belonging to that format
• Multi-file formats, especially audiobooks, should be able to display each tracked file without requiring a filesystem scan
• Existing actions like read/play/download/delete should remain available per file and per format

Risks

Are there any backwards-incompatible changes?

Yes, potentially, but the first pass should stay small. The current database and many APIs assume book_file is both the format and the concrete file. The rollout should preserve current API response shapes where possible and introduce new issue/file-aware APIs before removing or renaming old fields.

Special attention is needed for:

• reader route resolution
• download endpoints
• progress keyed to book_file_id
• additional-file attachment/detachment
• file deletion and cleanup
• cover and metadata writing
• OPDS/Kobo/app API payloads
• library scans and watcher events

Does this project have special implications for security and data privacy?

Yes. The new file layer will become the source of truth for filesystem paths, deletion, downloads, and file mutation. Permission checks must remain at least as strict as today, and API responses should avoid exposing more filesystem path detail than existing endpoints intentionally expose.

Could this change significantly increase load on any of our backend systems?

The migration and initial scan/backfill may increase load. After rollout, the model should reduce repeated filesystem reads for multi-file formats because Grimmory can read track/file state from the database.

Does this project have any dependencies?

This RFC should land before rich audiobook functionality. It does not depend on the full Work / Edition model, but it should use terminology and boundaries that can migrate cleanly into that model later.

Alternative solutions

One alternative is to keep book_file as-is and add an audiobook-specific audiobook_track table. That would unblock some audiobook work, but it would create a one-off model and leave the same abstraction problem for other multi-file representations.

Another alternative is to wait for the full Work / Edition migration. That may be architecturally cleaner long-term, but it blocks practical improvements like rich audiobook management. This RFC creates a smaller seam that can later become the issue/manifestation/media-object layer.

Another alternative is to keep rediscovering multi-file formats from the filesystem. That avoids a migration, but it makes ordering, missing-file handling, per-file actions, and repeatable processing unreliable.

Implementation and rollout plan

1. Audit every current book_file usage and classify whether it means logical format/issue, concrete file, or both.
2. Add the new concrete file table and model classes.
3. Reframe or introduce the logical issue/format table while keeping compatibility with existing book_file IDs where possible.
4. Migrate existing single-file records into one issue with one file.
5. Migrate existing folder-based audiobook records into one issue with many files.
6. Update scan/rescan and watcher paths to create/update issue/file rows consistently.
7. Update reader, download, delete, cover, metadata, and progress services to resolve through the new model.
8. Update API DTOs to expose issue/format-level data and, where needed, nested concrete file data.
9. Add compatibility shims for existing clients and deprecate ambiguous fields only after feature parity is restored.
10. Use the new model as the prerequisite for rich audiobook functionality.

Does this project require a migration?

Yes, but it should be a contained migration: create a format/issue record for each current book_file-style representation, then create one or more child file records under it.

Migration requirements:

• Back up or preserve existing book_file identifiers where possible.
• Create one logical issue/format per existing book_file row.
• Create one concrete file row for each existing single-file row.
• Expand folder-based audiobook rows into one concrete file row per audio file.
• Preserve hashes, file sizes, technical metadata, chapter metadata, progress associations, and primary-format ordering.
• Avoid migrating unrelated book/work/edition metadata as part of this change.
• Provide a rollback plan that can restore the previous book_file shape or keep a compatibility table/view.

Is this project in an experiment or feature flagged?

The database migration itself should not be user-toggleable, but UI and API surfaces that expose nested file details can be rolled out gradually. During rollout, Grimmory should continue to serve existing book/file responses until downstream clients and feature work have moved to the new model.

Success Criteria

How will you validate the solution is working correctly?

• Existing single-file EPUB, PDF, CBX, M4B, M4A, MP3, and OPUS books continue to scan, display, read/play, download, and delete correctly.
• Existing folder-based audiobooks migrate into one audiobook issue with one tracked file row per audio file.
• Multi-file audiobook track order is stable across scans, rescans, watcher events, restarts, and playback.
• Existing progress, bookmarks, reading sessions, metadata, covers, OPDS/Kobo/app payloads, and download flows remain compatible.
• Deleting or moving a book/format updates all related concrete file rows correctly.
• Scanner and watcher ingestion produce the same model for the same filesystem layout.
• Rich audiobook functionality can consume tracked file rows without doing ad hoc folder scans.

What monitoring and alerting will you do to ensure this project doesn’t decrease performance and reliability?

• Log migration counts for created issue/format rows and concrete file rows.
• Log folder-based audiobook expansions, including skipped unreadable files.
• Monitor scan/rescan duration before and after the migration.
• Surface missing-file and hash-mismatch states clearly.
• Track errors in reader, download, delete, and metadata-writing paths after rollout.

[imnotjames]

A physical file layer: one row per filesystem object associated with that representation.

I would suggest the term "concrete" rather than "physical" to avoid conflation.

[zachyale]

Done

[imnotjames]

The first pass should keep the current Book -> available formats mental model and insert one level beneath it:

Should I take this literally? Would we be renaming the existing table to something else?

Add a physical file table using the file-specific fields that currently live on book_file, such as path, filename, size, hashes, technical metadata, and missing/deleted state.

What would happen to those fields on the existing table?

Do you have suggested final schemas for these tables?

[imnotjames]

Existing folder-based audiobook rows should become one audiobook issue/format row with one child file row per audio file in the folder. That should be the primary focus; deeper metadata-specific improvements can remain follow-up work.

So this would need to be a task that migrates somehow? Or how else would we do this? Eventual consistency?

[zachyale]

Correct, it would be a app migration startup task. This structure currently exists within backend/src/main/java/org/booklore/service/migration/AppMigrationStartup.java- we would introduce a new migration task to manually re-scan all existing folder based audiobooks after the initial DB migration has occurred.

[imnotjames]

The database migration itself should not be user-toggleable, but UI and API surfaces that expose nested file details can be rolled out gradually.

How would this gradual rollout work? Do you mean that we'd implement a new feature flagging solution? Or we'd keep the APIs following the same responses?

[zachyale]

I simply mean we can continue to keep the APIs following the same response structure for compatibility until we're ready to deprecate them fully, since the current book_files is effectively analogous to the proposed book_issues

[imnotjames]

Does this have any interactions with the "folder based" settings for a library? Or is that only for the semantics of scanning rather than how we need to store book files?

[piffio]

This keeps the door open for a future Work / Edition model,

This Work / Edition model is mentioned a few times, but there is no link to any relevant discussion / RFC. Can we add it so that people without familiarity with it can have the full picture? Thanks!

[imnotjames]

Perhaps https://github.com/orgs/grimmory-tools/discussions/1332

[piffio]

OK, so that is Part 2 of the sequence. My bad, I'm just reading the first one now, but still worth it to make this explicit in the text as I found it confusing and kept wondering if the description of that model would appear later in the RFC

[piffio]

Add or refactor toward a logical issue/format table with fields such as:

• id
• book_id
• book_type / format
• display_name
• is_primary
• is_physical
• added_on
• description
• format-level technical fields where appropriate, such as total duration, chapter count, codec summary, archive type, fixed-layout flag, and format-level hashes if needed

I'd strongly recommend for all the main refactoring RFCs that will touch the data model to include detailed ER diagrams (Mermaid is a good option) with comprehensive coverage of all the fields / attributes and clarifying the nature of relationships (1-to-1, 1-to-many, etc).

It makes it so much easier to reason about them and to ensure all details are properly considered compared to a text description, which will always have some gaps or things "taken for granted".

[piffio]

• OPDS/Kobo/app API payloads

As a side note, this is quite important: I noticed the other day that physical books appear in the OPDS API/catalogue, but of course they can't be downloaded, therefore causing confusion.

I wonder if we should consider adding a layer (likely at the application level) to map characteristics per format. e.g. physical books cannot be read / downloaded / streamed, audiobooks cannot be read, etc.

Caveat: it's been a while since I've looked at the code, so something might be there already, but if not, this refactor could consider making that logic more robust than it is today