2.4.0

The biggest architectural refactor since the v2 rewrite, a production-ready Gunicorn HTTP server, file verification and compression, TitleDB becomes a first-class citizen, library-watching improvements, uv and pipsupport, and Windows compatibility.

Backend separation of concerns

There was a significant design flaw in the backend architecture since the v2 rewrite: all files and database read/write operations were executing in the same process as the Flask server, which handles the incoming and outgoing HTTP requests.
This prevented longer, process heavy tasks from being implemented, as their execution would have been tied to the main Flask server and API, impacting their performance and stability.
For example, each Flask app instance spawned a watchdog thread for monitoring library changes, then at least one other thread to handle library changes, file identification and changes in the database. So the HTTP server could never be run in parallel to handle more requests concurrently. We were stuck having to use a Flask development server, in a single process, adding hacks on top of hacks to handle background tasks, and it really showed its limits when several users are using your shop.

This version introduces separation of concerns between different components of Ownfoil, each being independent, responsible and optimized for its role: the HTTP/API service, and the Worker service.

1. The HTTP/API service: production ready Gunicorn server, multiple threads handling concurrent requests, enqueuing background tasks as needed.
2. The Worker service: a pool of background worker processes executing the enqueued tasks, processing all database and heavy operations in parallel.

Unlike before, where background/scheduled tasks were wrapped with thread locks and debounce functions within the HTTP server process, the architecture is now clean and allows much more performance and stability. Ownfoil is able to handle a lot more traffic and also execute heavy tasks concurrently in the background. The Web UI should already feel snappier, library management significantly faster for large libraries, and this will allow traffic heavy features that would not be possible if still using the Flask dev server.

The workers dynamically scale, immediately after their configured count changes. You can scale them up when adding a lot of files, then size them down when done. The current and pending Tasks are visible in the Web UI: you can see the live progress of the current tasks, all the pending tasks waiting for an available worker, and the list of workers with their ongoing executions. This can be used to scale the workers pool accordingly, and see the scheduled jobs.

Automatic file verification

One aspect of library curation is to know whether or not the files in your collection are valid. This means different things for different users: some want to ensure every file is original and intact, while for others a modified but working file is enough. File verification is a tedious chore, but a mandatory one to make sure your backups will install and work, or if they need to be replaced.

Ownfoil will now automatically verify every supported file, once it has been successfully decrypted and identified, in a background task. Using signature and content hash validation, each file will be classified as either:

• valid: original content from Nintendo, intact.
• repack: content has been modified, but metadata has been updated to allow integrity verification and it matches.
• modified: content has been modified but integrity metadata has not been updated.
• corrupt: the integrity of the content cannot be verified.
• signature ok / signature failed: set for a signature-only verification, when hash verification has not been run.
• unverified: never checked, because verification is off, the keys are missing, or the file has not been processed yet.

A corrupt file should be deleted or replaced, as installation will fail, but anything else should be fine to keep.

Automatic file compression

A collection of backups can amount to a lot of storage space taken. Compressing them using the lossless nsz format helps reducing their impact on disk and has other significant advantages like smaller transfer size and faster install speeds. It is completely transparent to the clients when installing, so taking the time to compress your collection is well worth it.

Ownfoil now automatically compresses every supported file (both nsp and xci) in a background task, if enabled. All compression settings can be configured in the Settings UI: compression level, mode (auto, which follows nsz's own choice, or forced solid or block), long distance mode, block size, threads. Before deleting the uncompressed source file, every compressed content is decompressed back and hashed against the source's content, to ensure compression was lossless. Files that verification found corrupt are skipped rather than compressed. Compressing files is a rather long task, you can follow its progress in the Tasks page.

Compression is I/O and CPU heavy, so there is a parallelism limit that is configurable separately from the worker count, to avoid saturating your disks.

TitleDB as a first-class citizen

Until now the TitleDB (metadata about the whole Switch catalog, used to identify files and display Title's info) was just downloaded as raw json files and loaded into memory. That caused several issues: those json files were the only source of data to identify contents in a file, and the whole catalog had to be loaded into memory to look up the Title IDs. This made the identification process slow by reading the entire raw files every time, and very fragile to concurrent tasks: TitleDB would use a lot of RAM so it needed to be unloaded from the process memory when not needed. I think you can see how that made parallel identification tasks in different processes unnecessarily complicated.

Ownfoil now manages its own, locally stored, TitleDB reference in a separate database, meaning a Title ID lookup is just an instantaneous database query. This makes an identification task significantly faster, without using a large memory allocation, and more importantly independent, meaning identification tasks can be run in parallel by any number of workers concurrently. If you have a large library, increasing the amount of workers will nicely scale when identifying all your files.

There is also now an override mechanism: any field from the upstream TitleDB reference can be superseded, by multiple sources of metadata given their priority. It currently supports user entry > extract from file > original TitleDB, merged field by field. The user facing interface for registering custom entries and the automatic value extraction from files will be implemented in the next version. Release dates are now recorded for base games, updates and DLC.

Last but not least, a GraphQL API has been implemented: this allows accessing any data present in the databases (Files and TitleDB) efficiently by customizing the request to retrieve only the needed fields. Previously, a library cache was generated after every library change, and the Web UI would just retrieve and load this file to show the library titles. The Web UI has been ported to use GraphQL: Titles data is built live, on request, to retrieve the actual data as indexed by Ownfoil, paginated and with browser cache support. The library.json cache file is gone, along with all the issues that came with it: debounced functions in threads, out of date snapshots, race conditions. The UI can no longer be inconsistent with regards to your collection, a newly scanned or edited game appears as soon as it is in the database, and the library page loads page by page so it opens quickly even with a large collection. Any list can be filtered and sorted by whichever field you choose, in either direction.

This GraphQL API is the backend that will support all admin operations on Titles in the Web UI, and also efficiently serving your collection. You can explore the possibilities by running Ownfoil and accessing the GraphQL playground available at /api/graphql.

Faster and more reliable library watching

The file watcher now picks a native filesystem observer when it can, and falls back to polling only where it has to, with automatic detection of the platform and of whether the watched path is local or on a network share. Native watching means library changes are picked up immediately, without the CPU cost of polling a large tree. Directory removals, Windows folder deletes and the events Ownfoil itself causes (an organized rename, a compression's partial output) are handled explicitly, so the watcher does not react to its own work.

Add uv support for installing Ownfoil, Windows compatibility

With this release pip and uv packaging support has been implemented. This means Ownfoil can now be run with the single command uvx ownfoil and you are good to go, or installed persistently with uv tool install ownfoil. This should make it easier for people to serve their library on Windows, see the updated Install doc for details.

While Ownfoil has been primarily developed to work on a Linux server, a lot of compatibility changes and fixes have been implemented to bring feature parity between the systems. On Windows the first run creates an ownfoil.bat next to your config/ and data/ directories so you can start it again by double-clicking, the Web UI opens in your browser once the server is ready, and the server's LAN IP is shown in the Setup page so you know what address to point your console at.

Rewrite Install and Usage documentation

All of Ownfoil documentation lived in the repo's Readme file. This is now just a landing page, leading you to more detailed Installation and Usage pages. The installation doc has been rewritten to add every supported method, including the new uv packaging installation, along with details on how to upgrade and select an available version to run. The usage doc starts with a First steps tutorial on how to get running, explains all features and their configuration, and then all available Settings reference.

Previously the most recommended way of using Ownfoil was through Docker, and this can have a high barrier to entry. With the simplicity and broader compatibility of uv, along with a more complete documentation, the barrier to entry is hopefully much lower.

Issues fixed

• #329 Slashes (/) in {titleName} create unintended subdirectories during organization - forbidden characters are now replaced with their full-width equivalent.
• #327 OwnFoil does not check for title updates - upstream updates have resumed, and a cache error that was preventing metadata update has been fixed.
• #326 FR: file validation - implemented as automatic file verification.
• #325 Sphaira fails to detect when running behind cloudflare - client detection no longer depends on the reverse proxy, and follows Sphaira's headers since v1.0.6.
• #323 PollingObserver interval should be configurable - the watcher now uses a native observer where possible, and the polling interval is configurable per path.
• #322 SQLite WAL mode needed for multi-process task queue - enabled, along with write serialization between workers.
• #319 Organizer breaks Sphaira file serving - the folder column no longer loses the library root when a file is moved.
• #311 Flask server crashes during TitleDB loading on ARM (Raspberry Pi) - TitleDB is no longer loaded in memory at all.
• #310 Game(s) with very long names causing problems with organization feature - Windows path length limits are now enforced.
• #303 database is locked error - fixed by WAL mode and serialized writes in the task queue.
• #297 Windows library path cannot be deleted or scanned from the UI.
• #272 [REQUEST] Windows friendly releases? - uvx ownfoil now runs Ownfoil on Windows without Docker or a Python environment to manage.
• #253 Feature Suggestion: Make Library Generation Async - the library cache is gone, the UI reads the database live through GraphQL.
• #184 process name change? - all Ownfoil processes now contain ownfoil in their name.

Other fixes worth mentioning:

• the organizer no longer renames or duplicates files it has already organized, and empty folders are cleaned up after a file is moved.
• removing a multi-content file now removes all of its associated apps, and removing a library no longer leaves rows behind.
• file downloads are counted once per client, and only when the file is actually served.
• an unknown file id returns a 404 instead of crashing, and Range requests support proper seeking.
• CNMT identification failures now report the actual error instead of a generic message.
• settings.yaml is written atomically, fixing crashes when a read happened during a write.

What's next

The next release will focus on the UI and Titles identification. With the solid background-task mechanism and an API that can efficiently interact with the databases, creating pages to view and edit title data and manage files will come naturally.

To prevent a halt in TitleDB updates from causing inconsistencies in libraries, metadata will also be extracted directly from files, ensuring that titles are identified and viewable even when they are missing from the upstream TitleDB.