Skip to content

Data Storage

Michael Riddle edited this page Aug 19, 2026 · 11 revisions

Data Storage

KHost Cue is local/offline-only: every list, cache, and setting lives in the app's private on-device storage behind a small set of interfaces. This page documents the storage pattern, the exact files on disk, and how to add a new persisted type safely.

The local-store pattern

Every JSON-backed store derives from JsonFileStore<TItem, TCache> (src/KHost.Mobile.Infrastructure/Services/JsonFileStore.cs), which owns:

  • The Gate (a SemaphoreSlim(1, 1)) that serializes every public method.
  • The in-memory cache, and the rule that LoadAsync/SaveAsync assume the caller already holds the gate.
  • The crash-safe write (.tmp sibling + atomic rename, via the injected IAtomicFileWriter) and the corrupt-file quarantine — a JsonException moves the bad bytes to a .corrupt sibling and the store starts empty rather than crashing.
  • The Changed event, raised through RaiseChanged() after the gate is released and only when something actually changed.

The store classes themselves (JsonFileSongListStore, JsonFileTonightStore, JsonFileSingerStore, JsonFileVenueStore, JsonFileLyricsCache, AtomicFileWriter, SingerFileNames) are internal sealed — KHost.Mobile.Infrastructure.csproj grants <InternalsVisibleTo> to both KHost.Mobile.UnitTests and KHost.Mobile.IntegrationTests, which is how the integration suite drives them directly while every other project sees only the I* interface in Abstractions.

A subclass supplies which file, which serializer context, and a label for log lines. Most stores cache the stored list as-is and use the one-parameter JsonFileStore<T>:

internal sealed class JsonFileFooStore : JsonFileStore<Foo>, IFooStore
{
    private readonly string _filePath;

    public JsonFileFooStore(IAppDataDirectory paths, ILogger<JsonFileFooStore>? logger = null, IAtomicFileWriter? writer = null)
        : base(logger ?? NullLogger<JsonFileFooStore>.Instance, writer)
        => _filePath = Path.Combine(paths.AppDataDirectory, "foo.json");

    protected override JsonTypeInfo<List<Foo>> TypeInfo => FooJsonContext.Default.ListFoo;
    protected override string Label => "Foos";
    protected override string PathFor(Guid? key) => _filePath;

    public async Task<IReadOnlyList<Foo>> GetAllAsync()
    {
        await Gate.WaitAsync();
        try { return await LoadAsync(); }
        finally { Gate.Release(); }
    }

    public async Task AddAsync(Foo item)
    {
        await Gate.WaitAsync();
        try
        {
            var items = await LoadAsync();
            items.Add(item);
            await SaveAsync(items);
        }
        finally { Gate.Release(); }

        RaiseChanged();   // after release, and a real change happened
    }
}

Three optional overrides cover the stores that differ:

Override Used by Why
CurrentKey JsonFileSongListStore, JsonFileTonightStore Returns the active singer, so the cache invalidates on a switch and the file path follows. See Per-singer namespacing.
Project / Flatten JsonFileLyricsCache Its in-memory shape is a Dictionary keyed by title+artist while the file is a flat array — the only store whose cache isn't the list. It derives from the two-parameter JsonFileStore<TItem, TCache>.

Adding is never an overwrite. JsonFileSingerStore and JsonFileVenueStore both take an id from outside (an imported profile or venues file carries the ids it was exported with), and both re-key a newcomer whose id is already in use. Two rows sharing an id would be indistinguishable to UpdateAsync, which edits the first, and to RemoveAsync, which deletes every match. A non-colliding id is kept, which is what lets a venue import keep performances resolving to the venue they were tagged at.

The per-singer write rule is enforced by the base: LoadAsync records the key it read under, and SaveAsync writes to that key's path — never a freshly re-read active-singer id. A singer switch landing between a load and its write cannot put one singer's songs into another singer's file.

Plus the source-gen context (see Serialization) and a services.AddSingleton<IFooStore, JsonFileFooStore>(); line in AddKHostInfrastructure (src/KHost.Mobile.Infrastructure/Project.cs) — the registration has to live there, not in the head's MauiProgram.cs, because the store classes are internal and only their own project can name them.

One DI wrinkle: JsonFileSongListStore is registered twice in AddKHostInfrastructure — once as itself, once as ISongListStore resolving to the same instance (sp.GetRequiredService<JsonFileSongListStore>()) — so callers that need the concrete type (the profile export/import path) and callers going through the interface share one singleton rather than getting two independent caches.

File inventory

Everything below lives under the app's private data directory (see On-device paths) unless noted otherwise.

File Holds Store Serializer context
song-list-{id}.json (per singer) / song-list.json (no singer active) List<SongListItem> — the wishlist JsonFileSongListStore SongListJsonContext
tonight-{id}.json (per singer) / tonight.json (no singer active) List<TonightEntry> — tonight's on-deck set JsonFileTonightStore TonightJsonContext
singers.json List<Singer> — the shared roster JsonFileSingerStore SingerJsonContext
venues.json List<Venue> — the shared venue list JsonFileVenueStore VenueJsonContext
lyrics-cache.json List<LyricsCacheEntry> (in memory: Dictionary<string, LyricsCacheEntry> keyed by trimmed, lowercased title+artist) JsonFileLyricsCache LyricsCacheJsonContext
album-art/<SHA-256 hex of source URL> raw cover-image bytes, one file per unique artwork URL (no extension) AlbumArtCache — binary blob, not JSON
<file>.tmp in-flight write; renamed over the target on success (see Atomic writes) AtomicFileWriter.WriteAsync n/a
<file>.corrupt a file that failed to parse, moved aside instead of being silently overwritten AtomicFileWriter.Quarantine n/a

Settings are not in this table — they use MAUI Preferences, a separate mechanism (see Settings storage).

Import/Export (src/KHost.Mobile.UI/Components/Pages/ImportExport.razor, src/KHost.Mobile.Infrastructure/Services/SingerProfileCodec.cs) writes/reads JSON too, but to a user-chosen location via the OS share/save picker, not into AppDataDirectory:

Export / import Shape Codec method Serializer context
Singer profile SingerProfile — one Singer + their full List<SongListItem> (including sung history) SingerProfileCodec.Serialize/ParseProfile ProfileJsonContext
Venues export bare List<Venue> SerializeVenues/ParseVenues VenueJsonContext

SingerProfileCodec.Detect(json) classifies a picked profile-import file by its JSON root — an object carrying a Singer property is ProfileFileKind.Profile; anything else (or invalid JSON) is ProfileFileKind.Invalid — so the import page routes to ParseProfile without the caller re-parsing. ProfileFileKind (src/KHost.Mobile.Abstractions/Models/ProfileFileKind.cs) is a standalone enum, alongside every other persisted model.

SingerProfile excludes the Tonight set (on-the-night, device-local), Venues (shared, exported separately) and Settings — a profile import/export moves identity + wishlist only.

Per-singer namespacing

The song-list and Tonight stores are namespaced per singer through the injected ISingerFileNames (src/KHost.Mobile.Abstractions/Services/ISingerFileNames.cs, implemented by internal sealed class SingerFileNames in src/KHost.Mobile.Infrastructure/Services/SingerFileNames.cs):

string SongList(Guid singerId) => $"song-list-{singerId:N}.json";
string Tonight(Guid singerId) => $"tonight-{singerId:N}.json";

{singerId:N} is the dash-less GUID format ("N"), e.g. song-list-3fa85f6457174562b3fc2c963f66afa6.json. JsonFileSongListStore, JsonFileTonightStore and JsonFileSingerStore (which deletes a removed singer's files) all take an optional ISingerFileNames? names = null constructor parameter, falling back to new SingerFileNames() — the same optional-DI-parameter shape as the logger and the clock.

Both stores take an optional IAppSession? session in their constructor:

  • With a session wired in (the real app), LoadAsync reads session.ActiveSingerId and resolves the singer's own file via PathFor(singerId). It caches _loadedFor alongside the in-memory list; a mismatch between _loadedFor and the current ActiveSingerId invalidates the cache and reloads. SaveAsync writes to _loadedFor (the singer the data was loaded for, captured before any await), never a freshly re-read ActiveSingerId — so a singer switch landing mid-operation can't write one singer's data into another singer's file.
  • With no session (session is null — the integration-test path, or pre-seed before any singer exists), PathFor(null) falls back to the unsuffixed file name (SingerFileNames.DefaultSongList = "song-list.json", DefaultTonight = "tonight.json").
  • The store subscribes to session.ActiveSingerChanged and re-raises its own Changed event on a switch, so UI bound to Changed reloads the new singer's data exactly as it would after any mutation.

tests/KHost.Mobile.IntegrationTests/Infrastructure/Services/PerSingerStoreTests.cs exercises this directly: two singers get independent lists and independent on-disk files, a switch fires Changed exactly once (not on a same-singer "switch"), and the session-less constructor falls back to song-list.json.

Atomic writes

IAtomicFileWriter (src/KHost.Mobile.Abstractions/Services/IAtomicFileWriter.cs), implemented by internal sealed class AtomicFileWriter (src/KHost.Mobile.Infrastructure/Services/AtomicFileWriter.cs), gives every store crash-safe writes. JsonFileStore<TItem, TCache> takes it as an optional constructor parameter (IAtomicFileWriter? writer = null, defaulting to new AtomicFileWriter()) and exposes it to subclasses as the protected Writer property:

public async Task WriteAsync(string path, Func<Stream, Task> writeContents)
{
    var tmp = path + ".tmp";
    await using (var stream = File.Create(tmp))
        await writeContents(stream);
    File.Move(tmp, path, overwrite: true);   // same-volume rename → atomic
}

The .tmp file must sit in the same directory as the target so File.Move is a same-volume atomic rename, not a cross-volume copy. An app kill or power loss mid-write leaves only the .tmp file — the rename either completes (new data) or never happens (old data intact), so the previous good file always survives.

Writer.Quarantine(path) is the complementary half: when a file fails to parse on load, it's moved aside to <file>.corrupt rather than left in place to be silently clobbered by the next successful save — preserving the bad bytes for manual recovery, and it logs a warning (ILogger<AtomicFileWriter>, optional/defaulted the same way as every store's logger) naming the file it quarantined, so a corrupt-file event leaves a breadcrumb instead of degrading silently. Best-effort (returns false on a locked/vanished file, logged as a second warning); the caller starts empty either way.

Serialization

Every persisted type goes through a System.Text.Json source-generation context ([JsonSourceGenerationOptions(WriteIndented = true)] [JsonSerializable(typeof(List<T>))] public partial class FooJsonContext : JsonSerializerContext;) rather than the reflection-based serializer. This keeps (de)serialization trimming-/AOT-friendly on the MAUI heads — the source generator emits the metadata the trimmer/AOT compiler needs at build time, instead of JsonSerializer reflecting over types at runtime (which the trimmer can strip and NativeAOT can't do at all). WriteIndented = true throughout so every on-disk file stays human-readable for debugging.

When adding a new persisted type, you must add it to a JsonSerializerContext — miss it and it fails at runtime, not compile time. JsonSerializer.SerializeAsync/DeserializeAsync calls in this codebase always pass an explicit SomeContext.Default.SomeTypeName metadata argument (never the ambient-reflection overload), so a type the generator doesn't know about throws NotSupportedException: … is not compatible with source generation the first time a store tries to read or write it — not when the project builds. Concretely:

  • Add [JsonSerializable(typeof(List<Foo>))] to the owning context (a new one, or an existing one if the type is naturally grouped — e.g. SongListJsonContext also declares Performance because it's nested inside SongListItem).
  • If the type is only ever serialized as a bare element inside another serialized type (like Performance inside SongListItem), the generator still needs its own [JsonSerializable(typeof(Performance))] line if anything serializes/deserializes it standalone (SongListJsonContext does, for exactly this reason).
  • Existing contexts, one per feature: SongListJsonContext, TonightJsonContext, SingerJsonContext, VenueJsonContext, LyricsCacheJsonContext, ProfileJsonContext (all in src/KHost.Mobile.Infrastructure/Serialization/).

Not every type in src/KHost.Mobile.Abstractions/Models/ needs a context — RatingConfig, RatingContext and SurpriseOptions are pure records built at read time from IAppSettings (see Ratings & history, Surprise Me), never written to a file of their own, so they carry no [JsonSerializable] entry.

Settings storage

Services/IAppSettings.cs + Services/MauiAppSettings.cs back every toggle and tunable (auto-fill, YouTube/Spotify/lyrics links, Tonight, Surprise, per-performance rating, update checks, undo window, launch destination, rating weighting, venue detection radius/history, spelling-suggestion level, catalogue region, import lookup delay, haptics, 12/24-hour time, etc.) via MAUI Preferences — a flat, synchronous key/value store, not the JSON-file mechanism above. Every property is Preferences.Default.Get(key, defaultValue) / .Set(key, value), string-keyed ("settings.autofill_metadata", …), with each default matching the hardcoded value it stands in for, so a fresh install is unchanged.

Consequences of that separate storage mechanism:

  • Not in a profile export. SingerProfile (see File inventory) carries only Singer + List<SongListItem> — Settings never travel with an exported/imported profile.
  • Not per-singer. Unlike the song-list/Tonight stores, MauiAppSettings has no singer-awareness at all — every setting is one flat value shared by every singer on the device (confirmed in Singers: Settings sits in the "Shared across everyone" column, alongside Venues).
  • Platform-native storage, not a KHost Cue-owned file: on Android this is shared_prefs/*.xml under the app's private data directory (see On-device paths); scripts/backup-device-data.cs captures it alongside files/*.json for exactly this reason. It is not readable/editable as plain JSON the way the stores above are.
  • LastActiveSingerId (which singer to reopen on launch) and VenuesSeeded/TutorialCompleted/etc. (one-time bootstrap flags) live here too — they're settings-shaped state, not domain data, even though they're not user-facing toggles.

Album-art and lyrics caches

Lyrics (Services/JsonFileLyricsCache.cs) follows the plain JSON-store pattern above — lyrics-cache.json, keyed by trimmed, lowercased title+artist (not the Matching/ normalizer — no accent-stripping or punctuation-folding here), including cached misses (Found = false) so a confirmed "no lyrics" answer never re-hits LRCLIB. Cleared via Settings' danger zone (ClearAsync); CountAsync backs the "X cached" display.

Album art is two layers, not one:

  • Services/AlbumArtCache.cs (IAlbumArtCache, registered singleton) is the disk layer: each cover is a raw image file under album-art/, named by the SHA-256 hex hash of its source URL (no in-memory copy — a repeat request just re-opens the cheap on-disk file). ClearAsync deletes every file in the folder (Settings' "Clear album art"); CountAsync walks the folder
    • Covers are written through IAtomicFileWriter (a .tmp sibling, then an atomic rename), the same as the JSON stores. "Is it cached?" is File.Exists, so a write torn by an app kill would leave a truncated image that counts as a cache hit forever: the card renders broken and never re-downloads. A failed atomic write never reaches the rename, so no target file exists and the next request fetches it again. The read path also treats a 0-byte file as not cached.
  • Services/AlbumArtService.cs (IAlbumArtService, registered scoped — it talks to the WebView through IJSRuntime, itself scoped per Blazor Hybrid WebView) is the in-memory layer on top: it owns a song-id → blob: object-URL map internally, but the UI never reads a URI directly — it calls the public ViewFor(song), which returns an AlbumArtView (a ready --kh-card-art: url('…') CSS declaration, a Loading flag, or AlbumArtView.None) and, by asking, kicks off discovery/download/caching for a cover that isn't ready yet. (The private UriFor(song) helper behind ViewFor is what actually reads/populates that URL map.) This map is not persisted — it's rebuilt from the disk cache (or a fresh network lookup) as the UI re-requests covers, and is held until a singer switch, an edit, or a cache clear.

The in-memory map is capped, viewport-aware rather than keyed on the rendered card count — My Songs keeps every scrolled-past card in the DOM and re-requests its cover on every render, so a cap keyed off rendered count would evict a cover and have the very next render immediately re-request it, forever. wwwroot/js/art-visibility.js runs one IntersectionObserver over every element carrying data-art-song and reports the on-screen set to AlbumArtService.SetVisibleAsync/VisibleArtChanged; nothing currently visible is ever evicted. Off-screen covers get OffScreenCovers (40) worth of headroom for scrolling back before EvictOffScreenAsync drops the longest-gone first. See DEVELOPMENT.md → Design notes for more.

On-device paths per platform

Platform FileSystem.AppDataDirectory (JSON stores + album-art/) Preferences (Settings)
Windows (unpackaged) %LOCALAPPDATA%\KHost\khost.mobile\Data\ — the Data subfolder, not the parent khost.mobile\ (see the gotcha below) app-local settings store
Android /data/data/khost.mobile/files/ /data/data/khost.mobile/shared_prefs/*.xml
iOS / Mac Catalyst the app's private sandboxed container (MAUI Essentials-managed; no exact path is documented in this repo, and there's no adb-equivalent backup tool for it here) platform-native preference store

FileSystem.AppDataDirectory is the Data SUBFOLDER on Windows, i.e. %LOCALAPPDATA%\KHost\khost.mobile\Data\ — the parent folders are the appxmanifest PublisherDisplayName (KHost) and ApplicationId (khost.mobile). Seeding or inspecting persisted state on Windows must target Data\, not the parent khost.mobile\. A build from before the publisher/id rename wrote to the legacy %LOCALAPPDATA%\<User Name>\com.companyname.khost.mobile\Data\, which current builds ignore.

Backing up device data (Android only today) — scripts/backup-device-data.cs, a .NET 10 file-based script run with dotnet run, pulls files/ + shared_prefs/ off the device via adb run-as khost.mobile (Debug builds only, no root needed) into a gitignored device-backups/*.tar.gz:

dotnet run scripts/backup-device-data.cs -- backup                 # timestamped .tar.gz -> device-backups/
dotnet run scripts/backup-device-data.cs -- list                   # what backups exist
dotnet run scripts/backup-device-data.cs -- inspect <file.tar.gz>  # peek inside one
dotnet run scripts/backup-device-data.cs -- restore <file.tar.gz>  # push a backup back onto the device
#   -s, --serial <serial>   target one of several attached devices (also honors $ANDROID_SERIAL)
#   restore also takes -y/--yes to skip the confirmation

Always back up before a device deploy that might reinstall (a signing-key mismatch from a fresh machine/regenerated debug keystore, a package-id change, or a manual uninstall-to-fix-a-crash all silently wipe on-device data — a redeploy alone does not). See Building & testing for the full deploy workflow this fits into.

Testing storage

KHost.Mobile.IntegrationTests exercises the JSON stores against real file I/O (not mocks) via TempAppDataDirectory (tests/KHost.Mobile.IntegrationTests/TempAppDataDirectory.cs) — an IAppDataDirectory backed by a fresh, unique temp folder per test instance, deleted on Dispose():

public sealed class TempAppDataDirectory : IAppDataDirectory, IDisposable
{
    public string AppDataDirectory { get; }   // Path.Combine(Path.GetTempPath(), "khost-tests", Guid.NewGuid():N)
    public string FilePath(string fileName) => Path.Combine(AppDataDirectory, fileName);
    // Dispose(): best-effort Directory.Delete(AppDataDirectory, recursive: true)
}

FilePath(name) lets a test seed or hand-corrupt the raw JSON before a store ever touches it, exercising the load/migrate/corrupt-file paths directly rather than only through the store's own API. Coverage by file:

  • JsonFileSongListStoreTests.cs, JsonFileTonightStoreTests.cs, JsonFileSingerStoreTests.cs, JsonFileVenueStoreTests.cs, JsonFileLyricsCacheTests.cs, AlbumArtCacheTests.cs — each store's own CRUD/corrupt-file/quarantine behavior.
  • PerSingerStoreTests.cs — the namespacing contract described above (independent files per singer, Changed on a real switch only, unsuffixed-file fallback with no session).
  • SingerFileNamesTests.cs — the {id:N} file-naming helpers.
  • SingerProfileImportTests.cs — the profile import/export round-trip (SingerProfileCodec).
  • AtomicWriteTests.cs — AtomicFileWriter's crash-safety contract directly.
  • ProjectTests.cs — both DI entry points (AddKHostInfrastructure, AddKHostClients): every registration resolves from a container built with ValidateOnBuild, the stores are singletons, the song-list store is one instance behind both of its registrations, and each typed client reaches the host's handler hook.
  • Each store's tests also pin that a file written by an earlier build still loads — literal PascalCase JSON, not a re-serialized list, since a naming policy added to a context would round-trip through itself while orphaning every file already on a device.

The integration test project targets plain net10.0 (no MAUI workload needed) and pulls in the MAUI-free source it covers (models, stores) via ordinary project references to KHost.Mobile.Abstractions, .Common, .Infrastructure and .Clients — none of which touch the MAUI workload. See Building & testing for the run commands and the unit-vs-integration split.

See also

  • Architecture — where these stores sit in the app's overall shape.
  • Singers — the multi-singer model this per-singer namespacing serves.
  • Import & export — the profile/venue file round-trip that sits on top of these stores.
  • Building & testing — running the integration test suite, and the device-deploy data-loss gotchas.
  • Conventions — the broader local-store and client-backend coding patterns.

Clone this wiki locally