Feedback after importing a big (~2,700 series) tagged library, matching gets stuck and I think there's an easy-ish fix

[lboyce]

Hey! First off, huge thanks for Omnibus. It's an insanely exciting project and I am so grateful that you've decided to build it. A little background on me: I've been a Komga user since it was first released, I run Kavita too, and Mylar has been my downloader for many many years. Honestly, the idea of a single all-in-one manager and downloader with real user account support is something I've been dreaming about for a long time, so stumbling onto Omnibus was kind of a big deal for me. (Full disclosure, I'm also building my own iOS comic reader app, and part of why I'm putting Omnibus through its paces is that I'd love to be able to integrate with it natively down the road.)

So I finally committed and started moving my whole collection over. About 2,700 series and 37k issues, all coming out of Mylar3, so everything is already tagged with ComicInfo.xml and has a series.json. Wanted to share how it went, because most of it was great but I hit a wall that I think anyone with a big library is going to run into. Assuming I didn't miss something (which could certainly be the case).

The good stuff first. Indexing was fast, and everything that actually matched came in perfectly. My test publisher matched 100% with zero mislabels. Once I split my annuals out into their own folders it stopped misreading the parent series, and it correctly skips folders that only have metadata and no actual file. Really solid foundation.

Then I hit the matching wall. Here's what happened: the library scan matched the first ~227 series and then slammed into ComicVine's rate limit (200 an hour) and just stopped. Everything after that got dumped into the Smart Matcher as unmatched. No big deal, I figured it would pick back up once the limit reset. But based on my use-case it apparently doesn't. I waited over two hours and the number never moved on its own.

The Smart Matcher then shows those 200+ series as unmatched, but even then it just fills in a suggested match and waits for me to click Accept on each one but still hits the same rate limit, so it only gets through a couple hundred before stalling again, but also now forcing me to manually fulfill the library matching instead of the system doing so on it's own. The hope was that I could ingest a large library, let it run on it's own for a day or two (whatever the CV rate limit needs for cooldown to populate), and come back and fix any issues I find.

So I'm assuming that the automated matching doesn't resume on its own after the limit resets. It basically gives up. So on a 2,700 series library I'm looking at ~200 matched and 2,000+ sitting in a manual queue that I'd have to grind through at ~200 an hour, clicking accept and eyeballing each guess so I don't lock in a wrong match. That's realistically days of clicking, and it's the main thing stopping me from bringing my whole library over.

Here's the part I think is the real kicker though. The ComicVine IDs are already sitting in my files. The ComicInfo.xml has the ComicVine issue ID right there in the Web and Notes fields, and the series.json has the ComicVine volume id (comicid). If Omnibus just read those, it wouldn't have to search and guess at all. It could match directly by ID, which would be instant, wouldn't burn the rate limit, and would never pick the wrong volume. And since Omnibus already writes a Mylar-spec series.json with comicid in it, reading that same field back in feels like it might be a pretty small lift? I do understand that a library like mine that already has that info isn't always going to be the case, but having that option to assist in the matching would be a huge advantage for large libraries getting migrated over from other downloaders like Mylar.

So my wishlist, roughly in order of how much it would help:

• Have an option to match the Comic Vine ID if it's already in the files (ComicInfo Web/Notes, or the series.json comicid) instead of searching. This one alone would basically make tagged library imports just work.

• Auto-retry the unmatched queue in the background after the rate limit resets, so a big library finishes matching itself overnight instead of me babysitting it.

• Maybe auto-accept the high confidence Smart Matcher suggestions so I'm not clicking Accept a couple thousand times, and it can leave suggestions it doesn't feel confident about.

A couple of smaller things I noticed while I was in there: titles with a slash like Hack/Slash don't match (I think the slash trips up the search).

And in my experience, the web UI gets really sluggish and unresponsive while a big scan is running, so I had to just wait it out and check back later.

Again, none of this is a complaint. I love the app and I'm rooting for it. Just wanted to share what a large migration actually looks like right now, and where I think it could get a lot smoother. More than happy to test anything or send over files and logs if it would help!

[lboyce]

One other thing I'm trying to figure out how to do. After ingesting my library I'm trying to find a way to mass "monitor" titles. It'd be really great if I could at least filter by "Ongoing" series (or ended) so that I could add my on-going series to monitoring status. Right now I'm having to just go through the library and eye it and monitor one series at a time.

[hankscafe]

Thank you for that detailed feedback. For your ask regarding reading the comicinfo.xml and series.json, that absolutely should be happening so that is a bug that I will be looking into.

I will definitely take all of this and see what can be done to integrate these items and improve the app experience.

I would ask which version of the app you are currently using, main branch or rust-engine branch?

Ultimately, I am working on merging the branches as performance did become an issue with the fill node branch (main).

Again, thanks for that great feedback.

[lboyce]

Thanks so much!

Right now I'm using the main branch, v1.1.0.beta.076. I just saw the mention on here about the rush-engine branch yesterday and the performance improvements. I do have a feeling that would help with the UI during scans. I'll likely wait until that is merged before using it though. But excited to try it!

I'm really excited about the future of this program. Thanks so much for putting it together.

[hankscafe]

Recently pushed v1.1.0-beta.083 to the unified branch and it covers pretty much everything you listed here.

Your wishlist:

1. Matching by embedded IDs - The scanner now reads the ComicVine volume id straight out of series.json (comicid) and the issue IDs from ComicInfo.xml, including the Web and Notes fields. A folder with a series.json matches instantly with zero API calls, so a tagged Mylar library like yours should basically import itself without touching the rate limit. This also works for CBR files now, which the old ComicInfo reader couldn't open at all.

2. Auto-retry after the rate limit - There's a new background sweep that runs hourly and works through the unmatched queue. It checks file evidence first (free), then falls back to searching. It tracks the ComicVine budget and stops about 30 calls before the wall, then picks up where it left off on the next run. So, a big import finishes itself overnight instead of stalling at ~227 and waiting for you.

3. Auto-accepting suggestions - There's a new Match Confidence Mode setting: Trust (auto-accept file IDs and confident name matches, with a tunable threshold), Confirm (file IDs auto, suggestions wait for you), Auto (only near-exact matches), or Custom (no automation). Default is Confirm.

Also in this build:

• Hack/Slash type titles search correctly now. Turned out ComicVine's own search returns nothing when there's a slash in the query, so we strip it before sending.

• Your mass-monitor ask from the comments: there's a Status filter (Ongoing/Ended) on the library page now, so you can filter to Ongoing, select all, and bulk monitor in one shot.

• A "Prefer Embedded File Metadata" toggle, so syncs only fill in blanks and never overwrite what came from your files.

• Found a bug while digging into this: on a fresh scan, every issue in a folder was getting stamped with the first issue's provider ID. That's fixed, each file gets its own now.

While I was in there, I also fixed the credits sync (writers/artists/characters were coming in empty on unified) and the scan now stores all the ComicInfo metadata directly, so your tagged files should populate everything without waiting on a sync.

Please let me know how it goes for you as you are the first individual with such a large library to use the app and provide feedback.

[hankscafe]

Quick follow-up — just pushed v1.1.0-beta.084 to unified with a couple of additions that came directly out of this thread.

1. You can now see the background sweep working. In beta.083 the sweep did its job but was invisible series just quietly disappeared from the unmatched queue and there was no way to tell the automation was running. The Smart Matcher now has a Background Sweep card showing when the last run happened and exactly what it did (e.g. "41 matched from file metadata · 3 auto-matched by search · 7 left for you · 12 deferred (API budget)"), plus a Run Sweep Now button if you don't want to wait for the hourly schedule. For an import your size, this gives you a live view of the queue burning itself down instead of having to diff the unmatched count by eye.

2. Sweep notifications. There's a new notification event (works with Discord/Pushover/Telegram/Apprise) that fires only when a sweep actually matches something so you can kick off a big import, walk away, and get pinged as it works through itself, without hourly "nothing to do" noise.

3. Metron per-issue credits (opt-in). Less relevant to your Mylar/ComicVine setup, but for Metron users: Metron's list API doesn't return creator credits at all, so Metron-matched issues only had credits if the files carried them. There's a new settings toggle that makes syncs fetch full credits (writers, artists, characters, story arcs) via per-issue detail calls budgeted against Metron's daily limit, resumes on the next sync if it runs out, and each issue only ever costs one call. Off by default since it is quota-heavy on large libraries.

Also fixed two bugs in the import-path Metron sync while in there: re-syncs could overwrite ComicInfo-derived credits with empty values, and previously deep-fetched issues were getting demoted and re-fetched unnecessarily. Both now follow the same "never overwrite what came from your files" policy as everything else.

Let me know how this works and please share any feedback. I'm going to be looking into your other post about the size of your library
as well, although some of this may help with that as well.

I'm always looking to improve the app and experience so I will see what can be done.

Thank you for all of the feedback.