[Enhancement]: More Flexible Library Structure

[AshbyGeek]

Describe the feature/enhancement

My library REALLY didn't scan well. I keep all my audiobooks as single files (almost all .m4b, but there are a couple short stories that are just an mp3 file. Should probably convert them to .m4b too). I also greatly prefer not to create directories that are going to hold a single file.

This means that my directory structure is something like this:

• Brandon Sanderson
  • Cosmere
    • Stormlight Archive
      • "Book 1 - The Way of Kings.m4b"
      • "Book 2 - Words of Radiance.m4b"
      • "Book 3 - Oathbringer.m4b"
      • etc
    • Mistborn
      • "Book 1 - The Final Empire.m4b"
      • "Book 2 - The Well of Ascension.m4b"
      • etc
    • "The Sunlit Man.m4b"
    • "Warbreaker.m4b"
  • Rechoners
    • "Book 1 - Steelhear.m4b"
    • "Book 2 - Firefight.m4b"
    • "Book 3 - Calamity.m4b"
  • "Snapshot.m4b"
  • "The Frugal Wizards Handbook for Surviving Medieval England.m4b"
• [more authors]
• [single books]

The Scanner took a look at my folder structure for all of this and decided that Brandon Sanderson has a single book. Under the 'files' tab that book lists all 41 of Brandon Sanderson's books (and short stories) in my collection.

Eventually I figured out that putting .m4b files into folders pretty well sorted things out, but it would be REALLY nice to not need that.

The relevant function seems to be groupFileItemsIntoLibraryItemDirs in server/utils/scandir.js. Could that function be changed to be aware of .m4b files and return them as individual books always? That should get most people with a library like mine up and running faster.

Long term though it seems like a generally more flexible system would be useful; and possibly also adding the ability to modify the relationship between books and audio files after the fact in the web-ui. Or perhaps with a .config type file in folders that are intended to contain multiple files intended to be part of the same book.

[nichwall]

This can get complicated if metadata files (either ABS generated or from somewhere else) or cover art is stored in the same directory. For the most part it can probably be inferred that anything with the same beginning part of the name is the same item, but many times it's just a cover.[ext] or metadata.[ext], with no way to tell which library item it goes to. Some people store multiple covers of the same book as well.

The documentation shows each library item in its own directory, but this could be more clear.

Sort of relevant issue: #528

[AshbyGeek]

Ok, I get that. It still could be done but would require essentially 2 significantly different ways of scanning a directory. Those would have to be backed by documentation and decision logic. Gets to be quite a bit of work and still leaves a rather inflexible scanning system.

I do like suggestion #528. I've seen systems like that; some that worked well and some that seemed unnecessarily finicky. A good implementation would resolve some of the underlying rigidness of the scanner. That issue linked a tool that only allows for one pattern though, which still limits the system's flexibility. If instead it worked more like a .gitignore allowing multiple lines of patterns regex style, that would be dang awesome.

What of using a .config type system? Add a few audiobookshelf.config files to your library directories to override default scan settings for that folder and subfolders. Things like:

• specifying whether files in the current directory all belong to a single book or multiple individual books
• specifying whether directories in the current directory are expected to be authors, series, book titles, or ignored
• specifying file/directory name/extension patterns that should be ignored or treated as metadata or cover art
  Yeah, this takes a lot of inspiration from .editorconfig and .gitignore files.

P.S. Should I move these comments to that thread and discuss them there?

[AshbyGeek]

Also, I agree that a quick touch-up of the documentation would help relieve the initial issue, which was my misunderstanding of the system.

[nichwall]

I agree with you that more configuration would be nice, but also not sure how long it would be until it's implemented. More options means more edge cases, and I haven't put much thought into something like that (not sure if advplyr or someone else has either with how many moving parts there are on the project).

P.S. Should I move these comments to that thread and discuss them there?

This thread is probably fine. I think there's an issue that has more discussion, but I can't find it so I just linked to that one.

Also, I agree that a quick touch-up of the documentation would help relieve the initial issue, which was my misunderstanding of the system.

Yeah. A lot of these little things need to be clarified for newer people, so thanks for working on that :)

[AshbyGeek]

Let me know if you have suggestions for the exact wording. I'm not exactly a word-smith.

[advplyr]

Another thing to consider is that some audiobooks are comprised of multiple m4b files so we couldn't assume that an m4b file is an individual book.

[alderete]

I've run into a similar issue with my own library of audiobooks. My organization comes from 15+ years of using iTunes to organize my audiobooks. If you have the "Keep iTunes Media folder organized" setting enabled, iTunes puts all books by the same author into a single folder. (Music gets grouped into albums, which are translated into separate folders, but iTunes doesn't didn't do that for audiobooks.)

Multi-part books are "grouped" by the Title field (which I think is actually Album in the actual ID3 tags). Exact same value for Title means you're part of the same book. Differentiation between parts and sequence control is done by either the Name field (Chapter in ID3 tags?), or disk and track numbers. (I forget some details here, but could research it if this is a serious avenue of investigation.)

The thing is, most of my books are a single .m4b file, with chapter markers, and my metadata is fantastic. Books without DRM have been merged into a single file, and I've been diligent over the years in making sure the metadata is correct and consistent for authors, titles, series, and so on.

What I really want is a setting in AudiobookShelf "Ignore folder structure in favor of embedded metadata". I was hoping that disabling the "Folder structure" item in the library Scanner settings would do the trick, but apparently not. Sub-folder grouping into a book happens whether that's enabled or not. (That seems a bit counter-intuitive.)

The documentation makes it seem like, once you start down the road of sub-folders, you have to go all the way down that road, or you end up with a mess like mine. Which means there's two options for folder organization:

1. Fully structured (labor intensive) folders, sub-folders, sub-sub-folders, etc., or
2. An undifferentiated mess of all audiobooks in one top folder.

Really would be nice if there was a middle option. Grouping by author in folders is both very natural, and matches how iTunes did it for more than a decade. (Not that iTunes was ever the perfect audiobooks library manager. On the other hand, haven't found anything better in the years since Apple moved to the Music app, either...)

[advplyr]

What I really want is a setting in AudiobookShelf "Ignore folder structure in favor of embedded metadata". I was hoping that disabling the "Folder structure" item in the library Scanner settings would do the trick, but apparently not. Sub-folder grouping into a book happens whether that's enabled or not. (That seems a bit counter-intuitive.)

That is the order of precedence for setting metadata on the book, it doesn't have to do with how the books are grouped. In the book scanner guide the first step is to group files into books https://www.audiobookshelf.org/guides/book-scanner

I'm not sure I understand what your suggestion is to handle both single file books and multi-file books in the same directory structure.

[ElDubsNZ]

I'm not sure I understand what your suggestion is to handle both single file books and multi-file books in the same directory structure.

Metadata. It should be able to override the folder structure rules.

So if I've got the seven Harry Potter audiobooks in one directory, by default audiobookshelf should identify them as a multi-file book as usual. But this should be overridden if ABS identifies metadata that makes clear they're distinct books. Such as album name on mp3 files, or title on opf/nfo files etc...

This means ABS can work with the way we store our files, rather than us storing our files in a way that works for ABS. Just creates flexibility.

[alderete]

That is the order of precedence for setting metadata on the book, it doesn't have to do with how the books are grouped. In the book scanner guide the first step is to group files into books https://www.audiobookshelf.org/guides/book-scanner

I understand this. This is why I wrote that, once you start down the sub-folders route, you have to go all in on it. (Which I do not want to do.)

Question for you: Do you not consider the title of a book to be a piece of metadata? I certainly do. It's the most fundamental piece of metadata, but it's still metadata. I don't see the purpose of forcing the use of folder name to set the book title (and thus group/separate files into books).

I'm not sure I understand what your suggestion is to handle both single file books and multi-file books in the same directory structure.

My suggestion is, ignore the directory structure. Don't group on it before you start processing. Use the embedded metadata to set the title, and group on that.

I get that it's not for everyone. I'd be shocked if folks who have already done all that work to create the elaborate folder structure would want to undo that work. I certainly don't want to repeat all the work I did to get my metadata correct!

But using all of these nested folders feels like a "lowest common denominator" solution that's ignoring the value of embedded metadata. Why would you want to separate the metadata about a title from the actual files? Embedding all of the details in the folder names, in a folder structure that might look familiar to someone using a computer in the 90s (or 70s) seems like a step backwards. We have this great solution, separate, structured metadata fields, a significant improvement over just using semi-structured string file and folder names. Why not use it? Or, at least let folks who want to use it still be able to use AudiobookShelf?

But, in the end, it's your project. If you don't see the value, not going to fight about it.

[advplyr]

It's not about seeing the value in it. It is about it being a viable option.

I've spent a lot of time now working with meta tags for audiobook audio files and I don't see how you could determine based on the meta tags whether an audio file represents an entire book or is a track.

The best we could do is take a guess at it, which is fine as an additional setting to opt-in to, but there is no fool proof way to handle this that I can see. If you can think of an implementation that handles this and want to write some pseudocode or mock something up somewhere I'll take a look at it.