[Feature Request] Need a better way to update the database from file tags

[troycarpenter]

In listings for artists and albums, there is a choice "Update from tags" which will re-read tags from files and update the database appropriately.

I have NOT figured out how to make the same thing happen on a catalog level repeatedly. I have tried the Clean / Verify / Update commands on the catalog, and while some combination of those commands finds changes, but many times those commands do NOT find all changes, and I have to resort to using the "Update from tags" commands at a album or artist level to catch everything.

This may be tied to the database problems I'm seeing in #3313, but I'm not sure. If it is, then it would be clear evidence that the database lockups are causing loss of data in the database. I've been using various external tools to fix tags on my library, but I'm having a hard time getting Ampache to see the changes in a scalable manner (meaning I don't have to touch each asset via the GUI).

[lachlan-00]

you can set up a watch using inotify. then any time a file changes it gets updated including when deleted

https://github.com/ampache/ampache/blob/develop/docs/examples/inotifywait.sh
https://github.com/ampache/ampache/blob/develop/docs/examples/inotifywait.service

When you run inotify it catches the kernel notifications for file changes and then runs the script. the updateCatalogFile cli option performs an action per file and then you don't have to actually run anything manually.

[troycarpenter]

Thanks. I will have a look at it and see if it helps my situation.

[mitchray]

I'm seeing the same while I'm going through adding lyrics, they are only registering after doing the manual Update from tags

updateCatalogFile and updateCatalog should have the same effects in theory, just single vs multi but that might not be the case?

[lachlan-00]

They do the same things but for songs updateCatalog does it by album and updateCatalogFile does it per song.

Song::insert is different to the catalog verify method (Catalog::update_song_from_tags()) and you're not getting lyrics when you add?

[mitchray]

Hmm well, I just tried to reproduce it on a fresh catalogue but the lyrics were updated with every method I tried, so I'll monitor it again to see if and how it returns

1. new catalog = reads lyrics
2. update from tags (album view) = updates lyrics
3. update from tags (artist view) = updates lyrics
4. run:updateCatalog -cage = updates lyrics (but didn't previously)
5. "Update all" in Show Catalogs = updates lyrics (but didn't previously)

[troycarpenter]

I still occasionally come across incorrect media data that is only fixed by hitting "Update from tags". For instance, I pulled up "The Guardians of the Galaxy" soundtrack Volume 2 and noticed a few errors (like the album title wasn't proper, as well as the AlbumArtist was wrong. Once I hit "update from tags", the database corrected the data on that album.

As I've said earlier, I had expected the catalog update/verify functions to find and correct those errors. I see in the logs that somethings get caught while others don't. In the most egregious cases, I have to move the album out of the directory structure, clean the catalog to remove the entries, then move the album back and then Add or Update the catalog to fix issues.

I did have this set:
catalog_verify_by_time = "true"

But now I've commented that out (since the default is false) and trying a verify on one of my catalogs. So far I'm seeing it catch lots of data differences and migrating things. Also running into database deadlocks in the process.

If this works, then maybe I'll verify all the catalogs, then put that variable back to true, based on the large database warning.

[mitchray]

Oh that's interesting, I have that set to true as well

Here is the check to determine whether to skip updating:

ampache/src/Module/Catalog/Catalog_local.php

Lines 673 to 675 in c4f9662

	// check the modification time on the file to see if it's worth checking the tags.
	if ($verify_by_time && ($this->last_update > $file_time || (array_key_exists('min_update_time', $row) && (int)$row['min_update_time'] > $file_time))) {
	continue;

[troycarpenter]

Since it's evening here, I just kicked off a "verify all" with that setting to false to run overnight. I've already seen it (via the logs) find differences in the first catalog it came to, one I thought was already synced with the database after I updated Musicbrainz tags with Picard.

[troycarpenter]

At least 10 hours later, the verify is still happening, but close inspection seems to show it's rescanning the same media over and over. I think it got stuck in a loop on my first catalog. I've reset the docker container and now have manually started a verify on the second catalog in my list (and I see it updating based on tags, so yeah, the original "verify all" never made it out of the first catalog it seems).

[troycarpenter]

I've noticed that there's a difference between running a verify from the GUI, and running it from the CLI. When I run a verify from the GUI on as single catalog, there's lots of output and the process never seems to conclude.

However, if I run it from the CLI, it blew right through all the catalogs in no time flat. What I don't know if if the CLI process actually found different tags or not.

[troycarpenter]

Another update. I modified the tags for an artist and all the artist's albums (26). I did an update from the CLI (./cli run:updateCatalog -cea Restricted local). Some output, but it doesn't look like those albums updated.

I then initiated a verify from the GUI on that catalog, and the log shows it finding all the tag changes.

I don't know what the best method is for getting these changes in the database.