Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve synchronization of track metadata and file tags #2406

Merged
merged 24 commits into from Jan 9, 2020
Merged

Improve synchronization of track metadata and file tags #2406

merged 24 commits into from Jan 9, 2020

Conversation

uklotzde
Copy link
Contributor

This PR introduces a simpler, more robust, and extensible synchronization between track metadata that is stored in the database and file tags. We currently only store a subset of properties in the database.

I have used and tested this code for many weeks and didn't notice any issues or lost file tags. It is also needed for synchronization of tracks with external libraries like #2282 that store an extended set of metadata.

Reorganization

  • Remove special case handling of certain properties from TagLib code
  • Remove special case handling of certain properties from TrackInfo/AlbumInfo/TrackMetadata
  • Move tag migration code from TrackDAO into TrackRecord
  • Move all code for merging imported file tags into TrackRecord

Import

SoundSourceProxy::updateTrackFromSource(): Import and merge file tags when loading track objects on the fly. This will populate all missing fields that are not (yet) stored in the database.

Export

Track::exportMetadata(): Import and merge file tags once again before exporting metadata into file tags. We have to reload the file tags anyway to check for differences as we already do now, because files should only be modified if differences are detected. Additional debug logs show both the existing and the modified metadata before writing file tags.

@uklotzde uklotzde changed the title Improve synchronization of track metadata with file tags Improve synchronization of track metadata and file tags Dec 19, 2019
@uklotzde uklotzde added this to the 2.3.0 milestone Dec 22, 2019
@uklotzde
Copy link
Contributor Author

I would like to get this into 2.3 to avoid maintaining 2 largely different versions.

@ronso0
Copy link
Member

ronso0 commented Dec 22, 2019

Could you share a simple routine to test this?

@uklotzde
Copy link
Contributor Author

Even if EXTRA_METADATA is enabled (a prerequisite) the extended properties are not visible in Mixxx. You should not notice any changes, unless you send data to aoide. The improvements are mainly behind the scenes and ensure that newly added fields in the Mixxx database get populated lazily from file tags as expected.

I will think about enabling some trace logs to report all properties when loading a track.

src/track/trackrecord.cpp Outdated Show resolved Hide resolved
Copy link
Member

@daschuer daschuer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some first comments.
Unfortunately, the functional changes are hard to track.

src/track/trackrecord.h Show resolved Hide resolved
src/track/trackrecord.cpp Outdated Show resolved Hide resolved
src/track/trackrecord.cpp Outdated Show resolved Hide resolved
src/track/trackrecord.cpp Outdated Show resolved Hide resolved
@Holzhaus
Copy link
Member

Btw we should really add a flag to activate EXTRA_METADATA and enable it for the CI builds so that the code is built and tested on Travis and Appveyor.

@uklotzde
Copy link
Contributor Author

uklotzde commented Dec 23, 2019

@Holzhaus We currently don't have a dedicated feature flag in master that enables the __EXTRA_METADATA__ code, only PR #2282 makes use of this. Any ideas?

@uklotzde
Copy link
Contributor Author

@daschuer I agree that the changes compared to the current state are hard to track. It is a major redesig.

Currently all the code for synchronizing metadata from the database with file tags is scattered all around. We have special case handling on different levels in TrackDAO and the low-level TagLib code, where it does not belong.

@Holzhaus
Copy link
Member

@uklotzde Have a look at this branch: https://github.com/Holzhaus/mixxx/tree/extra-metadata-flag

It's untested though, since I'm currently travelling and had to use my phone. Using vim on a touchscreen ain't easy btw ;-)

@uklotzde
Copy link
Contributor Author

@Holzhaus I hesitated to add a new feature flag without an actual use case apart from aoide. But we could do this.

@Holzhaus
Copy link
Member

The flag would obviously be disabled for release builds, but it would make it easier to test this. How do you test it currently? Do you put a #define __EXTRA_METADATA__ somewhere? Or do you export CXXFLAGS=__EXTRA_METADATA__?

@uklotzde
Copy link
Contributor Author

uklotzde commented Jan 3, 2020

I found some issues in the way cover art is guessed. Files in the folder were not considered if reloading metadata from source, only embedded covers. The CoverInfo source of all files without embedded cover is reset to UNKNOWN and the LibraryScanner will revisit them on the next run. Not a big deal but annoying.

Due to merge conflicts I have not fixed it on master and only in this branch.

...to consider not only embedded cover art but also image files in
the folder of the track's file. Otherwise the source of the cover
info would be reset to UNKNOWN.
src/library/coverartutils.cpp Show resolved Hide resolved
src/sources/metadatasourcetaglib.cpp Show resolved Hide resolved
src/sources/metadatasourcetaglib.cpp Show resolved Hide resolved
src/track/bpm.h Outdated Show resolved Hide resolved
src/track/track.cpp Show resolved Hide resolved
Removes redundant code out of TrackDAO that doesn't belong there!
@uklotzde
Copy link
Contributor Author

uklotzde commented Jan 4, 2020

  • Removed redundant code from TrackDAO that doesn't belong there
  • Don't try to extract the embedded cover twice if it is not available when reloading metadata

src/track/bpm.h Outdated
// or not.
// TL;DR: If metadata export is enabled ID3 tags will be rewritten
// for all files with a fractional bpm values even if their metadata
// has not been modified.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment need to be updated.

@daschuer
Copy link
Member

daschuer commented Jan 9, 2020

Thank you LGTM. I have also done some brief testing, so we can risk to merge this 1k PR
Thank you for the work.

@daschuer daschuer merged commit 1473440 into mixxxdj:master Jan 9, 2020
@@ -47,4 +49,88 @@ bool TrackRecord::updateGlobalKeyText(
return false;
}

namespace {

void mergeReplayGainMetadataProperty(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a warning now:

/home/jan/Projects/mixxx/src/track/trackrecord.cpp:54:6: Warnung: »void mixxx::{anonymous}::mergeReplayGainMetadataProperty(mixxx::ReplayGain&, const mixxx::ReplayGain&)« definiert, aber nicht verwendet [-Wunused-function]
   54 | void mergeReplayGainMetadataProperty(
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, damn. An #ifdef is missing. Fix will follow.

@uklotzde uklotzde deleted the dev_library_metadata branch January 9, 2020 22:43
@Holzhaus Holzhaus added this to Done in 2.3 release Mar 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
2.3 release
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

4 participants