Speed up publish when no media files were altered

[frankenjoe]

Currently publish() always checks if media was changed. To do that the checksum of all media files in the database has to be calculated. This can take quite some while on large databases. However, most of the time only the metadata is changed and maybe new media is added. That existing media changes is a rather rare case. So I wonder if we should give the user the option to skip the test for altered media.

[hagenw]

So I wonder if we should give the user the option to skip the test for altered media.

I think that would be an easy and good solution. In the long run it could also help that you don't have to download all media files with audb.load_to() if you just want to fix the header or tables.

[hagenw]

But of course it's not exactly the same, as you might not alter existing media files, but add new ones.

[hagenw]

The "Find media" part seems indeed to be the slowest part of publishing a database. We cannot easily avoid this with new data (besides maybe providing the opportunity to provide pre-calculated values?).

But for updating large databases we should definitely provide an option to skip it.

[hagenw]

Speed of checking existing media has increased, but it might still be a problem when you have a large number of files. On the other hand when adding the argument to skip checking, we introduce a possible source of error during publication.

[frankenjoe]

The worst use-case is if you neither upload or alter media, but only change the metadata.

On the other hand when adding the argument to skip checking, we introduce a possible source of error during publication.

I would say we could take that risk given the extreme speed up we would gain.

[frankenjoe]

#216 now implements a solution without adding a new argument. Media files that are referenced in the tables and are part of the previous version, must no longer exist in the build folder since for those files, we can safely assume they remain unchanged.