Facility to merge TMSU databases

[eigengrau]

For users who store their files in a distributed fashion (say, WebDAV, git-annex, etc.), it might be useful if TMSU included a sort of «merge driver» to unify two TMSU databases that have diverged. This way, one wouldn’t have to necessarily make sure only one client updates tags at a time.

I guess the problem would be tag deletions. A union merge strategy would bring back deleted tags. So this would entail a three-way merge instead. Not sure if it’s the most pressing issue, but having support for decentralized tagging would be awesome.

[oniony]

Could you please talk through an example as I'm not sure what you mean.

[eigengrau]

Sure! Suppose you store your files on Dropbox/git-annex/etc. You probably want to do so for the TMSU database as well. This is possible, as long as one makes damn sure not to use TMSU on two clients at the same time; otherwise the files will diverge, and (depending on your sync solution), you wind up with two conflicting copies of the DB file. What would be nice, though, is if TMSU had a facility that could merge the diverged databases back into one coherent set.

If there are only additions, a coherent set might be constructed by going through file IDs and constructing the union of tags. The problem is that one client might delete a tag. And the only way of knowing if a tag was deleted is when you have the common, base version of the file from whence the two copies came. So I guess, given the complexity of it, this feature won’t be too enticing to you because probably only git-annex would be able to support it (because it gives you the base version of the file).

It’s more of a luxury problem, I guess :) … Keeping my data spread out over lots of git-annex clients, I would finde it awesome, but most people can probably deal with making sure never to run tmsu on more than one machine at a time.

[oniony]

OK thanks for the detail.

Firstly I imagined users would store the database with the files such that the database is always in sync with the files. I suppose that's not always viable though.

I'll keep this open as it's a valid point: but it might be some time before I come to look at it.

[0ion9]

For the time being, isn't this something that can be implemented by a journalling wrapper?
It seems to me it could be very simple:

• use the wrapper instead of TMSU directly
• The wrapper calls tmsu with all the commandline arguments it was provided
• If the return status is normal, then it records the full TMSU command that it executed in a shell-script format 'journal' (keeping in mind the need to properly quote all arguments in case they have spaces or other special characters), along with the current working directory -- eg cd "/home/me/files"; tmsu tag "music4.mp3" "metal"
• if the return status is not normal, it does not record the command.

I am imagining the process of 'merging' as simply executing the journal file, then, if its return status is normal, truncating the journal file to 0 bytes.

This is pretty much "the simplest thing that could possibly work", but it seems robust for all normal operation of TMSU.

The only potential gotcha I've noted is that the current working directory may be different on different machines, for example because of differing usernames. A solution that would address many cases including most Dropbox-related uses, is to substitute the user's current home directory with "$HOME" in the journalled cd commands (as in cd "$HOME/files"; tmsu tag "music4.mp3" "metal").

HTH,FWIW.

[eigengrau]

I like the simplicity of the idea; the problem tough, is that with a journal, there comes a point when you need to commit the journal to the data store. And at this point, the original problem reappears, namely that you can’t atomically “lock” the database between different clients; as such the files can still diverge, and would require a three-way-merge after all.

[0ion9]

Well, yes; the only way to get rid of that problem 100% is for TMSU to communicate with the other machines when performing operations (which, TBH, seems way out of its scope). But automatically journaling and then automatically committing that journal, seems like it achieves the goal generally as long as you aren't tagging the exact same file differently within a minute on different machines. (which, given that you are only one person, seems like a reasonable assumption ;)

(journalling also has some advantages for conflict resolution that TMSU itself doesn't have and AFAICS cannot have -- eg. you can include timestamps to help resolve what tagging takes priority)

[oniony]

I don't think this is going to happen any time soon.