Add direct menu option to setup bidirectional sync with another database

[akontsevich]

Summary

Currently bidirectional sync does not work or not implemented. Seems current DB file merging syncs currently opened DB, but other DB file left untouched. This function does not work like original in KeePass which synchronizes both files. Please add the feature or fix it, still need to use KeePass for sync. Also why do You ask password for another DB file as it has the same pass by default. KeePass does not ask it.

Related to #90 and #637.

[droidmonkey]

Bidirectional sync is supported with KeeShare in 2.4.0. See #2109

[akontsevich]

@droidmonkey What is KeeShare, how to use it, see no synchronization there. Very inconvenient comparing to KeePass.

[droidmonkey]

We are almost done with our documentation.

[akontsevich]

@droidmonkey with KeePass I just go to menu, select file - and it is done. Why it can't be done same way in KeePassXC?

[droidmonkey]

We are not aiming to be EXACTLY like KeePass.

[droidmonkey]

I think your request is actually to make a direct menu action that sets up a synchronization with some other file. This can be achieved with KeeShare on the root group, but not from the file menu. If you edit the root group you can setup a sync with some other database file.

[akontsevich]

We are not aiming to be EXACTLY like KeePass.

@droidmonkey Why not to take best from it? I started to use KeePassXC because it worked more convenient than KeePass: Ctrl-C, tray icon, native KDE look and feel, Hi/Low/DPI, etc. Now with recent changes it is not - forced to get back to KeePass.

If you edit the root group you can setup a sync with some other database file.

Very complex and inconvenient, need to setup KeeShare first. Why not reopen this ticket and implement for many many users which want this simple feature?

[akontsevich]

Awesome! Thanks!!! :)

[janedough1331]

I second this. Very inconvenient. And i recall last time i tried using keepassXC i loved it, but this was really the nail in the coffin that made me go back to keepass2

[droidmonkey]

Lol what? The nail in the coffin was a one time setup function that is fully accessible through the group edit??

[Dannniello]

Maybe I'm missing something, but KeeShare Synchronize it is much different option than Synchronize in KeePass.

KeeShare Type: Synchronize is for constant database sharing, i.e. every save will export data to file provided in path. If path is not available - save will generate error with information that export failed.

In my case I need manual "one time only" synchronize option - exactly the same like it is working in KeePass. Why? Because I use it to synchronize database between my laptop and desktop from time to time.

Workaround - use merge command two times:

keepassxc-cli merge -s /path/to/desktop/database.kdbx /path/to/laptop/database.kdbx
keepassxc-cli merge -s /path/to/laptop/database.kdbx /path/to/desktop/database.kdbx

It works, but it is not perfect - files after that will have different checksum... KeePass Synchronize is generating "perfectly" synchronized files (the same checksum).

[droidmonkey]

KeeShare is an entirely different feature than synchronize the database itself.

[dmrzzz]

I feel much enlightened by @Dannniello's comment; I found this issue months ago while looking for a KeePassXC operation called "synchronize" (by analogy with KeePass), but never even noticed the "merge" operation until now.

I get the impression that what OP and several others really want here is not even KeeShare Synchronize at all, but rather a menu action that performs a very (conceptually) simple one-time two-way merge using the same master key.

I think part of the difficulty is that this area of KeePassXC feels a bit like hunting squirrels with tactical nuclear weapons (to borrow a favorite quote from a former teacher). The KeeShare feature and its documentation in QUICKSTART.md are geared toward the complex use case of sharing a subset of your credentials with other people, which is pretty amazing, but not everybody needs that, and it's not at all obvious at first glance how to apply this to the much simpler use case of propagating changes between what we logically think of as two copies of the "same" database (no subsets, no other people, no different keys). Possible solution: a new quickstart section "Using Merge" which covers how to merge changes one-way from database A into database B (and what this means for item history), how to do a two-way merge that updates both A and B, and then how to automatically synchronize them.

[akontsevich]

hunting squirrels with tactical nuclear weapons

LOL @dmrzzz good point! Need to make soft for people needs I think. I think topic has 2 superfluous words: KeeShare is not the only way to implement the feature.

[ocumo]

Thanks @dmrzzz for putting the issue so clear. I share 100% of your points.

The fact is, as so many people trying to move to KeePassXC from KeePass2, I have the exact same need of a bidirectional sync with another database, a very common use case that has been so very well described by others: I just need to sync my database between my laptop and my desktop. I do this with seamlessly with KeePass2, as explained by others. I would like to use KeePassXC, though, but without this being possible except in a convoluted way, either by the "killing a fly with a cannon" or with an inconvenient and risky operation of merging and then copying the merged file over the other, that brings no advantages.

Now that I have finally convinced my wife to use a password manager, I won't even try to explain to her that to sync her laptop database with that in her desktop PC, she would have to start learning how to fiddle with bash commands or to learn a complicated concept of group sharing, which is not even remotely intuitively suggesting "synchronization", other than apparently it happen as some kind of secondary effect as long as one configures certain options in a special way (with no documentation available -at least it has taken me a lot of time of searching and reading to no avail).

After reading that this request has been moved from the v2.5.0 milestone, I truly hope that that doesn't mean the issue will be closed now?

Please dear devs, consider the use case of simple users that are not developers and that need a practical solution for the simple problem that all these people before me have explained so well. As soon as this feature is available, there will be many people that will start using this fantastic fork of KeePass/X/....

Thanks a lot anyway for a fantastic work so far!

[droidmonkey]

@ocumo how are you syncing databases between your phone and computer? Are you using webDAV or some other similar technique? I simply use cloud file storage (One Drive, Google, Dropbox, etc) and that works 100% of the time. Our merge algorithm is very robust.

[akontsevich]

droidmonkey added this to the v2.6.0 milestone 3 minutes ago

Thanks a lot!!!

[ocumo]

@droidmonkey, I do not have a satisfactory solution yet for my phone, although I have been using keepass2Android, but I simply do the silly way: many times I just copy the kdbx file to the phone and though that's one-way only, I mostly have a "phone is read only" silly policy. Other times I have sync with the KeePass2 but that requires a bit of command line work, so it's faster (though dumber) the other way. But I should say that the phone is not a priority for me so far, as long as I find a consistent solution first for all PCs at home.

[ocumo]

By the way, thank you so much for the milestone!

[phoerious]

KeePass2Android can open directly from WebDAV, Dropbox, etc. Changes made to the database on the phone are immediately synced back to your cloud storage. Haven't had any issues with that.

[srguglielmo]

I just wanted to add my two cents. Up until a few weeks ago, I used KeePass 2 (the original project) across Windows, Mac (via mono), and Linux (via mono). All three on an almost daily basis. When MacOS removed support for 32-bit applications (required to run KeePass via mono) recently, I discovered KeePassXC. I love that it runs natively. However, switching to it not only broke my syncing workflow, but makes it very difficult to convince lesser-technical people (like my family and friends) to use.

For my syncing workflow, I have a separate kdbx file stored on each of my devices, plus one in a cloud storage account. The file in cloud storage is my "source of truth". My sync process is:

• Open the local file (e.g. macbook.kdbx) in KeePass.
• Download cloudStorage.kdbx.
• Perform a sync.
• Upload cloudStorage.kdbx (overwriting the original).

Sometimes, I make a change to Site A on my MacBook and a change on Site B from my Linux desktop. I want to keep both changes. I can't do that anymore without opening one file, running a merge, opening the other file, running another merge.

We are not aiming to be EXACTLY like KeePass.

I understand, but two-way sync simply makes sense here. It's not "trying to be like KeePass", it's just being user friendly. It isn't logical to default to one-way sync without a big red bold warning about potential data loss.

Thank you for re-opening and re-evaluating this request!

[droidmonkey]

To be clear, macbook.kdbx == cloudStorage.kdbx? If so, why go through complications and just have one file? Whatever mainstream cloud storage solution you are using is bound to have version integrity.

You mention lesser technical users, but your sync setup is very complex (relatively speaking). Whereby if you just dropped your kdbx into iCloud or Dropbox you'd be 100% functional with KeePassXC.

What I am trying to say is that the process that people are using bi-directional merge/sync for is what is broken. In almost all cases that have been presented to us it is easily solved with a modern process.

[phoerious]

I am using and modifying my database on multiple devices and I am rarely or almost never experiencing conflicts. And when I do, I can simply merge them together automatically using KPXC's merge feature.

Since autoreload and automerge is enabled by default, the only way you can get conflicts is when two users modify the database in rapid succession (i.e. faster than Dropbox can propagate the changes). That can happen and it happens more often the more users you have, but 60 conflict files seems extreme. Check your settings.

[droidmonkey]

We added a "Save Database Backup" feature in 2.6.0 that would satisfy @firepainting scenario. Since he said he never opens the database anywhere else, just saves them to various locations, a sync is not necessary. We decided not to support triggers or scripting for security reasons. You can do all that outside keepassxc with bash.

[phoerious]

I think simple triggers were this sync scheme with four database copies, which doesn't need any scripting. That could be implemented without concerns for security.

[roberestarkk]

@droidmonkey firepainting actually says they save but don't open it on their phone, so it would work for that, but they have two linux machines running KeePass (which I therefore presume they open it on) and have had cloud save conflicts in the past.
https://i.imgur.com/DqsMjte.png

@phoerious I doubt that anything I can do settings-wise will help me.
The problem comes when the Dropbox client just flat out stops syncing after a while for no reason, or when my Australian potato internet starts syncing and then doesn't finish by the time I finish using my PC and turn it off, or a storm comes through and the internet goes away entirely, or any number of other problems that can happen because the internet is held together with string, duct tape, and chewed gum.

KeePassXC can't guarantee that my cloud sync provider will sync the DB file properly, completely, and quickly each and every time. Nor can it guarantee that it will always be running and have the file open to be able to merge it when the cloud sync app detects remote changes.
So I instead rely on KeePassXC to be able to periodically automatically bi-directionally sync a database file with another copy of itself like KeePass2 can.
This lets me totally and cleanly decouple KeePassXC from the mechanism for synchronising the database file, in a way that guarantees eventual lossless consistency between all instances of the database across the entire cloud without requiring any manual intervention, or any manual merging of conflict files (assuming the cloud provider can be trusted to always generate them).

[firepainting]

Hi again and thanks so much for the replies.

@roberestarkk

I sync databases mainly for backup, and I back up in as many places as I can. I use 5 cloud providers, all with the ability to sync with my 2 PCs, and decided to use them all as part of my backup system for my KeePass db because why not?

My current setup is that whenever I "save" my master db, the other four sync automatically.

I always thought it would be very reasonable (borderline a must) to have at least one backup and that that backup must be updated in real time once the master db gets updated. This means there must be at least 2 dbs in total that are automatically synced with each other. It might have been confirmation bias, but when I researched about "the need to sync" and "how to sync" etc., I thought many people needed the sync feature especially as a backup solution. It was a basic solution if you cared enough about your passwords.

This part I realize is where I may be mistaken now, because the lack of the feature up to now might mean that not that many people actually need it (but how do we know?), or that KPXC really is only for technical people, which I don't consider myself one.

If I'm following you, the devs thought there wasn't a need to sync another database because, in the case of multiple machines, the user only needed to open the db from the cloud server itself, like Dropbox? That it wasn't thought of as a simple backup solution? (Just to be clear, it's not my only backup solution, but syncing dbs is a huge part of it and especially gives me peace of mind.)

If that's the case, then I'm sure it is highly appreciated that the need is now acknowledged.

In terms of syncing and conflicted files, luckily Dropbox has been 100% perfect for me, in my 5+ years of using it. There was that one time when their servers seemed down (maybe location-specific) for a couple hours but no conflicted files ever, even on the times our internet's gone potato, too.

@droidmonkey

Yes I open my databases on my 2 PCs only--I just wanted to show how simple my setup is. And also wanted to point out that I don't trust only one cloud service provider due to my past experiences with conflicted files. For context, I work on stuff directly from Dropbox daily because of the need to sync my 2 PCs. It is backwards but it's what works for me atm. Anyway, it's just about creating multiple backups--the more cloud providers I could sync my dbs in, the better. If one gets corrupted, I have other copies.

I think it's worth noting that I've never had corrupted dbs, though. Conflicted dbs haven't given me any problems. KP2 would just create a duplicate copy--the conflicted db--but the main backup was still intact. I just had to delete the conflicted db. It's annoying but I find KP2 to be really stable that way. I understand that these conflicts could be attributed mainly to the cloud providers themselves.

Edit: Adding the issue I opened last year, #3768, which describes in detail my setup and use case for sync triggers.

[phoerious]

I use Nextcloud to sync my DBs, but the mechanism is the same for any versioned cloud sync (that includes Dropbox). I have the database synced to multiple (n) devices, so together with the Nextcloud server, it is mirrored to n+1 devices. Not all copies are fully up to date all the time until I have used every device after a change, but it's good enough to prevent a total loss (and I don't create new accounts every day, so changes are rather rare). The file itself is version-controlled by Nextcloud, so if I ever need to go back to an older version (never had the need to so far), I can---even if the changes are already propagated to all devices.
In addition, I include the Nextcloud sync directory of at least my main device in my regular incremental NAS backup, which keeps a history as well. All in all, I would say the risk of data loss is minimal with so many copies of which at least one is off-site.

[dssouza-ti]

Well, in a work environment where a team needs to use the same database, and all of them are open and being updated all the time, this sync feature is the only thing that works. Keeshare is useless in this case, as it can't share group structure and dropbox, google drive, onedrive and every cloud storage solution are also useless. For me, it doesn't matter if the user is a "technical person" or not, this is a very simple feature and one of the most useful on keepass2. It's also the only reason me, my team and at least 15 other users I konw still keep using keepass2, even though keepassxc is better in every ohter way. I also fail to figure out any logical reason for why this feature keeps being pushed forward and not implemented right away.

[roberestarkk]

Just taking a step back here...
@phoerious are you guys convinced of the need to provide an automatic periodic bi-directional sync with another local copy of the db file?
ie: Every x seconds, perform a bi-directional sync with the currently opened DB file and the DB file at filesystem location Y, without requiring credential re-entry each time.

If you are, then we don't need to keep trying to convince you and we can just wait patiently for the implementation stage.
If you are not, or are but only partially (eg: bi-directional and automatic but not periodic), please let us know because we really need you guys to be convinced, and we're happy to keep trying until then!

[phoerious]

We know it's a requested feature and will implement parts of it when we find the time. We want to avoid implementing the synchronisation itself, because it adds unnecessary complexity for a feature which external tools can do much better than we ever could.

[Odeen]

We know it's a requested feature and will implement parts of it when we find the time. We want to avoid implementing the synchronisation itself, because it adds unnecessary complexity for a feature which external tools can do much better than we ever could.

We're talking about a master-master-...-master DB scheme and not a master-slave scheme.
@phoerious Exactly which external tool can merge 2 existing DBs without data loss?

[phoerious]

KeePassXC can merge databases and with a temporary buffer database, you can sync without data loss over Dropbox. The latter part is the only thing missing at the moment. Please read the trigger spec.

[Odeen]

Is there already a bidirectional merge? Last time a tried it was not bidirectional....
I have to use Dropbox for a db sync? Really? How does it adapt to a master-master-master DB scenario?
If I have 3 PCs with 3 local DBs and a remote webdav copy. No dropbox is involved here.

[phoerious]

Yes, KeePassXC can merge bidirectionally and again, please read the trigger spec. It's linked a few posts above.

[Dannniello]

Merge it is not sync.

KeePassXC open db1 and merge with db2. Effect: db1 will have added all content of db2. db2 will not change at all.

KeePass open db1 and sync it with db2. Effect: db1 and db2 will be perfectly synced (file checksum of db1 and db2 will be the same).

Also sync is not directly related to Dropbox or other company file sharing service. It is only one of use cases... For example I do not want give my private password database to some big data harvesting companies like Google or Dropbox, but I understand that for most users it is OK. But I still need sync function (so I still use KeePass)...

[Odeen]

Yes, KeePassXC can merge bidirectionally and again, please read the trigger spec. It's linked a few posts above.

searched it but couldn't find it.... can you please link it to me?

[Odeen]

Merge it is not sync.

KeePassXC open db1 and merge with db2. Effect: db1 will have added all content of db2. db2 will not change at all.

KeePass open db1 and sync it with db2. Effect: db1 and db2 will be perfectly synced (file checksum of db1 and db2 will be the same).

Also sync is not directly related to Dropbox or other company file sharing service. It is only one of use cases... For example I do not want give my private password database to some big data harvesting companies like Google or Dropbox, but I understand that for most users it is OK. But I still need sync function (so I still use KeePass)...

I'm talking exactly about this.... Actually I don't even get why the "KeepassXC style merge" is usefull to anyone...
I'm talking about Keepass's synchronize command.

Why is this so hard to explain/get?

[phoerious]

Again, read the spec instead of complaining. And if you don't like Dropbox, use Nextcloud.

[Dannniello]

I see that some comments are not very helpful, so maybe I will comment "as it is": KeePassXC 2.6.2 do not have sync function, so there is no need to read spec and/or source code.

Merge probably is useful for someone, but not for everyone. It is the same with sync - for some it is probably completely redundant and unnecessary function. Why? Apparently their "password workflow" do not need it...

Sync function was not forked and probably will not be implemented if developers do not see the point of doing it... Especially that original sync function is probably much more complicated that it sounds: "The synchronization algorithm is rather complex and it would take many pages to describe in detail how it's working." (quote from https://keepass.info/help/v2/sync.html).

Good news is that KeePassXC and KeePass databases are still compatible, so for sync/edit/etc. database tasks - I'm using KeePass. For view and Auto-Type -> I'm using KeePassXC.

PS. KeePassXC developers done excellent job with properly adopt "Windows centric" KeePass to other systems (Linux especially:). Very thanks for it!

[phoerious]

https://keepass.info/help/kb/trigger_examples.html#dbsync