Block and ask for unlock when applications query the secret service

[tleydxdy]

Expected Behavior

When app launch and the database is locked, keepassxc would popup and ask for unlock.

Current Behavior

When app launch and the database is locked, the app fails to get password, therefore asking user to enter it again (or ask to re-authenticate in case of nextcloud), keepassxc sit in silence.
Then, when the application trys to save it, keepassxc then ask user to unlock, and stores a duplicate entry.

Possible Solution

Make querying password acts the same as saving password in this case.

Steps to Reproduce

1. setup secret service
2. expose a database group
3. launch a application that uses secret service, and save a password
4. lock database
5. re-launch that application again

Context

It causes nextcloud to ask for auth everytime system boots up, and my mail client sometimes can't get password when keepassxc is locked.

Debug Info

KeePassXC - 2.5.3
Revision: f8c962b

Libraries:

• Qt 5.14.1
• libgcrypt 1.8.5

Operating system: Arch Linux
CPU architecture: Arch Linux
Kernel: 5.5.9-arch1-2

Enabled extensions:

• Auto-Type
• Browser Integration
• SSH Agent
• KeeShare (signed and unsigned sharing)
• YubiKey
• Secret Service Integration

[Aetf]

This is indeed one place where keepassxc is not fully adherent to the secret service spec. The spec requires the query operation to be functional even when the database is locked (i.e. the application can still search into various fields of entries without unlocking the database first). But it is technically impossible for keepassxc to do so. Because in keepassxc, once the database is locked, there is no database in memory at all. So keepassxc simply replies with empty results in this case.

I guess that's why the behavior you are seeing:

• for querying, the application assumes it can get results anyway so it doesn't try to unlock first.
• for saving, the application knows this operation has to be done when the database is unlocked so it makes a separate special blocking request to unlock the database first.

As you can see, it is up to the application to request unlock the database, and there is no direct way for keepassxc to insert blocking operation (pop up a dialog) into non-blocking operations (query/saving).

There are two ways to go around this:

• make the application be aware of the situation and always ensure the database is unlocked before any other operation. As an example, Chromium saves a dummy entry to ensure unlocking the database (link).
• find a way to (mis-)use the semantic of the query reply to trick the application into thinking there is something needs to be unlocked. I'll play with this option but it's likely that this will not work.

[tleydxdy]

is it no possible to not send a reply until the user has unlocked it?

[Aetf]

That's exactly what I want to avoid. DBUS reply times out in about 20s. The application will probably react much worse to that than not finding entry. While the timeout limit is configurable, it is not controlled by keepassxc.

And I think limiting the user action with a timeout will do more harm than good. Unlocking the database is not always as easy as typing in the password, there may be hardware keys, password files, etc, not to mention people who just happen to type slowly.

[tleydxdy]

It's certainly a hackish solution, but it does solve a problem. another workaround I could think of is to have some way to get my startup script to only start those applications after I unlocked keepassxc, and make keepassxc keep the secrets in memory/cached when it's locked again.

[Aetf]

As a workaround, if your startup script is after login, and you use PAM for authentication, you can have a script to unlock keepassxc while login/unlock screen using pam_exec. It does require the login password is the same as the database, though.

[Skycoder42]

@Aetf as far as I understand, the only problem is that keepassxc does not store the "attributes" separately to the database, so when being queried for them, it cannot return them in the locked items array.

As the spec states, attributes are not secret. So why not just keep a local, unencrypted copy of all attributes, that is update whenever the database gets unlocked or saved.

What do you think?

[Aetf]

Attributes in the spec are not the same as attributes in keepassxc database. It is just an implementation detail that the keepassxc database entry custom attributes are used to implement "attributes" as stated in the spec.

The custom attributes are part of the database. And I don't think it's a good idea to save them separately unencrypted, which could leak the presence of entries/groups in the database because there needs a way to associate attributes to entries.

[Skycoder42]

How about making it an optional thing? It could be a checkbox in the settings, with a warning, that enabling this would expose all non-hidden custom attribute fields of all entries in the selected group. This would prevent accidental leaks, but still improve usage for power users. I for example have a group used exclusively for secret service entries, so it would not be a problem for me.

[Skycoder42]

Is that not an option for you?

[Aetf]

I found something in the DBus protocol that may help the situation: The org.freedesktop.DBus.Error.InteractiveAuthorizationRequired error and the ALLOW_INTERACTIVE_AUTHORIZATION flag in the message header. spec

So in theory we could return this error for SearchItem requests on locked databases, and the client should retry the search with ALLOW_INTERACTIVE_AUTHORIZATION flag set in the message and prepare for a longer timeout for a reply. And we can then interact with the user to unlock the database during the second search request.

However in reality I highly doubt any client would respect the error and behave correctly, given the only working implementation of the API is gnome-keyring which can search a locked database...

[stop5]

I have read the secret-service spec and i think it can work this way:

1. an Program requests an Password
2. KeepassXC says that the Item/Collection is Locked
3. The Program calls the Unlock funktion and gets an Prompt-Object back. The Program must wait until the Signal Completed is fired
4. The User Unlocks the database
5. The Completed Signal is fired and in the output Result we have the object path of the unlocked item
6. The Program uses the Password

With an Prompt we don't have to worry about timeouts from DBus, since we answer all Requests in time.
This way the Issue #4733 can also be solved, since the specs explicit say that an granted access to an Item can be given to only one program. We would only need replace the Unlocking with an Window that asks if we would like to grant Program XY access to the password.

[Aetf]

That's already what keepassxc does: returning prompt object for apps to react on if the collection is locked, just not per item locking state, which is #4733 (and now #5747) tries to do.

Anyway, the problem is before the first step. The program usually starts with a search to get a path to the item it wants, and the spec expects us to be able to search even when the database is locked, and there is no way in the search API to return a similar prompt object.

[scruloose]

@Aetf I'm just starting out with D-Bus and Secret Service, and I wonder if you can help me understand the current situation and limitations with KeepassXC and Secret Service.

Suppose I want to write a very simple client (in Python, probably) specifically to retrieve a single password from KeepassXC via Secret Service and pass it along to the process that needs it. Ideally, I'd like my client to run in the background with no UI, no terminal window, no user interaction.

Given that I can accommodate whatever order KeepassXC wants things done, is it possible to open a session to KeepassXC, check whether the database is locked/ask KeepassXC to unlock the database, block while KeepassXC handles prompting the user and unlocking the database if it's not already unlocked, then retrieve the password (using the identifier attribute which is already known to the client script)? Am I right that all of the above will work as expected as long as I don't try to do a search without first ensuring the database is unlocked?

I've got as far as setting up a group for the purpose and, provided the database is already unlocked, successfully retrieving a password from a test entry in the secret service group with secret-tool lookup. I guess the fact that secret-tool search --unlock <attribute> <value> fails silently if the database is locked (and doesn't trigger KeepassXC to prompt the user to unlock it) is due to the limitation under discussion here. Hence the idea to try writing my own client.

[Aetf]

@scruloose Happy to know that you are working on a new client. A standard will not be of much use if there's only one implementation for either the client or the server ;)

What you describe is all possible. And you can open new issues to discuss if you find something should be possible in the spec but not implemented so in KeepassXC.

Suppose I want to write a very simple client (in Python, probably) specifically to retrieve a single password from KeepassXC via Secret Service and pass it along to the process that needs it. Ideally, I'd like my client to run in the background with no UI, no terminal window, no user interaction.

As long as your client share the same user dbus session with KeepassXC

Given that I can accommodate whatever order KeepassXC wants things done, is it possible to open a session to KeepassXC, check whether the database is locked/ask KeepassXC to unlock the database, block while KeepassXC handles prompting the user and unlocking the database if it's not already unlocked,

Once you ensure the database itself is unlocked, the ordering doesn't matter.

As for unlocking, it will soon become a little more complex than unlocking the database once #5747 merges. You need to be prepared to handle the per-item locking state. And don't assume the unlocked item will remain so, because the user may revoke the authorization. You should handle these anyway as that's what the spec implies.

Except for what discussed in this issue, there's nothing deviated from the spec, so as long as you follow the spec, there shouldn't be many limitations.

then retrieve the password (using the identifier attribute which is already known to the client script)?

If you want to directly construct the dbus object path from the identifier, please don't do that, as the path may change and the usage of entry UUID in the path is an implementation detail. It's fine if you get the object path by using the identifier attribute to perform a search first.

Am I right that all of the above will work as expected as long as I don't try to do a search without first ensuring the database is unlocked?

Correct.

I've got as far as setting up a group for the purpose and, provided the database is already unlocked, successfully retrieving a password from a test entry in the secret service group with secret-tool lookup. I guess the fact that secret-tool search --unlock <attribute> <value> fails silently if the database is locked (and doesn't trigger KeepassXC to prompt the user to unlock it) is due to the limitation under discussion here.

Correct.

[ccryx]

So maybe I should have read the whole thread before posting this, since it's all kind of been mentioned before.
As far as I can tell, none of the possible workarounds are viable for different reasons mentioned throughout the comments.

I'll leave the rest up as a summary...

Hi, so I have run into this issue as well and wanted to chime in.
As far as I can tell and as far as I understand the spec I see the following ways to work around this:

1. Promt the user to unlock the database and return search results whenever that finishes. Not ideal for the reason you mentioned, but one could argue that it's the client's responsibility to deal with timeouts.
2. Promt the user to unlock the database. Return search results if that completes within a configurable time, since the actual timeout can be implementation specific AFAICT. Use libdbus' _DBUS_DEFAULT_TIMEOUT_VALUE as default. If the unlocking doesn't complete in time, return an empty result. Make this process transparent to the user, maybe with a timer at the unlock promt and disabling the prompt when it runs out.
3. Store the lookup attributes unencrypted with the database. This would be the most secret-service compliant alternative but could require a change in the db spec (which seems pretty much out of the question since it would break compatibility) or storing additional loose files with the database and the metainfo of which lookup attributes file goes with which db. This may be undesirable due to privacy concerns as well.

[scruloose]

Hey, so I have an idea for a workaround that I think hasn't been proposed and I think may break the deadlock facing the list of approaches suggested so far. I am ... not much of a coder, so please somebody do tell me if my idea is naive or just plain wrong, or whatever.

Anyway, what I'm thinking is as a runtime thing, before re-locking an unlocked database, we stash all of its lookup attributes in memory.

Some thoughts:

1. I presume this should be a configurable option, and probably disabled by default, given that it has privacy/security implications — very low-risk ones as I understand things, but still. Opt-in is nice. I think a ticky-box in the "Secret Service Integration" Settings tab saying something like "keep Secret Service lookup attributes in memory when re-locking database" should be reasonably clear, at least to those who are looking for it.
2. If I understand things right, this would largely resemble option № 3 as summarized by @ccryx above, but without the need for changes to the DB.
3. Also, if I understand things right, this would go a long way toward mitigating the privacy/security concerns about option № 3, by reducing the availability of the lookup attributes from "on disk in the clear all the time" to "in memory only after the user has successfully unlocked the DB".
4. I think it makes sense, further, to combine this with either of the two "block and prompt" options as well, so for queries that come in before the user unlocks the DB for the first time in that session or instance or whatever the exact word is, then we block and prompt and maybe just wait indefinitely or maybe enforce a time-out of our own.
5. Personally, I think I prefer "wait indefinitely" as the fallback plan. I mean, ultimately the spec and API should support returning a prompt object to the client, but until such time as that happens... We make a good-faith effort to answer the query for real, and it's up to the client to decide how long to wait for our answer before timing out and giving up. Right? Also, I think my proposed approach lightens the load on that decision a bit, by narrowing the circumstances in which it will come up.

Does that make sense?

[Aetf]

Ultimately, blocking the search for unlocking is still needed, just as @scruloose said. Whether we cache the look up attributes in memory or on disk for later searches is a minor issue in comparison.

Having thought about this again, I now feel that it may not be that bad to block, though. There are actually bugs (that I plan to fix but haven't got time to) in the prompt object handling code, which will block the call on prompt. And it seems no one has noticed or complained about that yet.

Now I'm considering implementing this (try unlock before search). A timeout search is arguably better than a search succeeded but returning nothing. I'll give this a try and see if it works will in practice.

[droidmonkey]

Makes perfect sense to me, I like your synopsis

[droidmonkey]

BTW @Aetf while you are here. We should document in our user guide how to use secret service integration and how to disable the inbuilt gnome or whatever managers. If you go for this last major change it would be good to include the documentation and acreenshots as well.

[michaelk83]

I'd like to add my vote in favor of the InteractiveAuthorizationRequired proposal. I think this is the correct approach to how this API should behave.

However, I agree that it doesn't solve the immediate problem for existing applications, so in the mean time we need some other solution such as proposed by @scruloose . But I do think this should be added to the Secret Service spec. The spec is still a "version 0.2 draft", so very much a work-in-progress (though apparently it hasn't been updated since 2011). As I noted elsewhere, it may be worth reaching out to Stef Walter (libsecret maintainer and Secret Service spec co-author) on this and other Secret Service concerns. Or at least engage the current libsecret developers over at GitLab. Nothing is going to change in the spec if they aren't aware of the problems.

One final thought: if KPXC enforces its own timeout for a blocking search attempt, and then falls back to an empty response, then return InteractiveAuthorizationRequired with the empty response.

[michaelk83]

On further thought, most applications that talk to Secret Service should be doing that via libsecret. AFAICT, libsecret can handle the InteractiveAuthorizationRequired internally in secret_password_lookup() and its variants before calling the client application's callback (or before returning, for the synchronous variants). Or, it can just set the ALLOW_INTERACTIVE_AUTHORIZATION flag from the start. Either way, to the client app this still looks like the same API, it may just take longer until the callback is called (or sync function returns). That should solve the issue for every app that uses the latest libsecret.

The other possible target is QtKeyChain. GNOME/Gtk users are far more likely to use gnome-keyring-daemon as the Secret Service backend, in which case, this doesn't affect them anyway. KDE applications are supposed to be migrating to QtKeyChain, rather then using libsecret directly (let alone DBus calls). Other Qt apps should use it as well. QtKeyChain can either handle the InteractiveAuthorizationRequired internally in a similar manner, set the ALLOW_INTERACTIVE_AUTHORIZATION from the start, or always send an unlock request when a password is needed.

It may actually be ok to leave this broken for KDE apps that haven't migrated to QtKeyChain. That should encourage them further to migrate. So I submit an updated proposal:

1. If the DB is locked, return an empty response with InteractiveAuthorizationRequired error (non-blocking). Don't attempt to unlock.
2. With an ALLOW_INTERACTIVE_AUTHORIZATION flag, show the unlock prompt (blocking, but only blocks the library's asynchronous process, not the client app).
3. Have libsecret make the above adjustments. (Either ask them to do it, or submit a PR. There is already a relevant issue).
4. Optionally, have QtKeyChain make similar adjustments. (Either ask them to do it, or submit a PR. The changes to either library should be pretty small.) This becomes mandatory if libsecret refuse the fix.
5. Let the libsecret devs (and/or Stef Walter specifically) know about the situation, and pursue an update to the Secret Service spec.
6. For KDE apps that haven't migrated to QtKeyChain, or until the fixes are deployed, the workaround is to unlock the DB manually (or via a script/etc), and remind the app devs to migrate to QtKeyChain.

This is a more holistic approach, taking into account the bigger picture outside KPXC. The advantage for KPXC is less new code and nearly zero security impact (no extra unencrypted DB schemas or attributes need to be kept anywhere).

If we really want to, we can still add @scruloose 's solution with a checkbox "Workaround for ...", off by default. In that case, the difference is just the extra checkbox and the bits outside KPXC. But if libsecret accepts the fix, this shouldn't be necessary.

@Aetf , @droidmonkey , thoughts? (sorry this is a little long)

[Aetf]

Thinking about the bigger picture is good.

The InteractiveAuthorizationRequired is essentially a hack to solve the problem within the current spec. But if we were going this far, fixing the spec makes more sense to me, given more or less it's the same set of people behind libsecret and the spec.

Another benefit of fixing the spec is that the search API's asynchronousness can be well expressed within the prompt concept already in the spec. So no need to introduce a completely different set of async mechanisms into the spec.

For example, there could be a new SearchEx that returns a prompt rather than results. (Just some rambling, I didn't give too much thought on this).

And we can bring this up together with #5747 (comment) given the current spec is still a "draft".

[Aetf]

most applications that talk to Secret Service should be doing that via libsecret

In addition to libsecret and QtKeychain, I think python SecretStorage represents another set of users.

[michaelk83]

The InteractiveAuthorizationRequired is essentially a hack to solve the problem within the current spec.

I think this is one place where the implementation(s) will have to lead the spec, rather than follow it. There's nothing like practical experience to iron things out.

Fixing the spec is ideal, but it will take time, even after posting to https://gitlab.freedesktop.org/xdg/xdg-specs/-/issues . There needs to be some back-and-forth with libsecret, and so on. Plus, it hasn't been updated in a long time, and its main authors have likely moved on to other projects. But posting to xdg-specs is the first step. Even if we gain nothing else, that would at least document the issues.

Specifically re InteractiveAuthorizationRequired, it seems like a sensible (and rather simple) thing to add, considering it's part of the official DBus protocol.

In addition to libsecret and QtKeychain, I think python SecretStorage represents another set of users.

QtKeychain uses libsecret internally (according to their readme). But it looks like SecretStorage uses DBus directly. :-/

[michaelk83]

At least some python clients may be using the libsecret bindings.

[michaelk83]

Reading the spec some more. In chapter 3: Collection and Items, it says:

An item or a collection may be initially in a locked state. When in a locked state the item or collection may not be modified in any way, and the secret may not be read. Client applications that require access to the secret of a locked item, or desire to modify a locked item, must unlock it before use.

The service must prevent modification of locked collections or items. On such an invalid access the IsLocked error should be raised.

It's a little ambiguous, but "invalid access" can be interpreted to include lookup of a locked item (the previous sentence can be taken as an example, rather than a complete list of all invalid access types). So I think returning org.freedesktop.Secret.Error.IsLocked is the correct way to go, and falls within the existing spec. If that can be combined with InteractiveAuthorizationRequired per standard DBus protocol (returning both simultaneously), even better.

Furthermore, in chapter 5: Lookup Attributes, it says:

Services implementing this API will probably store attributes in an unencrypted manner in order to support simple and effecient lookups. (emphasis added)

IOW, it expects the lookup attributes to be unencrypted, but does not guarantee or require it. So again, having them encrypted (and therefore returning IsLocked) is within the spec.

[Aetf]

Returning IsLocked on Collection.SearchItems is probably ok, because the collection itself can be locked.

But Service.SearchItems explicitly returns both locked and unlocked items. So IMHO samentically it's not appropriate to return IsLocked if we stick to the current signature. And many clients don't expect to handle IsLocked error when calling SearchItems on Service.

[michaelk83]

For Service.SearchItems it says:

locked: Items found that require authentication. (emphasis added)

So this can be treated as 3 cases:

1. Service (Collection) is locked, so search is not possible: return IsLocked error with empty arrays. This at least tells the client why the arrays are empty.
2. Service is unlocked, but specific items are locked: return those in the locked array. No error.
3. Everything is unlocked: return everything (that matches the search) in the unlocked array.

I agree this is a little wacky, but I think that's the best choice we have. And this does fall within "will probably [but not definitely] store attributes in an unencrypted manner" from chapter 5.

If IsLocked can be returned together with InteractiveAuthorizationRequired, that would be better, since that tells a more complete story of the situation.

many clients don't expect to handle IsLocked error when calling SearchItems on Service.

As we discussed earlier, most of them will be using libsecret, which can handle this internally (and I think, should handle it).