User/password not valid right after creation

[lindsaygrace]

Continuation of this issue from the old repository mongodb/vault-plugin-secrets-mongodbatlas#14

We're having the same issue where we get an 'Authentication failed' for requests made shortly after we receive a credential from vault.
This is unusable for us in production if retrieved credentials aren't actually valid when we retrieve them. For now we've needed to revert to using static credentials.

[themantissa]

@lindsaygrace thank you for the input. The reason for this is credentials are applied at the project level and so depending on the number of clusters may take some time to apply but they are returned immediately. We have an endpoint to check completion on the apply on our roadmap, if we soon have the ability to wait to return the credentials till verification of the credentials being applied across all clusters would this unblock you?

[lindsaygrace]

@themantissa yep, that should allow us to be able to use this plugin

[themantissa]

@lindsaygrace we have added an API endpoint to allow polling for pending and applied status for clusters, which will verify if a DB user(s) has been added to each cluster (so you'd need to poll each cluster in the project in a script for example - https://docs.atlas.mongodb.com/reference/api/clusters-check-operation-status/). Hopefully we can build this into the secret engine going forward. cc @kalafut

[antiqe]

Hi @themantissa, do you have ETA for integrating this check operation status in the vault plugin himself ? It's blocking us too, i know we can still build an additional script checking the status before running the service but it's not so clean and it's involved some specific command to retrieve the username & password that has been regenerated. Thanks for your help

[themantissa]

@antiqe I don't at this time but will update when we do. fyi @kalafut if you have any input here. Thank you both!

[kalafut]

@themantissa Propagation delays are often a nuisance, and handling them in a reliable way can be tricky. For example:

• If we've successfully created credentials, but the cluster check API continues to return PENDING, what is the action?
• If timeout, how long do we wait? If we do timeout, is that a failed request from the Vault perspective, even though MongoDB might still create the creds? Do we go into a rollback action (and how do we know that succeeded)?
• If there is potentially a lag between "APPLIED" and actual usability, consumers might be still face issues. In other words, if there is eventual consistency at play at all, then consumers need to be prepared with retry logic.

That said, the fact that MongoDB has such a status endpoint is definitely way better than trying to determine, for example, the same on AWS. If the "APPLIED" state is a strong guarantee of availability and we're able to establish a reasonable timeout logic, I'd be open to this.

[jbenassi]

Adding myself to the request list - I understand it could be complex to expect a user to be immediately available? I do see the message status bar active while the new user is being deployed which I assume is using the status info. Are you still prioritizing the "operation status" feature integration to this plugin?

[jbenassi]

I have a work-around in place that simply uses a spinner to wait until a login is successful in a timed loop. If nothing else, it won't move on to show error with "Authentication Failed" until the user is fully created.

[themantissa]

Wanted to update those watching this issue. We have researched the options to address this feature request and discussed it with the Vault engineering team. After this it became apparent that there is no safe and solid way to do this and it would be better to continue to function as most secrete engines do, which results in an eventually consistent experience in regard to secret creation. I encourage those who need to know when the database credential is fully available to use the status endpoint mentioned earlier in this issue, https://docs.atlas.mongodb.com/reference/api/clusters-check-operation-status/).