Can't delete secret directories (using the generic backend)

[brocoli]

Using vault delete path/to/secret I can delete secrets from the fault server, but then doing vault list path/, it will still list the "to" directory, even if it is empty.

I can't find any way to get rid of these directories and it's starting to clutter our vault server.

[jefferai]

Are you using the file data storage backend?

[brocoli]

Yes, and I've noticed that deleting the empty folders directly in the backend works. Thanks!

[jefferai]

Hi @brocoli ,

That came up on the mailing list last week or so. I'm not entirely sure if it was coded this way on purpose or not -- my guess is that since requests to Vault are entirely async from one another, the author was nervous about one request seeing that a folder was empty and removing it and another request simultaneously trying to write a few value into it. There are probably ways around this race condition involving atomic filesystem operations -- except that any such solution has to not just be atomic on Linux and UNIX-y OSes but Windows and other less common OSes, too.

If you think you know of a suitable way to address this, PRs are welcome! In the meantime I was going to suggest that you can just remove directories yourself if you know they're not needed -- which you found out on your own. Glad it's working for you!

[stephanh]

I have the same problem with the etcd backend. We have empty folders that we can't delete using the Vault API.

[aizotov-aspiration]

We have the same problem with the DynamoDB backend

[souheilh]

Same problem with S3 backend

[netson]

Yes, and I've noticed that deleting the empty folders directly in the backend works. Thanks!

Can you elaborate on how you deleted the folders directly in the backend? I am also using the File backend and in the actual directory where the encrypted secrets are stored, there's no obvious relationship between an empty secret directory and the actual directories on the filesystem.

When I attempt to delete the path using Vault UI or the vault cli tools, the directory is not actually deleted.

I read in another thread that the paths may be kept on purpose to avoid race conditions with certain backends, but not being able to delete the paths at all seems messy, as when time passes, you will be left with a lot of clutter, since secrets come and go.

Using Vault 1.1.3

[netson]

Sorry, never mind, I found the "Delete secret" at the top of the secret listing in Vault UI! :-)

[maxadamo]

same problem with Consul backend.

[catsby]

Reopening this issue

[ggiill]

Same issue with Google Cloud Storage backend.

[time-less-ness]

I suspect this is just a +1 for gturetsky, but let me give a bit more technical detail.

If we delete keys, or even destroy keys, the keys themselves still remain, except that when you get them, they show as deleted or destroyed in the metadata.

They still remain in listings.

Of course, entire directories also remain there because all the keys inside them are still there.

We were testing things out and created a bunch of "delete_me" keys, and now we've got a couple dozen directories/keys crapping up our Vault installation. The keys/directories are very inconveniently located/named, so we really, REALLY want to get rid of them. And there are other keys/directories on the same KV engine that we want to keep as well.

We use the Google Cloud Storage (Postgres) back-end.

EDIT

Okay, we just discovered this command, which seems to fix all our problems.

vault kv metadata delete <key>

[aphorise]

Testing 1.4.1 or higher & using either consul, etcd or file empty paths seem to be removed when there are no more objects therein (used: vault list sys/raw/... to confirm before and after).

When dealing with secrets / auth engines - the best method to release and clean redundant paths (or sub paths therein) is by disabling them.

Alternatively if raw_storage_endpoint is enabled - then:

vault delete sys/raw/logical/…guid-kv…/…guid-path…/policy/metadata
vault delete sys/raw/logical/…guid-kv…/…guid-path…/archive/metadata
vault delete sys/raw/logical/…guid-kv…/…guid-path…/archive/salt
vault delete sys/raw/logical/…guid-kv…/…guid-path…/archive/upgrade
# // then there's no more guid of empty sub-path 

Also most stores offer a recursive deletion method (if needed) - such as:

consul kv delete -recurse vault/path/to/whatever
etcdctl del --prefix=true vault/path/to/whatever
# // …

In any case I do not think that this an issue any longer as was reported initially (maybe @brocoli can confirm?).

[mechdeveloper]

Same issue for me unable to remove sub directories in a secret-engine
OS Windows Server 2016
"file" storage backend for vault

[fede-r1c0]

Same problem with S3 backend

I had the same problem with the S3 backend, and the solution was very obvious and easy.
In the pod log, we saw:

error="failed to revoke token: failed to revoke entry: failed to delete lease entry: AccessDenied: Access Denied"

and the first result of the search gave us the answer that we were not seeing.

#6156

[mechaHarry]

Any clues on how to call destroy or metadata delete via golang api?
The currently available delete func only performs soft deletes, leaving paths behind in the UI list view for kv-v2.

The only way I have found to work is via ansible or python using hvac's method:
https://github.com/hvac/hvac/blob/9f29cad81b2fa3685030dff77b0d7f7642650368/hvac/api/secrets_engines/kv_v2.py#L356

Update:
Found sort of a workaround by creating a NewRequest with the metadata HTTP URL, and setting method as DELETE, then passing it into the RawRequest() func to get things going.

[Wolfsrudel]

How can we get this fixed? It is very annoying when you have empty folders lying around because you no longer need them, or even just had a typo that you can no longer fix.

[aphorise]

Which versions are you all using?

Also what about my earlier posting with full mount disabling and or direct storage deletion - there's also the 3rd DELETE method of: /secret/metadata/:path as also mentioned above by @mechaHarry.

Testing 1.4.1 or higher & using either consul, etcd or file empty paths seem to be removed when there are no more objects therein (used: vault list sys/raw/... to confirm before and after).

When dealing with secrets / auth engines - the best method to release and clean redundant paths (or sub paths therein) is by disabling them.

Alternatively if raw_storage_endpoint is enabled - then:

vault delete sys/raw/logical/…guid-kv…/…guid-path…/policy/metadata
vault delete sys/raw/logical/…guid-kv…/…guid-path…/archive/metadata
vault delete sys/raw/logical/…guid-kv…/…guid-path…/archive/salt
vault delete sys/raw/logical/…guid-kv…/…guid-path…/archive/upgrade
# // then there's no more guid of empty sub-path 

Also most stores offer a recursive deletion method (if needed) - such as:

consul kv delete -recurse vault/path/to/whatever
etcdctl del --prefix=true vault/path/to/whatever
# // …

In any case I do not think that this an issue any longer as was reported initially (maybe @brocoli can confirm?).

[ekhaydarov]

This is still true for azure backend. vault version 1.9.3

[aphorise]

I believe there should be some notes in the documetnaion relating to empty paths and raw_storage_endpoint approach that I've mentioned several times throughout this thread.

There really is no issue here - if there are too many redudant paths on any mount (logical guid) then the easiest approach to cleaning it is vault secrets disable ....

Alterantively having Vault remove empty paths each time that they occur (can be all the time too with single entry secret only) is as redudant as authoring them in the first place. As preventatives there's also the option of not using / character in paths at all.

[rajamohan-gemini]

Even i had this issue, i was not able to delete empty test directory. Receiving permission denied errors.

Code: 403. Errors:
1 error occurred:
permission denied

Below approach resolved my problem.

Path - mine/test
Vault version - 1.11.12
generic secret storage

1. Added metadata policy to the ACL.

        path "mine/metadata/*" {
        capabilities = ["create", "update", "read", "list", "delete"]
        }
   

2. This command deletes the entire directory vault kv metadata delete mine/test