Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KV Engine: Recursively list keys #5275

Open
vfauth opened this issue Sep 5, 2018 · 47 comments
Open

KV Engine: Recursively list keys #5275

vfauth opened this issue Sep 5, 2018 · 47 comments
Labels
community-sentiment Tracking high-profile issues from the community enhancement feature-request secret/kv

Comments

@vfauth
Copy link

vfauth commented Sep 5, 2018

Is your feature request related to a problem? Please describe.
With a KV engine, if I want to list all keys in the directory /foo/, it only returns keys directly under /foo/
For example, if I have the following keys:

/foo/some_key
/foo/bar/some_other_key

A LIST operation on /foo/ returns some_key and bar/, while I would like to have some_key and bar/some_other_key

Describe the solution you'd like
Add a parameter to either recursively return ALL keys in the provided path.

Describe alternatives you've considered
Another way to do it would be to add a parameter specifying the depth up to which look recursively for keys.

@extrail
Copy link

extrail commented Nov 27, 2018

This is a very useful feature, especially for the HTTP API

clickyotomy added a commit to clickyotomy/vault that referenced this issue Mar 23, 2019
Adds `-recursive', `-depth', `-filter' and `-concurrent' flags to the
vault CLI's `kv list' subcommand.

Description of the flags:
    * recursive     Recursively list data for a given path.
    * depth         Specifies the depth for recursive listing.
    * filter        Specifies a regular expression for filtering paths.
    * concurrent    Specifies the number of concurrent recursions to run.

Reference: hashicorp#5275.
@agaudreault
Copy link

agaudreault commented Jul 5, 2019

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine
./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

@QWERTY92009
Copy link

Also interested in this kind of feature.

@kir4h
Copy link

kir4h commented Jul 22, 2020

For anyone ending up here, I created a small cli to perform recursive kv read/list operations while we wait for the native solution.
Not very tested yet, I will be fixing bugs as they show up.

@antonu17
Copy link

@agaudreault-jive, thanks!
Btw, local readonly path="$1" doesn't work as you might expect. use local -r path="$1" https://stackoverflow.com/a/45409823

@vishalnayak
Copy link
Member

Issues that are not reproducible and/or have not had any interaction for a long time are stale issues. Sometimes even the valid issues remain stale lacking traction either by the maintainers or the community. In order to provide faster responses and better engagement with the community, we strive to keep the issue tracker clean and the issue count low. In this regard, our current policy is to close stale issues after 30 days. If a feature request is being closed, it means that it is not on the product roadmap. Closed issues will still be indexed and available for future viewers. If users feel that the issue is still relevant but is wrongly closed, we encourage reopening them.

Please refer to our contributing guidelines for details on issue lifecycle.

@kalafut kalafut added tmp/rt and removed tmp/rt labels Jun 25, 2021
@kalafut kalafut reopened this Jun 25, 2021
@dondrich
Copy link

Would love this feature

@rnsc
Copy link

rnsc commented Aug 25, 2021

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

var secretListPath []string

func isNil(v interface{}) bool {
	return v == nil || (reflect.ValueOf(v).Kind() == reflect.Ptr && reflect.ValueOf(v).IsNil())
}

// ListSecret returns a list of secrets from Vault
func ListSecret(vaultCli *vault.Client, path string) (*vault.Secret, error) {
	secret, err := vaultCli.Logical().List(path)
	if err != nil {
		log.Println("Couldn't list from the Vault.")
	}

	if isNil(secret) {
		log.Printf("Couldn't list %s from the Vault.", path)
	}
	return secret, err
}

// RecursiveListSecret returns a list of secrets paths from Vault
func RecursiveListSecret(vaultCli *vault.Client, path string) []string {
	secretList, err := ListSecret(vaultCli, path)
	if err == nil && secretList != nil {
		for _, secret := range secretList.Data["keys"].([]interface{}) {
			if strings.HasSuffix(secret.(string), "/") {
				RecursiveListSecret(vaultCli, path+secret.(string))
			} else {
				secretListPath = append([]string{strings.Replace(path, "metadata", "data", -1) + secret.(string)}, secretListPath...)
			}
		}
	}
	return secretListPath
}

@kir4h
Copy link

kir4h commented Aug 25, 2021

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

@rnsc Just in case you didn't notice my comment above and it can help, I wrote a small go based cli for this (#5275 (comment))

@heatherezell heatherezell added the community-sentiment Tracking high-profile issues from the community label Oct 12, 2021
@colinjcahill
Copy link

Adding my vote for this feature please.

@mechaHarry
Copy link

mechaHarry commented Feb 18, 2022

Ditto!
Essentially looking for some kind of vault-grep to find out any secret/engine/etc. that has some key, value, or pathname that matches a desired string.

@VWRalf
Copy link

VWRalf commented May 4, 2022

One more vote for search

@jsalatiel
Copy link

please implement this. It has been 4 years!

@tguvdamm
Copy link

This is a must have, listing all secrets now is a major issue.

@maxadamo
Copy link

if 103 people upvoted this feature request, maybe it's about time to start adding this feature 😸
For instance, Consul lists all the keys recursively, and Redis does it by default (using: KEYS *).
And whilst the solution from @kir4h works pretty well, it would be nice to expose this feature through the API, and libraries for any language can make use of this functionality.

@Sharkzeng
Copy link

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

@ndlanier
Copy link

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!
./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

This script doesn't work if you have spaces in the path name. Even with passing the argument with double quotes.

@DE110283
Copy link

Adding my vote for this feature please.

@NicolasTobias
Copy link

me too, seems Hashiguys doesnt want to

@finnstech
Copy link
Contributor

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

  • What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?
  • What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?
  • What if I have LIST or READ access to a secret within a path but not the parents within that path?
    Thanks!

@vfauth
Copy link
Author

vfauth commented Mar 15, 2023

Good news! For your question, I would say:

  • 404, to give no information about the key existence
  • 404, to give no information about the key existence
  • 200, because the access to this specific key has been allowed

@ndlanier
Copy link

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

  • What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?
  • What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?
  • What if I have LIST or READ access to a secret within a path but not the parents within that path?
    Thanks!
  • I agree with vfauth's thinking, give does not exist or permission denied, best to not let someone assume a secret or path exists if they don't have permission.
  • Same as above
  • Display the secret as the root path, or don't display any path information at all.

@rnsc
Copy link

rnsc commented Mar 15, 2023

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

  • What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

  • What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

  • What if I have LIST or READ access to a secret within a path but not the parents within that path?

Thanks!

For me it should silently not return anything that it cannot list or read in the sub tree.
Now of course if the given path cannot even be listed, a 404 would be good. If LIST is ok but no secrets nor any sub path can be read a 404 or a 201 empty response would be appropriate imho.

If you want to recurse from a top level path and you only have read access to a secret from two levels down I'd argue returning a 201 would be the correct behaviour. As it wouldn't show up in the UI either would it?

That's just how I would implement this for me though with the predicate that people know that recursively iterating over a path that you don't have the appropriate ACL for your token set to do is a user error and that given how this works we cannot hold the hands of the user too much without implementing insane logic.

@joshsleeper
Copy link

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

* What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

* What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

* What if I have LIST or READ access to a secret within a path but not the parents within that path?
  Thanks!

I think I disagree with some of the suggestions above, mainly because the topic of this issue is about recursively LISTing keys, not READing the secret data at those keys, right?

  1. keys that you can LIST but can't READ should absolutely present in the recursive list output.
    a simple label/indicator for keys that you don't have READ access to could be a nice enhancement, but imo the purpose of the recursive list command isn't to precisely enumerate all of your access permissions, but simply get an easy overview of the path/key structure.
  2. for a sub-path/key where a user lacks LIST permissions I wouldn't expect anything under that sub-path to be present in the output, full stop.
    even if the user (strangely) has other permissions on the sub-path/key like READ or WRITE, the focus here is on recursively listing keys, and if LIST is restricted, it's seemingly not something the admins want the user to be able to enumerate.
    from a user-friendliness standpoint, I would like to see the output inform the user that the unlistable sub-path/key exists but the user lacks LIST permission to enumerate the path further.
  3. this is a great question, and imo the answer is an error should be promptly returned by vault
    a recursive list logically can't start at a root the user doesn't have LIST access to, so attempting to do so at a path where LIST isn't permitted should return a nice simple error informing the user they don't have LIST permissions at that path.

@tysonkamp
Copy link

@finnstech This is great news. Is there any low/high confidence guess about when such functionality woudl be released? This quarter, this year?

@staehler
Copy link

staehler commented Jul 8, 2023

Is there any progress on this issue? I'm keen on using this feature.

@bitroniq
Copy link

Also interested in this feature

@bastian-schlueter
Copy link

Same here.
We had to super awkwardly implement width-first-search with limited depth in Terraform to be able to retrieve all the paths of secrets of a sub-tree.

@mitraed
Copy link

mitraed commented Aug 9, 2023

very interested in this feature

@ramizpolic
Copy link

Any updates on this? This would be a highly beneficial feature for our bank-vaults project.

@oyvhvi
Copy link

oyvhvi commented Aug 17, 2023

How can you not have search yet?

@bitroniq
Copy link

How can you not have search yet?

@oyvhvi exactly!

At the moment we are implementing our own vault-cli with all workarounds due to this missing feature.

@xraystyle
Copy link

Just adding another vote for this. I know search in Vault has been a highly contentious topic for literally years at this point, but even some basic search functionality in the KV engine would be incredibly beneficial. As a mitigation for all the performance concerns that are always brought up, how about making it an option one could enable or disable on a given KV engine?

@alalkamys
Copy link

alalkamys commented Nov 3, 2023

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

I built on your code, I had a use case to export all the KV secrets for a given path and save it locally in a secrets.json file. I refactored it to export the secrets rather than list them. My Vault was sitting behind Cloudflare Zero trust, I adjusted the script to optionally pass CF_TOKEN to by pass Cloudflare as well, thought people might find this helpful for their use case:

#!/usr/bin/env bash

# vault-kv-export.sh

set -eo pipefail

readonly ARB_TEMP_SECRETS_FILE="arbitrary_temp_secrets.json"
readonly TEMP_SECRETS_FILE="temp_secrets.json"
readonly SECRETS_FILE="secrets.json"

log() {
    local log_type="$1"
    local message="$2"
    local timestamp
    timestamp=$(date +"%Y-%m-%d %H:%M:%S")
    echo "[$log_type] [$timestamp] $message"
}

log_info() {
    log "INFO" "$1"
}

log_error() {
    log "ERROR" "$1"
    exit 1
}

traverse() {
    local path="$1"
    local result

    local headers=()
    if [[ -n "${CF_TOKEN}" ]]; then
        headers+=("-header" "cf-access-token=${CF_TOKEN}")
    fi

    result=$(vault kv list -format=json "${headers[@]}" "${path}" 2>&1) || log_error "Failed to list secrets: ${result}"

    while IFS= read -r secret; do
        if [[ "${secret}" == */ ]]; then
            traverse "${path}${secret}"
        else
            local secret_data
            secret_data=$(vault kv get -format=json "${headers[@]}" "${path}${secret}" | jq -r '.data') || log_error "Failed to get secret data: ${secret_data}"

            if [[ "${secret_data}" != "null" ]]; then
                echo "{\"path\":\"${path}${secret}\",\"value\":{\"data\":${secret_data}}}," >>"${ARB_TEMP_SECRETS_FILE}"
            fi
        fi
    done < <(echo "${result}" | jq -r '.[]')
}

main() {
    log_info "Starting secrets retrieval process."

    [[ -f "${ARB_TEMP_SECRETS_FILE}" ]] && rm -f "${ARB_TEMP_SECRETS_FILE}"
    [[ -f "${TEMP_SECRETS_FILE}" ]] && rm -f "${TEMP_SECRETS_FILE}"
    [[ -f "${SECRETS_FILE}" ]] && rm -f "${SECRETS_FILE}"

    if [[ -n "${CF_TOKEN}" ]]; then
        log_info "CF_TOKEN detected."
    else
        log_info "CF_TOKEN not provided. Headers will not be attached to Vault requests."
    fi

    local vaults
    if [[ "$1" ]]; then
        vaults=("${1%"/"}/")
        log_info "Retrieving all secrets under ${vaults[*]}.."
    else
        local headers=()
        if [[ -n "${CF_TOKEN}" ]]; then
            headers+=("-header" "cf-access-token=${CF_TOKEN}")
        fi
        log_info "No secret engine provided. Retrieving all secrets.."
        result=$(vault secrets list -format=json "${headers[@]}" 2>&1) || log_error "Failed to list secrets engines: ${result}"
        mapfile -t vaults < <(echo "${result}" | jq -r 'to_entries[] | select(.value.type=="kv") | .key')
    fi

    for vault in "${vaults[@]}"; do
        traverse "${vault}"
    done

    echo "[" >"${TEMP_SECRETS_FILE}"
    sed '$s/,$//' "${ARB_TEMP_SECRETS_FILE}" >>"${TEMP_SECRETS_FILE}"
    echo "]" >>"${TEMP_SECRETS_FILE}"

    jq . "${TEMP_SECRETS_FILE}" >"${SECRETS_FILE}"
    rm "${ARB_TEMP_SECRETS_FILE}" "${TEMP_SECRETS_FILE}"

    log_info "Secrets retrieval completed and saved to ${SECRETS_FILE}"
}

[[ "$0" == "${BASH_SOURCE[0]}" ]] && main "$@"

a sample output:

$ ./vault-kv-export.sh secret2/confluent_cloud

[INFO] [2023-11-03 15:06:15] Starting secrets retrieval process.
[INFO] [2023-11-03 15:06:15] CF_TOKEN detected.
[INFO] [2023-11-03 15:06:15] Retrieving all secrets under secret2/confluent_cloud/..
[INFO] [2023-11-03 15:06:27] Secrets retrieval completed and saved to secrets.json

reading secrets.json

$ cat secrets.json

[
  {
    "path": "secret2/confluent_cloud/global-creds",
    "value": {
      "data": {
        "data": {
          "confluent_cloud_token": "REDACTED"
        },
        "metadata": {
          "created_time": "2023-11-02T13:14:09.616943342Z",
          "custom_metadata": null,
          "deletion_time": "",
          "destroyed": false,
          "version": 1
        }
      }
    }
  },
  {
    "path": "secret2/confluent_cloud/service/credentials",
    "value": {
      "data": {
        "data": {
          "api_key": "REDACTED",
          "api_secret": "REDACTED"
        },
        "metadata": {
          "created_time": "2023-11-02T13:14:10.336792836Z",
          "custom_metadata": null,
          "deletion_time": "",
          "destroyed": false,
          "version": 1
        }
      }
    }
  },
...

Hope it helps

@biloocabba
Copy link

I am adding my vote, is there any news about when we can have it ? or at least what are the best alternatives for the moment?

@fvogl
Copy link

fvogl commented Nov 16, 2023

would also like to see this implemented

@JohnDuncanScott
Copy link

It would be great to get an update on this. The last official comment on this was March. Secret cycling without this is unbelievably frustrating, especially for complex systems you weren't around for at the beginning (e.g. is this value still used somewhere?).

@vickicello
Copy link

Adding my vote - this would be super helpful for our use case!

@MochaCaffe
Copy link

MochaCaffe commented Dec 9, 2023

Hi everyone,

I managed to implement a server-side recursive secret search feature by adding a new API endpoint to the KV engine.
This was done by modifying the existing KV secret plugin source code, which means that Vault needs to be recompiled locally.

You can check my work here if you're interested for carrying further testing: https://github.com/kosmos-education/vault-plugin-secrets-kv

As I mentioned in the repository, please note that this is a highly experimental feature which could potentially lead to stability issues or vulnerabilities.
In my case, the plugin was tested against a Vault instance with 6000+ secrets stored

image

@adiii717
Copy link

Any progress or developments regarding this? This feature would provide significant benefits.

@voiprodrigo
Copy link

I know this feature request is focused on CLI/API, but are there any plans to provide a tree ui for KV mounts on top of this upcoming work? I think that would make the UI much more usable for ops that need to deal on a daily basis with a lot of static secrets.

Also interested in knowing what the community thinks about this. Cheers.

@adrianlzt
Copy link

Using vault-kv-search and fzf creates a nice way to browse through kv secrets
foo

vault-kv-search --search=path kv -r . --json | jq -r .path | uniq | fzf --preview 'vault kv get --format=yaml ${} | faq -f yaml .data

@huyz
Copy link

huyz commented Feb 18, 2024

@adrianlzt What's the faq command? (And you're missing a closing quote)

@adrianlzt
Copy link

faq is like jq but with form input/output formats (yaml between them).

I'm finally using (as the first get of all keys in my environment is quite slow):

vault read -format json sys/internal/ui/mounts/ | faq -r '.data.secret | keys | .[]' | grep -vw -e sys -e cubbyhole -e identity | xargs -n 1 vaku folder list -p > $temp_vault_path_file
cat $temp_vault_path_file | fzf --preview 'vault kv get -format=yaml ${} | faq -f yaml .data.data' -q "$1 $2 $3"

@dirsigler
Copy link

6 years and no basic implementation of this feature?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-sentiment Tracking high-profile issues from the community enhancement feature-request secret/kv
Projects
None yet
Development

No branches or pull requests