Documentation or implementation bug? KV2 API docs has wrong method for create/update

[johnnybubonic]

Describe the bug
(X-REF: hvac/hvac#389)

When using the Vault CLI client, KV2 secrets are correctly written (and read, etc.). However, this does not represent the documentation. When debugging what I thought was an hvac bug, I discovered that hvac conforms to the API docs' request method (POST) - but the Vault client does not (PUT). e.g.

hvac:

POST /v1/secret/data/foo/bar/baz HTTP/1.1
Host: localhost:8200
User-Agent: python-requests/2.21.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-Vault-Token: [REDACTED FOR DEMO]
Content-Length: [REDACTED FOR DEMO]
Content-Type: application/json

{"options": {}, "data": {"x": "1", "y": "2", "z": "3"}}

Vault:

PUT /v1/secret/data/foo/bar/baz HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
Content-Length: [REDACTED FOR DEMO]
X-Vault-Token: [REDACTED FOR DEMO]
Accept-Encoding: gzip

{"x":"1","y":"2","z":"3"}

Docs:

POST /secret/data/:path

As a result, the API-compliant call fails with a permission error (as according to a policy), etc.

To Reproduce
See above (docs fix?).

Expected behavior
The given method in API docs works.

Environment:

• Vault Server Version: 1.0.1
• Vault CLI Version: 1.0.1
• Server Operating System/Architecture: CentOS 7.6.1810

[jefferai]

Vault doesn't make a distinction between PUT and POST.

[johnnybubonic]

Any ideas on why one fails and the other succeeds? It's been duplicated by another party. The only other difference, besides headers which I'd suspect are inconsequential, are the body - but the body also conforms to the API.

[jefferai]

Yes, the difference is one is writing data formatted for a KVv1 mount and the other is writing data formatted for a KVv2 mount. Note the difference in the data formatting.

Closing this as it appears to be an hvac issue.

[johnnybubonic]

But it's a KV2 mount, and the documentation for Create/Update Secret for KV2 states hvac's data formatting, not Vault's.

Path                Plugin       Accessor              Default TTL    Max TTL       Force No Cache    Replication    Seal Wrap    Options           Description
----                ------       --------              -----------    -------       --------------    -----------    ---------    -------           -----------
(...)
foo/bar/          kv           kv_[REDACTED]           system         system        false             replicated     false        map[version:2]    Test KV2 store.
(...)

[jefferai]

I honestly am not sure, but:

$ vault secrets list -detailed
Path          Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    Options           Description
----          ------       --------              -----------    -------    --------------    -----------    ---------    -------           -----------
cubbyhole/    cubbyhole    cubbyhole_988ad756    n/a            n/a        false             local          false        map[]             per-token private secret storage
identity/     identity     identity_87072d72     system         system     false             replicated     false        map[]             identity store
secret/       kv           kv_9e2bd87d           system         system     false             replicated     false        map[version:2]    key/value secret storage
sys/          system       system_2bb809ed       n/a            n/a        false             replicated     false        map[]             system endpoints used for control, policy and debugging

$ vault kv put -output-curl-string secret/foo a=1 b=2
curl -X PUT -H "X-Vault-Token: $(vault print token)" -d '{"data":{"a":"1","b":"2"},"options":{}}' http://127.0.0.1:8200/v1/secret/data/foo

$ curl -X PUT -H "X-Vault-Token: $(vault print token)" -d '{"data":{"a":"1","b":"2"},"options":{}}' http://127.0.0.1:8200/v1/secret/data/foo
{"request_id":"3c9653f2-6644-e3c5-323b-0d263cb8ce4d","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2019-02-08T20:25:20.157076345Z","deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}

$ curl -X POST -H "X-Vault-Token: $(vault print token)" -d '{"data":{"a":"1","b":"2"},"options":{}}' http://127.0.0.1:8200/v1/secret/data/foo
{"request_id":"a0af7c0b-fedc-1fda-dead-322bad88daec","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2019-02-08T20:25:25.029744779Z","deletion_time":"","destroyed":false,"version":2},"wrap_info":null,"warnings":null,"auth":null}

PUT and POST work exactly the same, and they're using data/options format for my KVv2 endpoint.

The CLI vault kv command does a preflight check to know whether it's talking to version 1 or version 2 and formats data appropriately. The CLI clearly thinks you are talking to a v1 mount.

My sneaking suspicion is that you are not talking to the same Vault server in all cases. Or, that the Vault server you're talking to via the CLI (perhaps due to a VAULT_ADDR setting) is quite old and only supports KVv1 despite version 2 being set in the options map. (To put it another way, you can set the value in the options map even if the current version of Vault doesn't actually support it.)

[johnnybubonic]

Hrm. I'll spin up a fresh instance sometime over the weekend and see if that's the case re: tainted env, but I was pretty careful to perform both against the same server unless I'm somehow running one server on 127.0.0.1:8200 and a totally different one on localhost:8200, as per the headers in the snippets. Vault server and client are both 1.0.1 and I hadn't installed it prior, so I know it's not a version mismatch.

Even more confounding is I get a permission denied in hvac but not Vault when writing to the same destination with the same token. ¯_(ツ)_/¯

What happens if Vault can't hit the /v1/sys/internal/ui/mounts preflight? Does it fall back to KV1? (Though I don't think that's it either, because that token also had the default policy enabled and I didn't see a 40x for the path in the packet dump.)

[johnnybubonic]

Apologies for the delay. I've confirmed this with a fresh install of 1.0.1 server and client. The documentation as per here is indeed incorrect. Sorry to be a stick in the mud, but it really is. Below is reproducible (minus exact token strings etc., of course), extensive evidence.

Setup

Server

Config

/var/tmp/testserver.hcl:

storage "file" {
  path = "/var/tmp/vault"
}
listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}
ui = true
api_addr = "http://127.0.0.1:8200"
log_level = "debug"

Policy

/var/tmp/pol.json:

{
    "path": {
        "secret/data/foo/bar/*": {
            "capabilities": [
                "create",
                "read",
                "update",
                "delete",
                "list"
            ],
            "required_parameters": [
                "a",
                "b"
            ],
            "allowed_parameters": {
                "a": [],
                "b": [],
                "c": []
            }
        },
        "secret/data/foo/baz/*": {
            "capabilities": [
                "create",
                "read",
                "update",
                "delete",
                "list"
            ],
            "required_parameters": [
                "a",
                "b"
            ],
            "allowed_parameters": {
                "a": [],
                "b": [],
                "c": []
            }
        }
    }
}

Invocation

export VAULT_ADDR='http://127.0.0.1:8200'  # unnecessary since it's in the config, but what the hey.
vault server -config=/var/tmp/testserver.hcl
(... separate terminal ...)
export VAULT_ADDR='http://127.0.0.1:8200'
vault operator init -key-shares=1 -key-threshold=1

Then yields:

Unseal Key 1: UTkBO1/tWiKWnEeMHpKr+lrHpbD1k07bdGyIzXydZGI=

Initial Root Token: s.5r5tmoobrdh1dGAUeQuMy4wa

Vault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 1 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Continuing...

export VAULT_TOKEN='s.5r5tmoobrdh1dGAUeQuMy4wa'
vault operator unseal
(...)
vault secrets enable -version=2 -path=foo/bar kv
vault secrets enable -version=2 -path=foo/baz kv
vault policy write poc /var/tmp/pol.json

Client

vault token create -policy=poc

Yields:

Key                  Value
---                  -----
token                s.1OhMEqUC0f6AxmlVdsKBqR5L
token_accessor       6fIqJ1cQtU6oSaFl75d646pc
token_duration       768h
token_renewable      true
token_policies       ["default" "poc"]
identity_policies    []
policies             ["default" "poc"]

Continuing...

Vault

export VAULT_TOKEN='s.1OhMEqUC0f6AxmlVdsKBqR5L'

hvac

import hvac

client = hvac.Client('http://127.0.0.1:8200', token = 's.1OhMEqUC0f6AxmlVdsKBqR5L')

Testing

Vault

vault kv put secret/data/foo/bar/baz a=1 b=2 c=3

Yields:

Success! Data written to: secret/data/foo/bar/baz

And per a packet capture:

GET /v1/sys/internal/ui/mounts/secret/data/foo/bar/baz HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L
Accept-Encoding: gzip
(---)
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Mon, 11 Feb 2019 17:31:15 GMT
Content-Length: 386

{"request_id":"adcf01df-517c-b6c6-c2ee-1901d9dc8176","lease_id":"","renewable":false,"lease_duration":0,"data":{"accessor":"kv_0722efa0","config":{"default_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0},"description":"key/value secret storage","local":false,"options":{"version":"1"},"path":"secret/","seal_wrap":false,"type":"kv"},"wrap_info":null,"warnings":null,"auth":null}
(---)
PUT /v1/secret/data/foo/bar/baz HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
Content-Length: 25
X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L
Accept-Encoding: gzip

{"a":"1","b":"2","c":"3"}
(---)
HTTP/1.1 204 No Content
Cache-Control: no-store
Content-Type: application/json
Date: Mon, 11 Feb 2019 17:31:15 GMT

hvac

client.secrets.kv.v2.create_or_update_secret('foo/bar/baz', secret = {'a': 4, 'b': 5, 'c': 6})

Yields:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/[REDACTED]/.local/lib/python3.7/site-packages/hvac/api/secrets_engines/kv_v2.py", line 122, in create_or_update_secret
    json=params,
  File "/home/[REDACTED]/.local/lib/python3.7/site-packages/hvac/adapters.py", line 106, in post
    return self.request('post', url, **kwargs)
  File "/home/[REDACTED]/.local/lib/python3.7/site-packages/hvac/adapters.py", line 276, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/home/[REDACTED]/.local/lib/python3.7/site-packages/hvac/utils.py", line 33, in raise_for_error
    raise exceptions.Forbidden(message, errors=errors)
hvac.exceptions.Forbidden: 1 error occurred:
	* permission denied

And per a packet capture:

POST /v1/secret/data/foo/bar/baz HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: python-requests/2.21.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L
Content-Length: 49
Content-Type: application/json

{"options": {}, "data": {"a": 4, "b": 5, "c": 6}}HTTP/1.1 403 Forbidden
Cache-Control: no-store
Content-Type: application/json
Date: Mon, 11 Feb 2019 17:33:06 GMT
Content-Length: 60

{"errors":["1 error occurred:\n\t* permission denied\n\n"]}

Nothing new here. Probably the data format, sure. But please, for goodness' sake, look at what your documentation says for KV2 API data format:

{
  "options": {
      "cas": 0
  },
  "data": {
      "foo": "bar",
      "zip": "zap"
    }
}

As I stated before, the Vault client's format does not conform to the API and passes. hvac does conform to the API documentation (again, for KV2) and fails with a permission denied.

This is even reproducible with curl, replicating the example straight from the docs - the only differences being the actual keys/values and policy:

payload.json:

{
  "options": {
      "cas": 0
  },
  "data": {
      "a": "4",
      "b": "5",
      "c": "6"
    }
}

curl --header "X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L" --request POST --data @payload.json http://127.0.0.1:8200/v1/secret/data/foo/bar/baz

Yields:

{"errors":["1 error occurred:\n\t* permission denied\n\n"]}

With a root token, this succeeds:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8200 (#0)
> POST /v1/secret/data/foo/bar/baz HTTP/1.1
> Host: 127.0.0.1:8200
> User-Agent: curl/7.63.0
> Accept: */*
> X-Vault-Token: s.5r5tmoobrdh1dGAUeQuMy4wa
> Content-Length: 94
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 204 No Content
< Cache-Control: no-store
< Content-Type: application/json
< Date: Mon, 11 Feb 2019 17:49:29 GMT
< 
* Connection #0 to host 127.0.0.1 left intact

Which normally would indicate a policy issue except as shown above, the default policy is also applied and the Vault client itself, using the same non-root token (with, obviously, the same policy) succeeds in writing.

Of course, the result is not what one would expect:

export VAULT_TOKEN='s.5r5tmoobrdh1dGAUeQuMy4wa'
vault kv get -version=2 secret/data/foo/bar/baz

Yields:

===== Data =====
Key        Value
---        -----
data       map[a:4 b:5 c:6]
options    map[cas:0]

Again, using the same data format as the KV2 docs.

When the format used with the Vault client, which is not the format used in the docs for KV2, it of course succeeds (and even with a non-root token):

payload.json:

{
  "a": "7",
  "b": "8",
  "c": "9"
}

curl -v --header "X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L" --request POST --data @payload.json http://127.0.0.1:8200/v1/secret/data/foo/bar/baz

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8200 (#0)
> POST /v1/secret/data/foo/bar/baz HTTP/1.1
> Host: 127.0.0.1:8200
> User-Agent: curl/7.63.0
> Accept: */*
> X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 34 out of 34 bytes
< HTTP/1.1 204 No Content
< Cache-Control: no-store
< Content-Type: application/json
< Date: Mon, 11 Feb 2019 18:01:34 GMT
< 
* Connection #0 to host 127.0.0.1 left intact

And yields the proper key/value pairs:

== Data ==
Key    Value
---    -----
a      7
b      8
c      9

Please fix the API documentation for KV2.

[jefferai]

Your packet capture confirms that you are writing to a version 1 mount with the client:

GET /v1/sys/internal/ui/mounts/secret/data/foo/bar/baz HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
X-Vault-Token: s.1OhMEqUC0f6AxmlVdsKBqR5L
Accept-Encoding: gzip
(---)
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Mon, 11 Feb 2019 17:31:15 GMT
Content-Length: 386

{"request_id":"adcf01df-517c-b6c6-c2ee-1901d9dc8176","lease_id":"","renewable":false,"lease_duration":0,"data":{"accessor":"kv_0722efa0","config":{"default_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0},"description":"key/value secret storage","local":false,"options":{"version":"1"},"path":"secret/","seal_wrap":false,"type":"kv"},"wrap_info":null,"warnings":null,"auth":null}
(---)

Note this in the output: "options":{"version":"1"}. The mount you'[re connecting to via the CLI is a version 1 KV mount, so the CLI formats the input parameters accordingly.

Note also the mount: secret/. You are mounting version 2 mounts at:

vault secrets enable -version=2 -path=foo/bar kv
vault secrets enable -version=2 -path=foo/baz kv

But in HVAC you're accessing secret/data/foo/bar. That's a different, version 1 mount.

[johnnybubonic]

Please review the documentation as per links above. The documentation for KV2 API is giving the KV1 data format.

Edit: Are specified -paths for kv2 not mounted on secret/data/? This indicates they are:

The version 2 kv store uses a prefixed API, which is different from the version 1 API. Before upgrading from a version 1 kv the ACL rules should be changed. 

[briankassouf]

No, paths are rooted at /. So since you mounted at -path=foo/bar the api path should be v1/foo/bar/data/baz. Instead you're accessing the secret/ mount which is a v1 kv mount. Secret backends cannot be mounted at a suffix of another secret mount. So nothing can be mounted with the prefix of secret/ unless you first remove that mount.

That documentation is referencing the API path, not the mount path.

[johnnybubonic]

I see. So the documentation assumes the default -dev server, then.

[johnnybubonic]

Yep, this is on me then. I offer my sincere apologies.

I'd like to humbly suggest some clarification of the above in the docs, ideally some sort of clarification via stand-in representations (i.e. <secret>/dev/team-1 opposed to secret/dev/team-1, etc.).