403: Permission denied, even as root token trying to lookup a token by accessor

[Caiyeon]

# This is on a sandbox system

$ vault list auth/token/accessors
Keys
----
04c7507e-33d5-bd82-632b-ba77b0d617bd
08d21210-f940-f8b0-f795-06a958bb02f7
193598b9-c0a0-dca3-2eb9-c3a1e65e0849
23de9916-e07e-0e3d-66ab-ee0e90d4455a
... ...

# A regular token lookup works just fine

$ vault token-lookup -accessor 08d21210-f940-f8b0-f795-06a958bb02f7
Key                     Value
---                     -----
accessor                08d21210-f940-f8b0-f795-06a958bb02f7
creation_time           1493309175
creation_ttl            1728000
display_name            approle
explicit_max_ttl        0
id
meta                    map[]
num_uses                0
orphan                  true
path                    auth/approle/login
policies                [default goldfish]
renewable               true
ttl                     1661640

# But this one specific accessor is unreadable

$ vault token-lookup -accessor 04c7507e-33d5-bd82-632b-ba77b0d617bd
error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup-accessor
Code: 403. Errors:

* 1 error occurred:

* permission denied

# Sanity check to make sure current token is root

$ vault token-lookup
Key                     Value
---                     -----
accessor                67a95bb3-a905-02ba-2a64-2b49c4b09cc0
creation_time           1493249400
creation_ttl            0
display_name            token
explicit_max_ttl        0
id                      goldfish
meta                    <nil>
num_uses                0
orphan                  true
path                    auth/token/create
policies                [root]
renewable               false
ttl                     0

Environment: Vagrant + VirtualBox on Windows 10
OS: debian/jessie64
Vault: v 0.7, -dev mode, pulled Apr 27, last commit: 15842ec2809798baa7cd99a825b9b837dc37b742

Steps to reproduce:
I have no idea. I was making a token-creator page for my vault ui, and I created maybe 20 tokens with short TTLs and a wide variety of nonsensical options. After ~15 minutes, this issue occurred.

Misc:
A vagrant snapshot has been saved, and loading it demonstrates the issue.
However, immediately upon loading the snapshot, I get a 403: bad request instead of permission denied. After awhile, I will get a 403: permission denied.

I can probably resolve this issue by just revoking or running /tidy but if this is an actual vault-side issue, I would assume it is of interest

[Caiyeon]

Here's what the audit log shows
Reading the permission denied accessor:

{
  "time": "2017-04-28T11:14:54Z",
  "type": "request",
  "auth": {
    "client_token": "",
    "accessor": "",
    "display_name": "token",
    "policies": [
      "root"
    ],
    "metadata": null
  },
  "request": {
    "id": "927ed7d0-d23d-20a8-d5b6-3fff5be01f5d",
    "operation": "update",
    "client_token": "hmac-sha256:cc6b8c0158fbd9aed75d3448fa2e6936e33491180f6c3d320eb2d3731ba45097",
    "client_token_accessor": "hmac-sha256:c0a79235b3d7da221d6b8d01d9d85e07192beb475b12a62416f9c16c93819c69",
    "path": "auth/token/lookup-accessor",
    "data": {
      "accessor": "hmac-sha256:6b72d322d46f56b9e0b25fa8cf751e228774f9dfc05fe2983a394c89f0180ac8"
    },
    "remote_address": "127.0.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "error": ""
}
{
  "time": "2017-04-28T11:14:54Z",
  "type": "response",
  "auth": {
    "client_token": "",
    "accessor": "",
    "display_name": "token",
    "policies": [
      "root"
    ],
    "metadata": null
  },
  "request": {
    "id": "927ed7d0-d23d-20a8-d5b6-3fff5be01f5d",
    "operation": "update",
    "client_token": "hmac-sha256:cc6b8c0158fbd9aed75d3448fa2e6936e33491180f6c3d320eb2d3731ba45097",
    "client_token_accessor": "hmac-sha256:c0a79235b3d7da221d6b8d01d9d85e07192beb475b12a62416f9c16c93819c69",
    "path": "auth/token/lookup-accessor",
    "data": {
      "accessor": "hmac-sha256:6b72d322d46f56b9e0b25fa8cf751e228774f9dfc05fe2983a394c89f0180ac8"
    },
    "remote_address": "127.0.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {},
  "error": "1 error occurred:\n\n* permission denied"
}

Reading a normal accessor:

{
  "time": "2017-04-28T11:14:57Z",
  "type": "request",
  "auth": {
    "client_token": "",
    "accessor": "",
    "display_name": "token",
    "policies": [
      "root"
    ],
    "metadata": null
  },
  "request": {
    "id": "1e338321-7199-44c2-cdf0-0697ac751eab",
    "operation": "update",
    "client_token": "hmac-sha256:cc6b8c0158fbd9aed75d3448fa2e6936e33491180f6c3d320eb2d3731ba45097",
    "client_token_accessor": "hmac-sha256:c0a79235b3d7da221d6b8d01d9d85e07192beb475b12a62416f9c16c93819c69",
    "path": "auth/token/lookup-accessor",
    "data": {
      "accessor": "hmac-sha256:e840e925567c0869eee76819b49867a780eef86939182412cd21b42cb89b2b56"
    },
    "remote_address": "127.0.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "error": ""
}
{
  "time": "2017-04-28T11:14:57Z",
  "type": "response",
  "auth": {
    "client_token": "",
    "accessor": "",
    "display_name": "token",
    "policies": [
      "root"
    ],
    "metadata": null
  },
  "request": {
    "id": "1e338321-7199-44c2-cdf0-0697ac751eab",
    "operation": "update",
    "client_token": "hmac-sha256:cc6b8c0158fbd9aed75d3448fa2e6936e33491180f6c3d320eb2d3731ba45097",
    "client_token_accessor": "hmac-sha256:c0a79235b3d7da221d6b8d01d9d85e07192beb475b12a62416f9c16c93819c69",
    "path": "auth/token/lookup-accessor",
    "data": {
      "accessor": "hmac-sha256:e840e925567c0869eee76819b49867a780eef86939182412cd21b42cb89b2b56"
    },
    "remote_address": "127.0.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {
    "data": {
      "accessor": "hmac-sha256:e840e925567c0869eee76819b49867a780eef86939182412cd21b42cb89b2b56",
      "creation_time": 1493309175,
      "creation_ttl": 1728000,
      "display_name": "hmac-sha256:2af22a3bec57a71a96685424de4f7e706fe385865a1c3ac8ca15b3479af70099",
      "explicit_max_ttl": 0,
      "id": "hmac-sha256:33faaaa27ae1ae8bcfd80224e68aaf328da930990743e99a243b67f0b3ec189c",
      "meta": {},
      "num_uses": 0,
      "orphan": true,
      "path": "hmac-sha256:a93cfdf89a26339c59ce4374dae5a33f386799e042fe5c9056917935646f25f9",
      "policies": [
        "hmac-sha256:41028d1a93eefb69d88faa364aefda7242a8f714054da8b57ea3732e0d4df5e5",
        "hmac-sha256:cc6b8c0158fbd9aed75d3448fa2e6936e33491180f6c3d320eb2d3731ba45097"
      ],
      "renewable": true,
      "ttl": 1659078
    }
  },
  "error": ""
}

[jefferai]

This is a duplicate of https://groups.google.com/d/msgid/vault-tool/da880586-b0f0-4ed0-a4be-5319fafd9719%40googlegroups.com?utm_medium=email&utm_source=footer right?

[Caiyeon]

It looks similar, but I am running a dev mode v0.7.0 vault.
I can also confirm that the accessor is un-revokeable

vagrant@goldfish:~$ vault token-revoke -accessor 04c7507e-33d5-bd82-632b-ba77b0d617bd
Success! Token revoked if it existed.
vagrant@goldfish:~$ vault list auth/token/accessors
Keys
----
04c7507e-33d5-bd82-632b-ba77b0d617bd
08d21210-f940-f8b0-f795-06a958bb02f7
193598b9-c0a0-dca3-2eb9-c3a1e65e0849
23de9916-e07e-0e3d-66ab-ee0e90d4455a
2ef2decf-8830-70af-61db-2e756df66523
535b6edf-d949-fa1d-e7d0-c003a2a095a4

vagrant@goldfish:~$ vault token-revoke -accessor 2ef2decf-8830-70af-61db-2e756df66523
Success! Token revoked if it existed.
vagrant@goldfish:~$ vault list auth/token/accessors
Keys
----
04c7507e-33d5-bd82-632b-ba77b0d617bd
08d21210-f940-f8b0-f795-06a958bb02f7
193598b9-c0a0-dca3-2eb9-c3a1e65e0849
23de9916-e07e-0e3d-66ab-ee0e90d4455a
535b6edf-d949-fa1d-e7d0-c003a2a095a4

Note that the second token-revoke command successfully removed the token, but the first token-revoke command did not actually revoke the token, despite reporting a success.

[jefferai]

Could you also send along the audit log entries from when the problematic tokens were created?

[Caiyeon]

Unfortunately I did not have an audit log enabled when I created it. I was merely testing my token creation page.

Also, there is only one problematic token.

[jefferai]

Does the token associated with the accessor that isn't dying still work?

[Caiyeon]

I wanted to try that too, but unfortunately that was probably a UUID because I left the ID parameter empty half the time.

Apologies for the lack of reproducible steps 😅

But indeed, if the token is valid and unrevokable (or even worse, reportedly revoked successfully) this may be quite an issue.

[jefferai]

I think it's more likely that the issue was with cleaning up the accessor. Any chance I can get that Vagrant box from you? Then I can investigate directly.

[Caiyeon]

Possibly. I made a bunch of short TTL tokens.
The vagrantfile and provision files can be found here https://github.com/Caiyeon/goldfish/tree/master/vagrant

Is there a way to transfer my vagrant snapshot? It's simply a virtual box snapshot right?

[jefferai]

Honestly, I don't know, I've never used Vagrant. But probably, yeah, so if you can just archive the entire vbox machine that would probably work.

[Caiyeon]

Well restoration of snapshot to the vagrant machine works fine. So worst case you may need to vagrant up before you restore from snapshot. I'll try to get you the snapshot Monday. 😃

[Caiyeon]

The snapshot is 2GB and the VM disk itself is another 4GB. I'm not sure how viable a transfer would be.

[jefferai]

If you can Dropbox or Google Drive it or something that's one way to do it. But also, since you have a snapshot, any chance you can try some tidying and see if it helps? We have a couple of additions/fixes to tidy coming for 0.7.1, I can point you to a build with them.

[Caiyeon]

I don't have enough space on any of my cloud storages at the moment. And since the environment is a dev server, I can't switch it to 0.7.1 anyway.

However, running tidy works:

vagrant@goldfish:~$ curl -X POST http://127.0.0.1:8200/v1/auth/token/tidy -H 'X-VAULT-TOKEN:<>'
vagrant@goldfish:~$ vault list auth/token/accessors
Keys
----
08d21210-f940-f8b0-f795-06a958bb02f7
193598b9-c0a0-dca3-2eb9-c3a1e65e0849
23de9916-e07e-0e3d-66ab-ee0e90d4455a

Seems like tidy does clean it up.

And 403 is no longer returned:

vagrant@goldfish:~$ vault token-lookup -accessor 04c7507e-33d5-bd82-632b-ba77b0d617bd
error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup-accessor
Code: 400. Errors:

* 1 error occurred:

* invalid accessor

[jefferai]

it's good to know it fixes it; we will have to look into how it's happening in the first place.

[vishalnayak]

@Caiyeon Do you have the command you used to create the tokens? (20 tokens with short TTLs and a wide variety of nonsensical options)

[Caiyeon]

@vishalnayak I have no idea. I was making a token-creator page for my vault ui, and I created maybe 20 tokens with short TTLs and a wide variety of nonsensical options. After ~15 minutes, this issue occurred.

Unfortunately, I nuked my vagrant box with vagrant destroy without realizing that destroy would also delete snapshots. I have a clone of the virtualbox image (but I can't seem to spoof it to work with vagrant anymore)

[jefferai]

Closing for now since we're not sure how to repro, but please let us know if you can/do!