Vault integration with strict vault token policy fails

[rikribbers]

Hi,

When running docker with vault integration it looks like the authentication against vault is broken. The behaviour described below was working in version 3.6.x and with 3.9.2 it breaks our setup.

We have a strict policy on the vault secret-id; I can be used only once.

However concourse is using it twice for login. In the logging you will see a "atc.credential-manager.login.succeeded" message and some "atc.credential-manager.login.failed" message complainig about an invalid secret-id.

Could this be because of some changes (I missed apearently) or is this indeed incorrect behaviour.

This is the docker compose:

version: '3'

services:
concourse-db:
image: postgres:9.6-alpine
network_mode: bridge
environment:
POSTGRES_DB: concourse
POSTGRES_USER: concourse
POSTGRES_PASSWORD: changeme
PGDATA: /database

concourse-web:
image: concourse/concourse:3.9.2
links: [concourse-db]
command: web
depends_on: [concourse-db]
network_mode: bridge
ports: ["8080:8080"]
volumes: ["./keys/web:/concourse-keys"]
environment:
CONCOURSE_BASIC_AUTH_USERNAME: concourse
CONCOURSE_BASIC_AUTH_PASSWORD: changeme
CONCOURSE_POSTGRES_HOST: concourse-db
CONCOURSE_POSTGRES_USER: concourse
CONCOURSE_POSTGRES_PASSWORD: changeme
CONCOURSE_POSTGRES_DATABASE: concourse
CONCOURSE_VAULT_URL: https://example.nl
CONCOURSE_VAULT_AUTH_BACKEND: approle
CONCOURSE_VAULT_AUTH_PARAM: "role_id=5894306a-5c16-8213-026f-8d8cdc48a9f4,secret_id=55c5123d-dab1-4438-7e55-aa8e76453d8f"
CONCOURSE_VAULT_PATH_PREFIX: /secret/concourse
concourse-worker:
image: concourse/concourse:3.9.2
privileged: true
network_mode: bridge
links: [concourse-web]
depends_on: [concourse-web]
command: worker
volumes: ["./keys/worker:/concourse-keys"]
environment:
CONCOURSE_TSA_HOST: concourse-web

Below some logging (only changed the url)

{"timestamp":"1522164991.980530977","source":"atc","message":"atc.credential-manager.login.succeeded","log_level":1,"data":{"lease-duration":3600,"name":"vault","policies":["concourse-client-token-policy","default"],"session":"3.1","token-accessor":"ad0eb5d6-d851-fc8d-8322-0374c4c50146"}}
{"timestamp":"1522165003.194758415","source":"tsa","message":"tsa.listening","log_level":1,"data":{}}
{"timestamp":"1522165003.197584629","source":"atc","message":"atc.listening","log_level":1,"data":{"debug":"127.0.0.1:8079","http":"0.0.0.0:8080"}}
{"timestamp":"1522165003.299190283","source":"atc","message":"atc.credential-manager.login.failed","log_level":2,"data":{"error":"Error making API request.\n\nURL: PUT https://example.nl/v1/auth/approle/login\nCode: 400. Errors:\n\n* failed to validate credenti
als: invalid secret_id "bac0deb9-811f-2ec0-1fcd-0b5a38d70e10"","name":"vault","session":"31.1"}}
{"timestamp":"1522165004.416620731","source":"atc","message":"atc.credential-manager.login.failed","log_level":2,"data":{"error":"Error making API request.\n\nURL: PUT https://example.nl/v1/auth/approle/login\nCode: 400. Errors:\n\n* failed to validate credenti
als: invalid secret_id "bac0deb9-811f-2ec0-1fcd-0b5a38d70e10"","name":"vault","session":"31.2"}}
{"timestamp":"1522165005.545902729","source":"atc","message":"atc.credential-manager.login.failed","log_level":2,"data":{"error":"Error making API request.\n\nURL: PUT https://example.nl/v1/auth/approle/login\nCode: 400. Errors:\n\n* failed to validate credenti
als: invalid secret_id "bac0deb9-811f-2ec0-1fcd-0b5a38d70e10"","name":"vault","session":"31.3"}}
{"timestamp":"1522165006.652223825","source":"atc","message":"atc.credential-manager.login.failed","log_level":2,"data":{"error":"Error making API request.\n\nURL: PUT https://example.nl/v1/auth/approle/login\nCode: 400. Errors:\n\n* failed to validate credenti
als: invalid secret_id "bac0deb9-811f-2ec0-1fcd-0b5a38d70e10"","name":"vault","session":"31.4"}}
{"timestamp":"1522165007.762223959","source":"atc","message":"atc.credential-manager.login.failed","log_level":2,"data":{"error":"Error making API request.\n\nURL: PUT https://example.nl/v1/auth/approle/login\nCode: 400. Errors:\n\n* failed to validate credenti
als: invalid secret_id "bac0deb9-811f-2ec0-1fcd-0b5a38d70e10"","name":"vault","session":"31.5"}}
{"timestamp":"1522165008.872875214","source":"atc","message":"atc.credential-manager.login.failed","log_level":2,"data":{"error":"Error making API request.\n\nURL: PUT https://example.nl/v1/auth/approle/login\nCode: 400. Errors:\n\n* failed to validate credenti
als: invalid secret_id "bac0deb9-811f-2ec0-1fcd-0b5a38d70e10"","name":"vault","session":"31.6"}}

[rikribbers]

nice indentation btw..

[baptiste-ls]

I have the same setup for Concourse 3.9.2 and Vault. IMO this looks like an approle backend configuration issue, have you tried recreating it to provision new role/secret_id?

[rikribbers]

Yes I have, but no success

vault write -address=${VAULT_ADDR} auth/approle/role/concourse-client
bind_secret_id=true
policies=concourse-client-token-policy
secret_id_num_uses=1
secret_id_ttl=1h
token_num_uses=0
token_ttl=1h
period=1h

and the underlying policy is

path "secret/concourse/*" {
capabilities = ["read"]
}

Will try some more today to get it up-and-running though. brb

[rikribbers]

I have been debugging the startup while tracing on the docker host using the following script (inside a magic container from registry.access.redhat.com/rhel7/rhel-tools)

while true; do pid=$(pgrep -x -f "/usr/local/bin/concourse web"); if [[ -n
then strace -f -o /tmp/strace.out -s 2000 -vvtf -p "$pid"; break; fi; done

I see two pids authenticating to vault, I don't know why but this seems wrong... i would expect only one.

workaround (unacceptable for our use-case though) increase the secret_id_num_uses to 2

[rikribbers]

We upgraded to the latest and greatest version and the issue has not been seen anymore.

so cleaning up and closing this issue