Document how vault approle is used

[chrishiestand]

If someone knowledgeable could fill in any gaps, I'd like to create a PR to address my own concerns with the documentation.

Without reading source code, it's difficult to tell how concourse uses the vault approle because the approle is somewhat complex and the concourse documentation is currently lacking some context. In particular we can help steer a vault admin creating an approle for concourse.

I believe these are the approle parameters which may factor in:

• secret_id_num_uses
• secret_id_ttl
• token_num_uses
• token_ttl
• token_max_ttl
• period

source: https://www.vaultproject.io/api/auth/approle/index.html#secret_id_num_uses

From what I gather (please let me know if this is wrong):

When the ATC starts with --vault-auth-backend approle it does not have a, or load a previous, vault token. So for this reason the approle secret id is always required and should never expire. So secret_id_ttl and secret_id_num_uses should be set as null (empty string).

The ATC will then use the approle id and secret id to create a token, and will try to refresh that token after CONCOURSE_VAULT_AUTH_BACKEND_MAX_TTL. But it will never try to get a different token while the ATC process is running. Because the ATC process might run for 1 second or 5 years, you might as well treat this token as a lifetime periodic token. And so you should set period accordingly - something on the order of hours or days might give concourse time to renew its token despite faults/failures. Therefore you can set token_ttl and token_max_ttl as empty strings.

[edtan]

@chrishiestand Thanks, I think documentation for Vault approle usage would be very helpful! I read through the source code, and here's what I found.

When the ATC starts with --vault-auth-backend approle it does not have a, or load a previous, vault token.

Correct, it doesn't load any previous tokens when the ATC starts. --vault-auth-backend approle is just used to determine the login API path, e.g. /v1/auth/<vault-auth-backend>/login Any --vault-auth-params are then put into the login request payload.

Next, the token renewal behavior depends on whether or not --vault-auth-backend-max-ttl is 0. If it's 0, Concourse will constantly renew tokens at half the lease duration. This seems to be intended for periodic tokens. If --vault-auth-backend-max-ttl is non-zero, it should match the Vault approle's token_max_ttl setting - Concourse will keep renewing the token until the token_max_ttl is reached, at which point it will then login again to obtain a new token. In both cases, Concourse will re-login if the token's lease has expired.

So to summarize, it seems like secret_id_ttl, secret_id_num_uses and token_num_uses should be blank/zero. But the other parameters - token_ttl, token_max_ttl, and period - can be set as desired.

[chrishiestand]

Thank @edtan! I installed via the helm chart and there is no chart exposure of --vault-auth-backend-max-ttl so I did not even consider how that might change the behavior.

So if you want concourse to regularly discard an old token and get a new token you should set --vault-auth-backend-max-ttl. And if you're okay with concourse always reusing the same (periodic) token then you can leave --vault-auth-backend-max-ttl unset, or set it to 0.

Regardless of any setting, once it's loaded the vault token never leaves the ATC's memory (ignoring swap), right?. So if someone is not concerned about their token being intercepted on the wire (or possibly via vault) then the two styles would be equivalent, security-wise. But if someone is concerned about their token being read on the wire, then setting --vault-auth-backend-max-ttl and using non-periodic tokens would be a possibly more secure approach because concourse would be replacing the tokens between TTLs?

Relatedly does one of those two methods mean sending the "keys to the kingdom" over the wire less frequently? It seems to me like neither approach is significantly better (or ideal). In the case of a periodic token, the periodic token will be sent over the wire and possibly never expire. If compromised this periodic token will give an attacker access to vault's concourse secrets. Similarly, when --vault-auth-backend-max-ttl is unset or 0 then the secret_id will be repeatedly sent over the wire which, if compromised, will give an attacker access to vault's concourse secrets.

I think the ideal behavior for the paranoid is that a token that expires is sent over the wire, and that token is used to generate a new token which expires even later. Concourse just stores the new rotating token every time it is refreshed. In this way, nothing sent over the wire could be used later than the token's TTL. Where to store this rotating token is a new problem though, and maybe why this approach isn't used? You'd have to store it somewhere stateful, but you probably wouldn't want it in the postgres DB. If kubernetes were a requirement, concourse could update the kubernetes secret with the new token value (but kubernetes is not currently a requirement).

Anyway, I digress. Thanks for your time and I'd be happy to hear your opinions on what I've written above. The main goal is to help people understand the security trade-offs of the parameters.

[edtan]

Yup, the Vault token never leaves the ATC's memory.

I agree with what you said concerning tokens being read on the wire. But maybe using the periodic token is slightly better, because if the secret_id is compromised it could be used to generate more tokens?

I like the idea of token rotation, but this would mean the approle policy would have to be updated to allow creating new tokens, right? So if a token was compromised, it could be used to create additional tokens, similar to if secret_id was compromised.

For implementing the rotating tokens, why would the new token need to be stored somewhere stateful? Couldn't it also be stored in memory?

[chrishiestand]

I like the idea of token rotation, but this would mean the approle policy would have to be updated to allow creating new tokens, right? So if a token was compromised, it could be used to create additional tokens, similar to if secret_id was compromised.

It's similar to the approle with secret_id but is better because the secret_id never expires but the rotating token would. A compromise would not just have to happen, but the attacker would need to compromise a "fresh" token.

For implementing the rotating tokens, why would the new token need to be stored somewhere stateful? Couldn't it also be stored in memory?

Because in my imagined scenario, only the rotating token could be used to access secrets or a new token. There would be no permanent credentials such as an approle secret_id. And so you'd need to be able to auth after restarting the ATC.

[edtan]

Ahh yes, good points. I don't think I fully understand the rotating token implementation yet, but perhaps we should create a new feature request for it?

[chrishiestand]

@edtan I've been considering your question. I looked into whether or not this is even possible with vault and it does appear to be possible by using a token role that creates orphaned tokens and has the capability of creating a new token of the same role.

On the downside I don't see any way to limit the number of such active tokens in vault (to prevent a successful attacker from creating unlimited new tokens), this would have to be done on the application/client side (relative to the vault api). Not that this is a net negative: the same problem already exists with the approle approach.

But more to the point, my current usage of concourse and vault is not so security sensitive as to necessitate this approach. If it were already available off the shelf, I would use it. But I would not be a good advocate for the feature request since I do not personally need it.

I'm going to close this issue. I will create a documentation-updating PR which summarizes the vault settings we've discussed. Thanks very much for your help @edtan.

[chrishiestand]

Actually, I'll let the PR close this issue.

[vito]

@edtan @chrishiestand Would y'all mind reviewing #172 if you have time? I fleshed out the docs quite a bit - now it covers approle auth and cert auth, complete with an example with Docker Compose.

[edtan]

I read through the new Vault doc, and I think it looks great! I like the example and it should be really helpful for users trying to set up Vault. I didn't know about docker-compose overrides, cool!

[vito]

@edtan Thanks for checking it out! 🙂

[chrishiestand]

@vito Thanks for the PR, and sorry I didn't get a PR on this yet myself.

It looks like the PR removes all references to using period tokens and switches to approles? Are periodic tokens being deprecated in favor of approles? If this was not explicitly in mind, I imagine that having documentation of both periodic tokens and approles would be useful?

I think the approle documentation is much better! I think, related to the above discussion, these should also be included (even if briefly):

• token_num_uses
• token_max_ttl
• period

And in the intro, a brief description of what is happening behind the scenes. IIRC, on initial startup, concourse sends the approle id and secret to vault, and get issued a token. Then subsequent authentication happens with the token, following the approle parameters described. Concourse will send the approle id and secret again, to generate new tokens when needed.

Just my $0.02 for now. I'm sorry I haven't and still don't have the time in the near future to create my own PR. I would not want my perfect to be the enemy of your better.

[vito]

@chrishiestand np, thanks for the feedback!

It looks like the PR removes all references to using period tokens and switches to approles? Are periodic tokens being deprecated in favor of approles? If this was not explicitly in mind, I imagine that having documentation of both periodic tokens and approles would be useful?

Hm, I must have had periodic tokens mixed up with something else. I could have sworn you couldn't create one that would be renewed forever - I thought they were limited by the system max TTL (768h) so I didn't want to recommend anyone use them. Looking at it now, it looks like as long as you're able to keep renewing it it should be fine. We'd just have to have a note to be careful about web downtime, since if it's down for longer than the --period the token will expire and Concourse will be locked out.

Might be worth introducing periodic tokens first and then lead into approle as it removes the problem with web being down for too long.

I think the approle documentation is much better! I think, related to the above discussion, these should also be included (even if briefly):

Sounds good.

And in the intro, a brief description of what is happening behind the scenes. IIRC, on initial startup, concourse sends the approle id and secret to vault, and get issued a token. Then subsequent authentication happens with the token, following the approle parameters described. Concourse will send the approle id and secret again, to generate new tokens when needed.

👍 I do have this in there but it's kind of awkwardly far into the document, so I'll move it up and add more details.