Renewing AWS Backend credentials (or getting new ones) #85
Comments
|
I too, would like to see an answer for this. Thank you. |
|
A short TTL is something you're looking for to renew the lease periodically, as long as your application is running. Spring Cloud Vault Config is lease-aware and renews leases by default ( The message you get ( You don't want to obtain new credentials all over because you would be required to propagate the new credentials to various places without actually knowing, where these credentials are really used. Obtaining new credentials creates a lot of new users and Spring Cloud Vault Config exposes credentials just to the Environment. Properties are picked up from there by configuration components and are be held in components that are not under control of Spring Cloud Vault Config. |
|
Thanks much @mp911de I'm asked my Vault admin to fix access to /sys/renew that and now I've moved from a 403 to a 400 during renewal. I'm probably missing 1 more thing. I will take that up tomorrow with them. One more question if you don't mind. I cranked up DEBUG and see a lot of these. The first one starts with 360. Is that the lease duration coming back from Vault? Again, I don't think this is something I need to specify in spring-cloud-vault-config. Cheers! |
|
Had a quick thought. I know how to curl this manually. |
|
Your lease has a ttl (lease time) = 360 and max_ttl (max_lease) = 360 set and that's why you can't renew the lease beyond 360 seconds. The log output shows a decreasing TTL. Usually, the ttl is short and the max_ttl is long (a month or so). Max TTL is intended as hard limit to enforce credential rotation. |
|
My Vault admin is telling me "Right now, the default TTL is set to 360 seconds and the Max TTL is set to 720 seconds". And he's saying that if it doesn't know how to request new creds when it can't extend the lease, then it will just break after whatever the max TTL is. Is that true? Or am I misunderstanding the difference between renewing a lease and getting new cred's? Remember that this is for the AWS backend and those credentials are temporary not static. The docs in Vault themselves get revoked at some point.
I'm going to play with this locally more today now that S3 is up :-) |
|
720 seconds max ttl means that after two renewals (720 seconds after creating) AWS credentials, they will expire and will no longer be valid. The Java AWS SDK uses a We will provide a user API in Spring Vault to work with renewable leases through Spring's |
|
Thanks again. Soo. This statement: "Max TTL is intended as hard limit to enforce credential rotation." I was just asked this. Does |
|
No, there's no rotation support here. The approach is to restart the application (or use Spring Cloud's |
|
Ah. Thank you. Now it's more clear to us. We didn't know you didn't support key rotation. There are at least some use cases where refreshing those credentials should not pose an issue. Our question is do you have any plans to support key rotation at all in the project? We need key rotation either way (especially for S3) and are not comfortable with using @RefreshScope or bouncing our apps on a schedule. If we could continue this conversation with someone planning your project roadmap , we might be able to contribute an optional key rotation integration back into this project if that is desirable? While we could write our own solution, I think there's a lot to be gained by utilizing the existing Vault integration and the lease renewal features that are already there. We just need key rotation too. Would this be interesting to this project? |
|
Key rotation is rather simple if you look only on lease/renewal/expiry. The complexity comes from the context: For Spring Cloud Vault (actually it's related to Spring Boot), you need to see how components are wired. Spring Boot initializes its components on startup and gathers configuration from various sources ( Boot applies the configuration to the components it bootstraps (i.e. copies the values obtained from If you limit credential rotation to a particular component ( You can build key rotation for your specific problem using Spring Vault that gives you access to the Vault API. Ephemeral configuration isn't supported beyond |
|
@Blackbaud-KevinHutson can you explain why you "are not comfortable with using @RefreshScope"? /cc @dsyer |
|
FWIW: I pushed a draft of SecretLeaseContainer container = new SecretLeaseContainer(vaultOperations, taskScheduler);
final RequestedSecret requestedSecret = container.requestRotatingSecret("mysql/creds/my-role");
container.addLeaseListener(new LeaseListenerAdapter() {
public void onLeaseEvent(LeaseEvent leaseEvent) {
if (requestedSecret == leaseEvent.getSource()) {
if (leaseEvent instanceof LeaseCreatedEvent) {
}
if (leaseEvent instanceof LeaseExpiredEvent) {
}
}
}
}); |
|
|
|
Vault lease renewal is intrinsic with regard to scheduled lease renewals as long as the application runs. Lease renewal is limited to Vault and specific to Vault. Renewing leases does not refresh the configuration, it's keeping the Vault lease alive, extending TTL. It does not require configuration refresh because credentials stay the same. It makes sense in that context to notify users about renewal, expiry and possible errors that happen. Only in the last stage of Vault leases, obtaining new secrets after the lease expired it makes sense to refresh credentials scoped to just the expired secret leaving other secrets untouched. IIRC refreshing Maybe there is something at Framework level that we could do to combine aspects from |
|
OK, so I can understand the need for a vault-specific event to refresh itself (so the client can continue to contact the server). If we restrict the focus to that, and also emit an
I don't think that's technically necessary - it might be the way the listener that looks for |
|
Yes, that's the right way to think about it. Spring Vault |
|
@spencergibb Sorry! It's been long time and I forgot to respond. Our team, after much discussion, overcame their issue with RefreshScope. That still seems to be the best option we have right now. So, we aren't blocked by this particular issue now. Thanks again! |
|
Thanks for your response. We're not finally decided on how to proceed with credential rotation for the integrations (AWS, Databases, …). I'm closing this ticket for now and we'll revisit the topic at a later time. |
|
I am running into the same issue using a database backend. Unable to get new creds once the lease expires |
|
@gigenthomas Use |
|
@mp911de - Wondering if you could provide some additional guidance on how to use SecretLeaseContainer ? Using a postgres backend What am I doing for the LeaseExpiredEvent ? ` ` Thanks in advance !! |
|
@gigenthomas Please post your question on StackOverflow. |
|
Hi guys, I implemented Lease Rotation for |
Looks good, but be careful. Your solution will work only with Hikari JDBC pool 3.2.0+ See here Also, how did you fix issue with By the way, solution works for |
|
Hey @frombrest
You are right. My solution is very specific.
Sorry, but with just this line, it would be difficult to help you. If you have a small sample that I could clone and try to reproduce on my machine, I would be glad to help. |
|
@ivangfr Don't worry... I've fixed my issue: Thank you for your solution. |
|
I've written a blog post about how to ensure that expiring Spring Cloud Vault dynamic database secrets are renewed, when reaching Hashicorp Vault's max_ttl: Hashicorp Vault max_ttl Killed My Spring App |
|
The follow-up post about how to rotate expiring relational Spring Cloud Vault database credentials without downtime is available: Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring |
Hi there,
I'm trying to use
spring-cloud-vault-configwith an AWS backend. It works great on Spring startup, hitting the AWS backend and giving me an initial set of credentials. But the minute it tries to renew the lease it fails. After talking with my Vault administrator, he pointed me here:According to Vault's "Lease, Renew and Revoke" section (https://www.vaultproject.io/docs/concepts/lease.html) it states "For example, with the AWS secret backend, the access keys will be deleted from AWS the moment a secret is revoked. This renders the access keys invalid from that point forward."
So, I'm trying to understand what my options are. He's giving me a TTL of only a few minutes for the S3 access for those cred's. I'm going to need to go back and get new cred's periodically. If I can't get the lease renewed, what can I do? It appears that
spring-cloud-vault-configonly hits that original endpoint on startup. Is there some other config or switch I can utilize that would get me new credentials? Or do I need to manually write something in Spring that hits that Vault endpoint (which would seem to defeat the purpose of this library)? Any thoughts?Example error that I'm getting back:
org.springframework.vault.client.VaultException: Cannot renew lease: Status 403 URI https://host:8200/v1/sys/renew/aws-dev/creds/S3FullAccess/bcad60d8-aad7-b01b-ee93-005eb67c4a56: permission deniedI'm using the configuration stated here:
http://cloud.spring.io/spring-cloud-vault-config/spring-cloud-vault-config.html#vault.config.backends.aws
with something like this in bootstrap.properties
Thanks so much! I feel like I must be missing something obvious.
The text was updated successfully, but these errors were encountered: