-
Notifications
You must be signed in to change notification settings - Fork 117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
last-applied-configuration contains plain text secret values #1118
Comments
I guess this is a repeat of #965. However, I really do think this should be changed, regardless of if people who have access to see the annotation would have access to the values or not. The difference is when doing an |
I'm not sure that pulumi that creates |
@confiq I'm aware. I wasn't stating that pulumi is creating it, just that creating a secret with pulumi is creating the annotation. |
This is a tricky problem as mentioned in #965 and I'm open to ideas about solutions. For those not following, this is a downstream consideration with kubectl apply and has been set as wontfix/by design. We have a few non ideal solutions:
It's worth noting that Considering this is set as "by design" upstream I'm inclined to close it, but happy to take feedback/considerations. |
I don't think I saw an explanation as to what we lose by switching away from the declarative method? |
I don't actually know how we would be able to switch away from the declarative object configuration. The Kubernetes imperative management has no knowledge of the previous state so it would be a very dramatic change in how we operate on the cluster. As an example: If we use |
Why would pulumi need to import the state? Create secret -> secret stored in statefile -> update secret -> check diff against statefile -> diff detected -> replace secret |
Sorry if it wasn't clear, that's largely what I meant by saying "import the state". |
The thing is it wouldn't just benefit |
That's totally understandable. It's not clear how we work around this upstream limitation at this time, due to how we actually perform the apply logic. |
The entire concept of the last-applied-configuration is such a k8s screw-up IMO. I don't even know where to begin. There's really no reason why this value cannot simply be a hash or checksum of sorts, and the actual content logged to the API servers somewhere if desired. This type of problem extends way beyond Pulumi. It's almost like they designed this really cool declarative config system... and forgot about Secrets, then decided to tell users "Well you should really not use apply with secrets..." Excuses.. excuses... there's plenty of ways this situation could have been avoided/protected from snooping. If the real values needed to exist then the values could simply be encrypted using the API servers certificates some how. Ah well.... pipe dreams I guess... since k8s doesn't yet have an access level that separates "get metadata" vs "get entire object" it's kind of a non-issue, but the day that granularity comes [if it does] this issue needs to be revisited by the k8s core team again. It's like 6 year old issue... not sure why hashing the value 'breaks' any of this but I digress. |
Issue is there but you can remove the annonate using kubectl annotate secret <secret-name> kubectl.kubernetes.io/last-applied-configuration- -n <namespace> |
This is a sufficient workaround for CI/CD processes, but would be best to not have it at all. The other problem with last applied is it limits configmaps to half of the actual available capacity. Although, one can argue having very large configmaps is probably not a great idea, it would be ideal if Pulumi could support some sort of patch capability instead of relying on apply-only. |
This should be fixed now that we automatically mark Secret |
To clarify, it's fixed in how pulumi displays it, or it's fixed as in pulumi no longer uses apply and therefore the annotation is not created? From what it sounds like, it's only fixed in how pulumi displays it, which should mean this issue should remain open then. |
Oh, I missed your earlier comment about the concern being the raw values being viewable from Kubernetes directly. From the Pulumi perspective, those values will always be encrypted now, but are still visible in the annotation with e.g., I'll reopen this for now, and it would also be addressed by #1659 in the future. |
I have a stack with about 200+ resources and somehow every single one of them has a ciphertext on the last applied configuration. When I switched from using hashivault -> default (service)... it made my stack previews take 5+ minutes. Is nobody else having this problem? This seems like an oversight to blanket encrypting the secrets for this, as now every single resource's last-applied-configuration will require encryption and the pulumi service with the default key seems too slow to be up for the job. |
This is most likely due to pulumi/pulumi#6445, which was fixed in the most recent CLI release https://github.com/pulumi/pulumi/blob/master/CHANGELOG.md#3230-2022-01-26. Are you using the latest and still seeing this? |
This did not seem to make a difference. In fact, converting from hashivault -> default took 5+ minutes, again, the exact same time. I even did some updates/previews against the state with 3.23.2+ so it should have cleaned out anything in the stack file if that was a requirement for the fix to take hold... This issue is specifically to EVERY resource that has a I posted a gist to the issue I'm describing, which I think is the root of the problem here: Pulumi must be "detecting" a secret in every one of these Deployment resources (200+ in my stack) and so it's just blindly marking the whole annotation data IN THE STACK as a secret... normally that sounds great, but,.. in reality, it's probably just pulumi detecting a poor secret combination that happens to match and just start masking everything it sees... I suspect this is a bigger issue at play all over the place and I'm hitting one of its edge cases. adding in here, if I attempt to export the stack with --show-secrets true, it takes just as long as it does to get to the first stage of the preview/update/whatever operation against the |
Fixed in #2445 |
Problem description
Creating a Secret with pulumi is creating the
last-applied-configuration
annotation. This contains the raw values of the secret.Maintainer note: These values are automatically encrypted in Pulumi state; the plain text values are on the actual Kubernetes Secret resource.
Suggestions for a fix
According to my research
kubectl apply
should not be used with secrets. I am not sure if this is what pulumi is using or not, but it seems like a possibility.The text was updated successfully, but these errors were encountered: