Extend secrets store abstraction to include ability to write secrets via Concourse

[andy-paine]

Would there be appetite for a fly add-secret command (and fly remove-secret) to re-use the global secrets store abstraction? The main benefits I could see for an end-user is:

• Don't have to know which secrets store is in use
• Don't need credentials for the secrets store
• Don't need the CLI or UI for the secrets store
• Don't need to know about what path to store secrets in (e.g. could just be fly -t myteam add-secret --name foo --value bar --pipeline mypipeline -> stores it in /concourse/{{.Team}}/{{.Pipeline}}/{{.Secret}} or whatever is configured on web node)

From an access perspective, Concourse would probably need a reasonably permissive set of credentials for the secrets store as it will need to write to every team (per team creds maybe if this is a concern?). I figured if access is largely done on a Concourse RBAC level (admins, owners and members(?)) and then include an OPA integration to allow for finer grained control.

I don't know what the future of global credential management looks like given var_sources is now available. My gut instinct says that there is still a need to global creds as having every pipeline configure access to a store would be awkward - in which case I think extending the abstraction down to writing creds could be useful, especially in an environment where the users of Concourse != operators of Concourse.

[vito]

Yeah, I think this would be super convenient. I opened an issue to float the idea a while back and it didn't get a whole lot of love (#2075). @zmb3 brought up a valid concern about the dangers of such a feature, so that's still something to keep in mind, though I think you're on the right track - maybe we can RBAC or OPA our way out of it.

re: var sources vs global, I think var sources are The Future™, but to get there we'll need to implement cluster-wide var sources (concourse/rfcs#39 (comment)) and/or project-level (concourse/rfcs#32) var sources which are inherited by pipelines within the project. It is indeed awkward to carry around var sources in each and every pipeline.

Thinking ahead to the future, where I hope to have credential managers implemented with Prototypes (concourse/rfcs#37), setting credentials would involve a container action, so we'd probably need some sort of async flow for fly add-secret (fly set-secret?). This would also imply some sort of write Prototype action.

Might also be interesting to think about the various different "types" of credentials which may be written - some credential managers, e.g. CredHub, support different types (e.g. user, password, cert) which can be automatically generated and/or rotated by the credential manager so that they never even have to be specified client-side. I'm not sure how widespread that is (haven't compared all the various cred managers), but it seems like a nice feature to support.

Thanks for bringing this up!