Support Vault as a backend for st2kv

[cydergoth]

Vault is an easy sell as a verified secure KV store in high security environments. It would be nice to be able to use a Vault deployment as a backend for st2kv.

[arm4b]

Currently, we have MongoDB as a backend for K/V store.
The proposal as I understand is to make the K/V backend configurable (Vault, etc).

@StackStorm/tsc @StackStorm/contributors any ideas about the feature and possible implications?

[cognifloyd]

I would love easier access to vault secrets, but I don't think changing st2kv is the way to do that. I think we should have a new vault function available in the same contexts where st2kv is available.

[cydergoth]

I would love easier access to vault secrets, but I don't think changing st2kv is the way to do that. I think we should have a new vault function available in the same contexts where st2kv is available.

So long as there is a way to disable the built in key store and demonstrate that to an auditor I'd be fine with that

[cydergoth]

..actually, on second thoughts, that wouldn't work for third party packs. It needs to be an API compatible replacement

[cydergoth]

So to clarify, FedRAMP has some very particular requirements around cryptographic algorithms used. It's a lot cheaper to tick a box for an existing, well known product than to have to prove compliance on a custom crypto impl.

[punkrokk]

@nmaludy and I talked a lot about this. We don't think it's hard, per sé, but if I recall correctly it was a function of making the Vault keystone available everywhere.

What seemed the easiest was to decorate the existing KV Astor's functions with something like a vault=true parameter. Then have a child class that implemented the appropriate API calls to vault.

There also needs to be a secure way to get the vault secret loaded into st2. The preferred way is via an instance store IAM role, but that's not available to everyone.

I seem to think there was one other blocker.

FWIW, there are a lot of ways to leverage vault in non-native ways ( eg vault agent and vault sidecar injection via annotations in K8s) that beg the argument is the integration worth it compared to the other tech debt that exists in ST2.

But... they do crypto right. And not a terrible keystone, but not built for performance like consul/redis. It's a secret store, not a plain vanilla KV store, though people do use it that way.

[cydergoth]

@nmaludy and I talked a lot about this. We don't think it's hard, per sé, but if I recall correctly it was a function of making the Vault keystone available everywhere.

What seemed the easiest was to decorate the existing KV Astor's functions with something like a vault=true parameter. Then have a child class that implemented the appropriate API calls to vault.

There also needs to be a secure way to get the vault secret loaded into st2. The preferred way is via an instance store IAM role, but that's not available to everyone.

I seem to think there was one other blocker.

FWIW, there are a lot of ways to leverage vault in non-native ways ( eg vault agent and vault sidecar injection via annotations in K8s) that beg the argument is the integration worth it compared to the other tech debt that exists in ST2.

But... they do crypto right. And not a terrible keystone, but not built for performance like consul/redis. It's a secret store, not a plain vanilla KV store, though people do use it that way.

So my feeling on this is that a generic API inside st2kv to call a 3rd party class a la the auth api would work. St2 shouldn't have to worry about key loads into vault; anyone using vault will be deploying it somehow and that process can handle the key loads. Making it a pluggable API means that people in K8s can use sidecar, other scenarios can implement their own solutions. It would also cover things like AWS KV and GCP KV if it were pluggable.

But whatever impl, I think it has to be a transparent dropin for st2kv so it is compatible across any 3rd party stuff assuming the st2kv functions

[cydergoth]

I have also just noticed that the st2 stackstorm-ha helm chart puts the secrets into Kubernetes "secret" store which is unfortunate as it is only b64 encoded there, not encrypted

[chris3081]

I agree a plugin is a good way to cover this and allow for more then just hashi vault, though I think it would be simpler to leave it to the plugin to organise its own token be that from a file like st2kv or from an instance profile/Azure MSI or any other trust system. I think this will allow for an easier implementation.

[mjtice]

We've written a custom vault function within our deployment that makes secret retrieval from Vault a breeze. IIRC, @xorso brought it up a while back (maybe in the Slack channel?) but because it requires upgrading Oslo it was turned down.

[chris3081]

How easy is it to add to the jinja filters? So at the moment you have {{ st2kv.* }} is it difficult to add jinja filters to the engine for say {{ hashivault.* }} or {{ mycustomsecretengine.* }} ?

[mjtice]

I'm not sure I understand, but here's an example of a task:

get_device_id_from_fstab:
    action: core.remote
    input:
      cmd: grep -iw <% ctx().mount %> /etc/fstab | awk '{print $1}'
      hosts: <% ctx().cdb_host %>
      username: oracle
      password: "{{ vault_get('data/dbsa/operations/test/oracle', 'st2.dba_refresh', mount='kv') }}"
      cwd: /tmp/
      timeout: 180

Where the first param is the secret path (in this case it's a kv2 secret), the second parameter is the st2.kv entry that holds the approle, and the last parameter is the secret mount.

[cydergoth]

This is good, but doesn't address the case of using a thirdparty pack which uses st2kv and transparently mapping that to vault.

[nzlosh]

+1 with @cognifloyd. Vault or any external secret provider would require a new API.

[chris3081]

I was looking at this over the weekend, would anyone object if we left st2kv as it was? Then created a new class called st2secret and then an abstract class you can write backends to? Doing this would mean st2kv users wouldn't have a breaking change, the community could then make a decision to move st2kv into the new framework if there is a desire.

[cydergoth]

Is there any reason we can't make st2kv a shim to different backends? How do we ensure that third-party packs using st2kv get appropriately redirected to vault to get their secrets?

…

On Sun, Feb 20, 2022, 4:41 PM chris3081 ***@***.***> wrote: I was looking at this over the weekend, would anyone object if we left st2kv as it was? Then created a new class called st2secret and then an abstract class you can write backends to? Doing this would mean st2kv users wouldn't have a breaking change, the community could then make a decision to move st2kv into the new framework if there is a desire. — Reply to this email directly, view it on GitHub <#5401 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPWAWSAD622K2FM6IIFPVDU4FUZDANCNFSM5GMA7HZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[chris3081]

The main concern I have is looking at the references to the st2kv code there seems to be a lot of dependencies on it. The st2 Key Vault also has a wider scope of use cases it can host unencrypted keys as well. Whilst I'm not opposed personally and in the past have suggested exactly what you have I do think there is some difference between the 2.

For example st2kv uses mongo to hold the keys and pulls the symmetric key from a file on the disk. In the secrets plugin IAM would likely be used to remove the need for persistent token storage and secrets would ideally be pulled at the last moment and only be in ram for a short time, in either case this would be down to the plugin to manage.

[rush-skills]

Can this be implemented as a config for st2kv backend, where we can choose between mongo/vault/(new-tool) and change internal st2kv implementations to shims for kv backends. This can also add an option to handle secrets with file-based token encryption (like right now), or let the backend provider handle it? Maybe even allow secrets in st2kv to be handled in a different store (vault) than normal keys (in redis/consul), and allow secret encryption using data store keys only if needed by the secret backend.
Right now, I am syncing my internal secrets to st2kv secrets and then using them inside st2, but ideally I would like to avoid the duplication and use the secret backend directly (vault etc.) in this case.

[chris3081]

My thoughts on this is we repeat the design pattern used in st2auth, but yes that's essentially what I would like to do too.

[jficz]

I'd like to see Vault as st2kv backend as well but I can also see tha the feature parity doesn't align.

I see another option here - do not use Vault as a backend for the kv storage but rather as a backend for encrypted values. Vault has something called "transit engine" which essentially works as "encryption-as-a-service" which could be leveraged for encrypted vaules while keeping the unencrypted vaules untouched.

The downside of this is that one would still need to manage the content of the st2kv store independently though.