Proposal: Supported Plugins - Internal & External

[cipherboy]

Plugin Proposal

This proposal outlines plugins to support, based on the four major types of plugins (auth, secrets, database -- a specialization of secrets, and storage).

The philosophy of this plugin reduction is as follows:

1. Minimize maintenance burden on the open source community.
2. Prioritize community supportable & debuggable OSI licensed integrations over prorietary integrations.
3. Become opinionated on an approach to plugins in general.
4. Allow a broader community to maintain external plugins.

In particular, this means that plugins previously built-in like database/mongodb no longer are first-class supported as they are not under an OSI license. Similarly, all cloud vendors' plugins have been moved external as well, for the community to support independent of the main core release cycle.

This proposal suggests aligning with Lars's proposal to move builtin, supported plugins into the main repository: these builtin plugins will be released with the binary and thus versioned similarly. External, community-supported plugins can be in separate repositories released as separate binaries & container images, allowing additional maintainers to be added. This has the added benefit of not tying these external plugins to the main repository's release cycle, allowing contributors to release as they see fit and not holding up the main release on third-party dependencies.

(To be clear -- For non-OSI integration plugins for which the broader community is interest in maintaining them, the suggested mechanism is via a separate repository, building external plugins (rather than built-in to the release binary). This will allow additional maintainers to be nominated (without commit access to the main repository) and allows for a separate release schedule more suitable for this plugin. This is to say, the broader OpenBao community can support additional proprietary plugins, but the core release artifacts and release cycle will not include these by default.)

This should greatly reduce the binary size as well, as many dependencies can be removed (as, for instance, the Azure SDK and other cloud provider SDKs can be removed as a dependency, which greatly contributed to binary size -- see https://github.com/hashicorp/vault/issues/22893#issuecomment-1712569767).

Lastly, storage plugins were removed. Only two types of plugins were officially supported in an Enterprise context upstream: Consul and Raft. Consul is no longer OSI-licensed, and no fork has been made aparent at this time. Other plugins lack features that Raft has (in the Vault Enterprise variant), and support for multiple backend storage plugins has caused other issues. Thus, the decision to simplify to Raft aligns more closely with what upstream has done in practice and greatly reduces our maintenance burden. In-mem and file backends are left for historical testing reasons and will not be supported in production (non-dev-mode) usage.

See repos.md for context on the different types of plugins.

Auth

name	repo
approle	github.com/openbao/openbao/builtin/credential/approle
cert	github.com/openbao/openbao/builtin/credential/cert
jwt	github.com/hashicorp/vault-plugin-auth-jwt
kerberos	github.com/hashicorp/vault-plugin-auth-kerberos
kubernetes	github.com/hashicorp/vault-plugin-auth-kubernetes
ldap	github.com/openbao/openbao/builtin/credential/ldap
radius	github.com/openbao/openbao/builtin/credential/radius
token	github.com/openbao/openbao/builtin/credential/token
userpass	github.com/openbao/openbao/builtin/credential/userpass

Database

name	repo
cassandra	github.com/openbao/openbao/plugins/database/cassandra
influxdb	github.com/openbao/openbao/plugins/database/influxdb
mysql	github.com/openbao/openbao/plugins/database/mysql
postgresql	github.com/openbao/openbao/plugins/database/postgresql
redis	github.com/hashicorp/vault-plugin-database-redis

Secrets

name	repo
database	github.com/openbao/openbao/builtin/logical/database
kv	github.com/hashicorp/vault-plugin-secrets-kv
kubernetes	github.com/hashicorp/vault-plugin-secrets-kubernetes
openldap	github.com/hashicorp/vault-plugin-secrets-openldap
pki	github.com/openbao/openbao/builtin/logical/pki
rabbitmq	github.com/openbao/openbao/builtin/logical/rabbitmq
ssh	github.com/openbao/openbao/builtin/logical/ssh
totp	github.com/openbao/openbao/builtin/logical/totp
transit	github.com/openbao/openbao/builtin/logical/transit

Storage

name	repo
raft	github.com/openbao/openbao/physical/raft
file	github.com/openbao/openbao/sdk/physical/file
inmem	github.com/openbao/openbao/sdk/physical/inmem

go-kms-wrapping

Initially no plugins will be removed from this, but the package will remain as a separate repository. However, if PKCS#11 bindings are contributed, this could warrant a migration from native seals to PKCS#11 libraries to reduce binary size and avoid the need to maintain each cloud provider's integration separately. This has the added benefit of easing cross-cloud migrations and being testable locally with SoftHSM.

Net Additions and Removals

Additions to Main Repo

• auth-jwt
• auth-kerberos
• auth-kubernetes
• database-redis
• secrets-kv
• secrets-kubernetes
• secrets-openldap

Removals to Community Supported

• auth-alicloud
• auth-aws
• auth-azure
• auth-centiry
• auth-cf
• auth-gcp
• auth-github
• auth-oci
• auth-okta
• database-couchbase
• database-elasticsearch
• database-hana
• database-mongodb
• database-mongodbatlas
• database-mssql
• database-redis
• database-redis-elasticache
• database-redshift
• database-snowflake
• secrets-ad
• secrets-alicloud
• secrets-aws
• secrets-azure
• secrets-consul
• secrets-gcp
• secrets-gcpkms
• secrets-monogodbatlas
• secrets-nomad
• secrets-terraform
• storage-aerospike
• storage-alicloudoss
• storage-azure
• storage-cassandra
• storage-cockroachdb
• storage-consul
• storage-couchdb
• storage-dynamodb
• storage-etcd
• storage-foundationdb
• storage-gcs
• storage-manta
• storage-mssql
• storage-mysql
• storage-oci
• storage-postgresql
• storage-s3
• sotrage-spanner
• storage-switf
• storage-zookeeper

[cipherboy]

@DanGhita writes on the draft version of this proposal:

Hello Alex,

For the storage plugins, shouldn't we keep at least one or two (open source) database storages?
As PostgreSql, for example. That might be helpful in implementing High Availability Vault services.
Many thanks,
Dan

[cipherboy]

@cipherboy replies:

---

@DanGhita Hi Dan,

Raft is kept and would suffice as a HA service.

IMO, being opinionated about storage would be in our benefit; many things cannot be implemented because it is difficult to support across a diverse set of backends. We could choose e.g., PostgreSql, but I'd suggest dropping raft if we do agree on that: Raft is a K/V store, but Postgres is a relational database... Choosing a single type of storage backend would be most ideal IMO... And we're presently mostly assuming K/V storage semantics, hence my suggestion to only keep Raft.

Further, upstream builds upon Consul (no longer open source) and Raft as the two officially supported backends for its Enterprise distribution, so making a similar choice would hopefully result in us aligning better with the existing code & many assumptions.

[DanGhita]

Hi Alex,

IMO, the interest in supporting a DB is that you can "inherit" from the DB built-in HA features (including Disaster Recovery).
But I understand your point, indeed, we probably should focus on the Raft (in the short-term).
However, trying to keep as much as possible the compatibility with some db-storage plugins could be useful in the long term.

And actually, I have an additional issue/misunderstanding related to the Raft and the HA: it is not completely clear for me which is the right workflow for joining the Raft cluster and auto-unsealing (when configured in auto-unseal with third-party KMS).
Concretely, I have the leader initiating and auto-unsealing fine. On the other hand, the (first and the others) follower will successfully join the Raft, but it will remain sealed. I can also see that the follower is able to reply to the join challenge received from the leader (and it can do this while it is still sealed ?!) .
For summarizing, I found this way to achieve the HA goal too complicated, while I think this method is far more natural. But probably, I’m missing something.

[cipherboy]

@DanGhita Do you mind posting what you're running and the error you're seeing? Perhaps in shell form so I can try to reproduce?

I must admit, I'm not as familiar with HA mode; I've mostly used Enterprise's replication modes as they're generally more useful than OSS's HA mode (the latter's additional nodes cannot service requests at all).

I can take a look after I finish getting the test suite working again... :-)

[DanGhita]

@cipherboy Actually, there is no (visible) error, but the issue is that the followers remain sealed after successfully joining the leader. As explained, all nodes (leader and followers) are configured with third-party KMS auto-unseal and they are all using the same KMS Key (seal configured with the same Key ID).
But don’t ‘worry, I will investigate by myself, I’m able to debug the code and I’m pretty curious to fully understand the internal core state-machine when it comes to deal with the (auto) unseal and raft/cluster join.

[JanMa]

I would personally like it if we could keep the auth methods for the major cloud providers but apart from that I agree with the approach to only support OSS plugins. This moves a lot of functionality to external community supported plugins, but allows us to properly maintain the ones we support.

One question I have left is where we draw the line wrt. OSS? Does that mean officially supporting source available projects like Elasticsearch or Consul is something we won't do?

[cipherboy]

To be clear, I think the community can and perhaps will support more non-OSS integration plugins, I just would like them to be external and not part of the main binary, even if they are part of the OpenBao namespace on GH. Any support would thus be decoupled from the main binary, and changes to the main binary would not necessarily be blocked on the wider ecosystem adopting them.

Considering Conusl and Elastic use the same license as Vault does now, and Vault's change is why we're here... I'd propose using OSI-approved licenses as the bar for inclusion here. The BUSL is not OSI approved and thus widely regarded as not being open source. So, they should be external, IMO.

Perhaps the bar should be: can a suitable staging environment for all features be tested using OSI licensed components?

This excludes cloud providers who publish OSI licensed, non-working (Mock APIs only) test containers, but allows things like, e.g., open core (given that the core has most of the functionality we need) and SaaS versions of open source running on proprietary cloud providers (e.g., managed MariaDB).

Edit to add: suppose we want to make some change that requires changes in both the core and in plugins. Let's use paginated lists for an example of one feature that's greatly requested upstream on OSS. There's some expectation, by the user base, that shipping this feature implies updating plugins to support it. By reducing the support burden of the core, we can ship the feature sooner, when the core builtin plugins are ready. For external plugins, we can file an issue, and should it be maintained, the maintainer or the community can pick it up and work on it on their own schedule, without blocking our ability to release the new core version with that feature.

OSI is a suitable criteria, as it most optimally allows us to be confident in the quality of the artifacts we release, without having to potentially spend personal or community money on such.

Obviously the logical extreme is to make nothing internal, but there are some optimization to updates and overhead that make being internal worthwhile.

This also maximizes our distributability, letting us potentially land in OSI-sensitive places such as distros or homebrew, with minimal differences to an upstream binary.

My 2c.

[cipherboy]

@JoeMurray wrote:

https://docs.percona.com/percona-operator-for-mysql/pxc/encryption.html is the only MySQL / MariaDB / Percona open source version that makes use of Vault. I'm a bit unclear on who maintained the keyring_vault plugin, but I believe it was Vault not Percona.

[cipherboy]

@JoeMurray As far as I can tell from that page, the keyring_vault plugin is a Percona plugin, not a Vault/OpenBao plugin. It talks to the K/V plugin, which is not going away, but also not... optimal. See the configuring Vault section:

$ kubectl exec -it vault-0 -- /bin/sh
$ vault login s.VgQvaXl8xGFO1RUxAPbPbsfN
$ vault secrets enable --version=1 -path=pxc-secret kv ### <<<--- HERE 

This means that keys are stored in the equivalent of text blobs, with no access control limitations: Percona must read the key on startup and caches it locally in memory, performing all operations locally. In contrast with say, a PKCS#11/KMS workflow (made possible with the Transit plugin), where the key would be present only in OpenBao and wouldn't allow export in plaintext.

All this to say, this workflow won't go away, but might also suggest Percona revisits how it accomplishes this. My 2c., though.

(ended up copying your comment here, I think you had loaded the page before I had made the move to here based on getting rough consensus on the proposal).

[naphelps]

Proposal Accepted 2024-02-01. Plugins will be pulled in-tree with the ability to fork as external repositories long-term should the community purse the the use of a registry.

[JoeMurray]

The proposal was based on sound principles and I'm glad it passed.

I am interested in understanding the functionality that the OSS database plugins that remain in the openbao repo provide, specifically the mysql one. The files are terse https://github.com/openbao/openbao/blob/main/plugins/database/mysql/mysql-database-plugin/main.go . I'm happy to help create/improve documentation if I can be pointed in the right direction.

[cipherboy]

Thanks @JoeMurray!

You can find some documentation in-tree here:

• https://github.com/openbao/openbao/blob/main/website/content/api-docs/secret/databases/mysql-maria.mdx (upstream live version)
• https://github.com/openbao/openbao/blob/main/website/content/docs/secrets/databases/mysql-maria.mdx (upstream live version)

And some more general information on the database engines here:

• https://github.com/openbao/openbao/blob/main/website/content/docs/secrets/databases/index.mdx (upstream live version)

PRs welcome :-)

The README in the root of the /website folder will give you some guidance to build a local docs server to audit changes, but if you have make and Docker/Podman around, cd website && make should suffice.

---

The TL;DR is that these database plugins do identity: a service requests credentials for the database and OpenBao will generate short-lived, on-demand credentials. This replaces the need to put static credentials in a K/V store (and, hopefully sets up the org for better credential management: each instance of apps that talk to the database can get their own credentials and won't share them). These database plugins don't do data-at-rest encryption.

[naphelps]

Edited 2024-04-01:

Removed database-redis from the list of supported plugins (https://redis.com/blog/redis-adopts-dual-source-available-licensing/). LF Valkey (https://www.linuxfoundation.org/press/linux-foundation-launches-open-source-valkey-community) future replacement.

[cognifloyd]

What about Login MFA? Can external plugins provide the Login MFA feature (Okta, Duo, PingID)? Or will they continue to live in openbao core?

At least these files (plus tests) include Login MFA bits:

• https://github.com/openbao/openbao/tree/main/helper/identity/mfa/
• https://github.com/openbao/openbao/blob/main/ui/app/models/mfa-method.js
• https://github.com/openbao/openbao/blob/main/vault/login_mfa.go
• https://github.com/openbao/openbao/blob/main/vault/identity_store.go
• https://github.com/openbao/openbao/blob/main/vault/request_handling.go

[cipherboy]

\o hello @cognifloyd -- MFA was left in the repository for now, mostly as an oversight on my part as I incorrectly assumed it was part of the Vault Enterprise offering and thus not part of the OpenBao code base. I'd love to move this to a plugin model as well, to further reduce the core binary size and help others contribute to the ecosystem more easily.

Are you willing to help out in this area? :-)

[cognifloyd]

My interest stems from a PR I submitted upstream that adds Login MFA support for Okta TOTP (before my change, only Okta push was supported). I am looking at contributing that to openbao as my upstream PR is in limbo.

I am not linking to the PR to avoid how github advertises the link in the original PR.

I don't have time to figure out how to add a new plugin type, but would openbao be open to my contribution, even though Okta is a paid service, not an open source project?

[cipherboy]

To try and clarify our position... our objection is not with paid services.

(In my personal opinion) it is important that businesses find ways to be profitable, especially when they sponsor open source development.

However, for an unpaid, volunteer project, dependencies on paid services are hard to maintain, support, and guarantee the working of prior to releases. Putting these in externally maintained projects, with separate release timelines and external maintainers, helps us maintain a working ecosystem without bogging down release testing. If a bug is found, e.g., in Okta auth plugin, it would simply need a re-release of the external plugin which operators can pull into their environments. It wouldn't require the core project to re-spinning the entire release. It also lets us minimize the size of the main binary by avoiding larger dependencies (read: cloud SDKs).

IMO, this will give us a better user/operator experience, as individuals who wish to contribute to particular areas can mostly self-organize (under the guidance & support of the broader community & TSC). They can opt to have different release cycles that reflect their propensity to fix particular bugs or ship new features/integrations independently of the broader core release timeframe. My 2c.!

(And as mentioned in a few issues and private discussions, with a proper plugin registry, this should simplify the distribution & operational side of this approach as well.)

---

To actually answer the question, given the code is presently in the beta and likely won't be subject to removal without a concrete proposal towards external MFA plugins, I think there's a substantial chance we can review and merge it, yes.

Once we get such a proposal, having the code already landed should also mean it'll land in the external plugin-ized form of this, too. :-)