Adds design doc for bundle CRD

[JoshVanL]

Signed-off-by: joshvanl vleeuwenjoshua@gmail.com

[jetstack-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JoshVanL

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JoshVanL]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SgtCoDFish]

I've left a bunch of comments for discussion; this is a great starting point and it's an interesting problem space.

(Feel free to ignore the nitpicks, they're not important but I thought it would be worth highlighting them)

[SgtCoDFish]

suggestion: I think Bundle would be tremendously improved by supporting the ability to include publicly trusted CAs, such as Mozilla's list. I worry that if we don't support a publicly trusted list, people will hack around it and get themselves into all kinds of trouble by statically including a bundle which will inevitably never get updated.

I don't know if such a publicly trusted list is required for a v1 release, but I think it would be a huge improvement to have it in place; it'd enable Bundle to essentially be the one-stop-shop for all trust needs for the vast majority of workloads.

[JoshVanL]

In two minds about this.

While I absolutely agree this would be a great addition, I feel we run the risk of;

1. The baked in trust bundles becoming out of date meaning the tool has an expiry data if left unmaintained,
2. Open a flood gate of "also add this default trust store" feature creep.

[JoshVanL]

Thank you for the review @SgtCoDFish! Hopefully have covered all your comments

[SgtCoDFish]

Replying to this here because it'll be easier to track. This is quite long; maybe we should have a call to discuss this.

While I absolutely agree this would be a great addition, I feel we run the risk of;

1. The baked in trust bundles becoming out of date meaning the tool has an expiry data if left unmaintained,
2. Open a flood gate of "also add this default trust store" feature creep.

I definitely understand why you'd be in two minds, but I'm not hugely concerned by either of these things.

1. The baked in trust bundles becoming out of date meaning the tool has an expiry [date] if left unmaintained,

I think there's a case to be made for allowing the tool to dynamically fetch updated trust stores at runtime, although there are definitely questions which would need to be answered to implement that (where to fetch from, how often, how to surface an updated trust bundle to cluster admins, etc).

Whether or not we implement dynamic updating at runtime, I don't think most cluster operators would be overly concerned by using a baked trust store, since a lot of them are just going to be trusting whatever's provided in their container images, be that alpine, centos, debian, whatever.

By "centralising" on cert-manager/trust, they potentially only have to update one trust store (via a Bundle) rather than having to keep track of the disparate trust stores in their running applications on different distros.

Plus if we stopped maintaining cert-manager/trust, I think they'd be fine; it's open source and they could easily create new versions with updated baked trust stores.

2. Open a flood gate of "also add this default trust store" feature creep.

In theory I'd be concerned about this, but I don't think in practice there are actually many lists of publicly trusted roots that people would want in the first place. Most Linux distros that I know of just use Mozilla's list, and the only other major list I'd expect to hear about is Chrome/Chromium's. If in the worst case people start requesting a bunch of stuff we don't want to add, I'm personally quite comfortable saying "no" 😉

UX Without Publicly Trusted Roots

My fear is that if we don't add support for a publicly trusted list of roots, the UX will suffer massively. The ideal UX for TLS clients connecting to servers IMO is:

• Create Bundle resource + mount resulting target ConfigMap into their containers
• Use TLS as normal both with public and private CAs

That is, they could mount the trust store and replace the system trust store for their container - so they'd not need to modify RootCAs in tls.Config, --ca-file in curl or whatever the equivalent is for their language / tool.

They could combine the mounted ConfigMap and the system trust store in the container, but that requires them to run something like this at container start-up time, which is a pain and is error prone.

Basically, my vision here is that cert-manager/trust could be the one-stop-shop for trust everywhere in a cluster, providing visibility and a central place to update trust stores.

[JoshVanL]

/retest

[SgtCoDFish]

/lgtm

I don't think holding this design doc PR open is providing any value - we can always discuss potential features in their own issues. Specifically I'll create an issue from this comment.

In any case, this doc is really good and well written as-is, so let's merge!