Proposal: Custom Compositions

[negz]

What problem are you facing?

Composition is the Crossplane feature that lets teams define their own opinionated APIs (i.e. Kubernetes CRs). We call these APIs "Composite Resources", or XRs for short. When a user creates an XR, Crossplane uses another kind of resource - a Composition - to determine what to do. For example a Composition might tell Crossplane that when a user creates an AcmeCoDatabase XR it should respond by creating a GCP CloudSQLInstance and a ServiceAccount. From our terminology documentation:

Folks accustomed to Terraform might think of a Composition as a Terraform module; the HCL code that describes how to take input variables and use them to create resources in some cloud API. Folks accustomed to Helm might think of a Composition as a Helm chart’s templates; the moustache templated YAML files that describe how to take Helm chart values and render Kubernetes resources.

A Crossplane Composition consists of an array of one ore more 'base' resources. Each of these resources can be 'patched' with values derived from the XR. The functionality enabled by a Composition is intentionally limited - for example there is no support for conditionals (e.g. only create this resource if the following conditions are met) or iteration (e.g. create N of the following resource, where N is derived from an XR field). These limits:

• Attempt to avoid the pitfalls of configuration domain-specific languages.
• Allow us to express "composition logic" as a custom resource that may be entirely stored within and validated at create time by the Kubernetes API server.
• Allow the Composition type to be relatively simple, avoiding it becoming a programming language expressed as YAML.

Limiting the functionality of our Composition type allows us to reduce Crossplane's learning curve and implementation complexity. At the same time it limits Crossplane's ability to address certain use cases. It's not uncommon for Crossplane users to resort to 'composing' Crossplane managed resources using more featureful tools such as Helm or jsonnet when they hit the limitations imposed by the Composition type. Furthermore, some Crossplane users simply prefer to express their intent in a language (or with a tool) they already know and feel comfortable with, such as Helm, jsonnet, Python, HCL, Typescript, etc.

How could Crossplane help solve your problem?

Crossplane could allow users to express "composition logic" using a tool of their choice. While it's true that users could simply use Helm, jsonnet, etc etc to compose Crossplane managed resources into higher level abstractions (like the aforementioned AcmeCoDatabase example), we feel that these abstractions are best exposed as APIs, not as client-side constructs.

Exposing abstractions as a (Kubernetes) API:

• Avoids the need to distribute (and version) client-side tooling as part of the collaboration process.
• Allows abstractions to be access controlled using RBAC.
• Increases tooling support - more tools have native support for calling REST APIs than (e.g.) building Helm charts.
• Further increases tooling support by leveraging the Kubernetes ecosystem - things like ArgoCD and OPA.

Crossplane could support users with advanced composition requirements (or existing tooling preferences) by supporting a "server side" BYO tooling approach to composition logic. The Crossplane maintainers typically refer to this idea as "Custom Composition".

With Custom Composition most of Crossplane's Composition machinery (XRDs, XRs, claims, etc) would remain in place, but the Composition type would be replaced by pluggable logic. A Custom Composition might look something like the following:

---
apiVersion: apiextensions.crossplane.io/v1alpha1
kind: CustomComposition
metadata:
  name: example
spec:
  compositeTypeRef:
    apiVersion: database.example.org/v1alpha1
    kind: CompositePostgreSQLInstance
  image: crossplane/example-helm-chart-composition:v1

The idea here is that composition logic would be implemented by an arbitrary OCI container. The container would accept a valid Crossplane XR as its (stdin) input, and output (to stdout) a (possibly mutated) version of the input XR, plus one or more valid composed Crossplane resources. This is not unlike the approach taken by KRM functions (which we may consider reusing).

Note that the name CustomComposition is not fixed, and in fact we may want to add this functionality to the existing Composition type such that users could choose between supplying an image or an array of resources and patches. The key reason to start with a distinct type is that it gives us the ability to introduce functionality at v1alpha1, and thus make breaking changes.

Some prior art on this topic:

• KRM functions
• This CDK-focused draft of a Custom Composition design document.

[negz]

Following up with some more thoughts:

---
apiVersion: apiextensions.crossplane.io/v1alpha1
kind: CustomComposition
metadata:
  name: example
spec:
  compositeTypeRef:
    apiVersion: database.example.org/v1alpha1
    kind: CompositePostgreSQLInstance
  # Take an array of functions so that we can stack/pipe functions.
  functions:
  - image: crossplane/example-helm-chart-composition:v1

We could also potentially use the config.kubernetes.io style annotations to indicate which output resource in the ResourceList is the XR, and which are composed resources. This would probably be config.crossplane.io or something similar under the crossplane.io domain though.

[jgillich]

Disclaimer: I've only been using Crossplane for a couple of days so I might get some things wrong here, but I thought I'd give my 2 cents on this matter.

I have a work-in-progress operator that is 90% composing resources and generating sensible statuses, and this is of course what Crossplane does really well so I'm considering adopting it. My initial expectation was that Crossplane might not provide everything I need but I'd be able to extend it where necessary. But despite my fairly basic requirements, I'm hitting some roadblocks that only seem solvable through a custom provider or controller that templates other provider resources for me, which doesn't play well with how compositions work and would make switching to Crossplane nonsensical altogether. One example is that I need to accommodate various helm charts and there are no standardized formats for secrets etc in helm land. Configuring a helm chart with a database user from provider-sql is often not possible because we can not access/transform secrets via patches (as far as I'm aware).

I think custom compositions are best thought of as an escape hatch, where you might not want to use it if you don't have to, but when you do it's because there are not other options. Being able to solve problems you don't yet know you are going to have is extremely valuable.

Some thoughts on the design:

• Functions are supposed to be deterministic and idempotent. Should they have apiserver access? I'd lean towards no, but there needs to be some way to read secrets, configmaps and perhaps any other api resource. We can have a reference syntax that provides values over functionConfig, something like this:

spec:
  functions:
  - image: crossplane/example-helm-chart-composition:v1
    functionConfig:
      # apiserver resource
      foo:
        from: 
          name: foo-resource
          namespace: example
          kind: Foo
      # static value
      bar:
        value: "bar"

• Are functions executed before or after patches are applied? I'd lean towards having functions be the final step but I don't fully understand the implications of either choice

• Using my helm example from above, would I be able to utilize a secret generated as part of my composition in a function that is called before the composition is applied? I assume not. So perhaps functions should also be able to specified on the individual resource level:

spec:
  resources:
    - name: db-role
      # ...
    - name: helm-release
      functions:
        - image: helm-only
      # ...
  functions:
  - image: all-resources

The execution order in this example being:

1. all-resources function
2. db-role patch
3. helm-only function (receiving only XR and helm-release via ResourceList)
4. ...

The key reason to start with a distinct type is that it gives us the ability to introduce functionality at v1alpha1, and thus make breaking changes.

Wouldn't we be able to do the same with v1alpha2 of Composition?

We could also potentially use the config.kubernetes.io style annotations to indicate which output resource in the ResourceList is the XR, and which are composed resources

Alternative suggestion, assuming the list is guaranteed to be ordered: The first item is the XR, the rest are resources. Or that plus the annotation.

[muvaf]

Slightly related, #2110 is also about customizing composition behavior but at a lower level that won't require you to give up using Composition completely.

[negz]

I'm adding this to the v1.7 release, which is due EOM. It's perhaps a little bit of a stretch to have an alpha implementation by that time, but I think it's possible.

[muvaf]

A few people asked in Slack, here is the old design doc for custom composition #1705

[janwillies]

It might be interesting to check how kustomize and kpt approach this:

• kpt functions: https://googlecontainertools.github.io/kpt/guides/producer/functions/
• Kustomize Plugin Composition API (#2299)

example for kustomize: https://youtu.be/YlFUv4F5PYc?t=1361

[negz]

I'm adding this to the v1.7 release, which is due EOM. It's perhaps a little bit of a stretch to have an alpha implementation by that time, but I think it's possible.

It was recently bought to my attention that v1.7 was accidentally slated for late March when it should actually be early March, so I'm going to remove this from v1.7. I think that's a bit aggressive.

[negz]

Hey folks! A few of us have started meeting to discuss this effort each week on Tuesdays at 9pm PST. If you’re interested in joining in and helping out please let me know and I’ll add you to the meeting invite. Our agenda and notes are at https://docs.google.com/document/d/1n8018YaDXCl_6E8y9J5untSZ4rH811dL75GPnERRQxA/edit#.

Edit: PS you can find us in #sig-custom-compositions on Crossplane Slack, too. :)

[negz]

Breadcrumbs to kptdev/kpt#2567, which discusses alternatives to containers as ways to run KRM functions.

[kzap]

Could I add my 2 cents here, and with the current draft you will end up making Crossplane turing complete at what cost? Every templating tool that has introduced constructs like this has had it misused and set some poor standards which make it harder for the eco system to stay simple and easy to use.

Please dont recreate DAGs or functions in YAML. That's already been done.

[thejosephstevens]

Hey all! We've started using Compositions as a key component for our cell-based architecture and there's a few things we're running into that I don't see simple solutions to today:

1. One common pattern we run into is that because we deploy resources into customer accounts, our access is restricted, and sets of resources can't be provisioned by us due to permissions constraints (e.g. IAM users, policies, etc). In circumstances like this with Terraform we have a pattern of "byoX" where you can pass config into the module like a VPC ID or user ID and internally to the module the count of the resource is set to 0 and a data reference is instantiated to fetch the resource by reference. It would be really helpful to be able to support a similar model in custom compositions (currently we're just looking at creating two compositions one for low-privilege resources, one for sensitive resources the customer can per-create).
2. For notation is valuable for cases where we want to instantiate a child composition of which there may be a variable number of instances with different configuration values. The hope would be that you could provide an array of objects and they could be mapped into instances of a resource. If this works similarly to Terraform's for_each argument, that would satisfy our use case. Similarly, passing an empty array of objects should work to deploy no instances of said resource.
3. We have notions of "default values" for different "sizes" of install that can change with some regularity. In terraform we implement this by having a two-layer module, where at the top level you can reference a size like "standard" or "ha", and that is used in the middle module to reference a set of default values for that size of install (each of which is overridable as a member of the environment_overrides object in the middle module). Right now our plan is to create default and overrides configs externally to crossplane that we would then merge into the XR declarations (making everything explicit in the claim instance), but this does require an extra layer of logic in our infra manager that we would be happy to remove. This falls into "nice-to-have" for us since we have a workaround.
4. We deploy across multiple different clouds in many different configurations and those config values can imply different resources being provisioned (e.g. s3 buckets in AWS vs GCS buckets in GCP). Right now we're solving this by just generating different compositions based on different high-level configurations, and keeping that logic external to crossplane in Jinja, but it would be desirable to be able to represent all of this in Crossplane and produce one OCI image that we could ship to any cloud and have it apply correctly by passing in aws vs gcp for cloud provider. This falls into "nice-to-have" for us since these are usually differences that can be known before deploying the composition to a cluster.

If I think of anything else I'll chime back in here, I'm really excited about the upcoming functionality!

[shanproofpoint]

Sorry if this question has already been addressed,

why couldn't we just expand on composition and just add these as yaml preprocessing ?
and if the functions are not specified, it will just work as it currently does today.

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[Feggah]

/fresh

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[elohmrow]

/fresh

[libracoder]

/fresh

[jbw976]

@libracoder Composition Functions (formerly called Custom Compositions) were released in v1.11 and are available to try out today! Check out this guide in the docs: https://docs.crossplane.io/knowledge-base/guides/composition-functions/

[libracoder]

Thank you @jbw976