✨ Ignition: add option to store User Data in plain text

[damdo]

What type of PR is this?
/kind feature

What this PR does / why we need it:

For Machine bootstrapping CAPA supports one main method: CloudInit and one experimental method: Ignition.

For the former, CAPA used to store the User Data that the bootstrapper would use to fulfill its job in the EC2 Instance User Data in plain text.

This was not considered the most secure way to provide this configuration to a machine as a process/subject with the power to query the instance metadata could be able to read this config, which may contain certs, keys and other forms of secrets.

As such to minimize this issue, a new default method was added, where the controller would now leverage AWS SSM to encrypt the user data before storing it in the EC2 Instance User Data. The previous method of storing User Data in plain text was kept available as an alternative option to users, which would be able to access it by providing the InsecureSkipSecretsManager option in the AWSMachine spec.

Later on, when the new Ignition bootstrapping support was added, the creators, already aware of the security implications of storing User Data in plain text,
decided to opt for a more secure default. After briefly looking at using AWS SSM encryption also for Ignition, they acknowledged that it wasn't possible due to Ignition's lack of support for multi mime types. Hence they went for using S3 Buckets to store the config in objects, and have ignition fetch it at bootstrap time from there, and this did the trick.

Up until now this has been the only option to achieve Ignition bootstrapping.
However even if this is a better option from a security standpoint than storing User Data in plain text, and as such the preferred option for a good default,
it requires the ability to provision/access S3 Buckets, which is not always possible/desirable in specific scenarios, as well as not suitable for certain type of bespoke CAPI bootstrap providers.

As such we would like to introduce for Ignition, similarly to what is available for CloudInit, the ability to store User Data in the EC2 Instance in plain text.
Of course we know this is not the best option from a security standpoint, and as such we want to make it a fallback option that users can leverage if their use case does not allow the use of S3 buckets. We mentioned in the docs, and in the API that this option is discouraged and we trust CAPA users will use this with care.

Special notes for your reviewer:

Checklist:

• squashed commits
• includes documentation
• includes emojis
• adds unit tests

Release note:

Ignition: add option to store User Data in plain text

[damdo]

/cc @JoelSpeed @vincepri @nrb

[JoelSpeed]

We should discuss with other maintainers before making any changes, because the code here LGTM without any modification.

However one thought occurred that I thought I would raise. The current mode, providing no modifications, is almost like a passthrough mode. I'm wondering if passthrough may be a better way to describe it, as users may have got their own system for encryption or whatever/things may evolve in the future to allow future expansion. Having a default that is passthrough, makes it open to any future modification since it's clear that the content will just be passed through to the metadata without any modification.

Could be more future proof than PlainTextUserData

[damdo]

/test pull-cluster-api-provider-aws-test

[dlipovetsky]

If the user is selecting a storage mechanism, then I think the two choices are:

• S3
• Instance Userdata

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-add-user-data.html

So I think InstanceUserdata would be a good name.

[richardcase]

/milestone v2.4.0

[Ankitasw]

Agree with what @dlipovetsky suggested.
This PR overall seems ok to me if we have a valid use case around this.
Overall code is also good, some small nits/questions from my end.

[damdo]

@dlipovetsky @richardcase as discussed during the CAPA meeting I changed the storageType value from PlainTextUserData to InstanceUserData (cc. @JoelSpeed)

@Ankitasw I've addressed your comments 👍

Thanks all for the input here.

[damdo]

Found the issue with the deprovisioning, let's see if E2Es pass now:

/test pull-cluster-api-provider-aws-e2e

[damdo]

@vincepri @nrb @richardcase "old" e2e passed 🎉
I'll re-add the new e2e case for unencrypted UserData, and see if that works.

[damdo]

/test pull-cluster-api-provider-aws-e2e

[vincepri]

Seems another test is failing:

STEP: Waiting for the workload nodes to exist - /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.6.1/framework/machinedeployment_helpers.go:106 @ 02/08/24 15:22:47.926
[FAILED] Expected
    <string>: cloud-config
to equal
    <string>: ignition
In [It] at: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/suites/unmanaged/unmanaged_functional_test.go:1105 @ 02/08/24 15:23:48.067

[damdo]

Seems another test is failing:

@vincepri
Thanks, yes I noticed that previously and fixed it 👍 We should be quite close to succeeding now 🤞
Let's see what the pending e2e run reports.

[vincepri]

/lgtm

Great work!

/assign @Ankitasw @richardcase
for approval

[Ankitasw]

/approve

Thankyou @damdo for your work 👏

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ankitasw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Ankitasw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment