[Cloud Provider][OpenStack] Support multi part mime for user-data

[openstacker]

Feature Request

Support multi part for user-data

Environment

Fedora CoreOS 30
https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-openstack.qcow2.xz

What hardware/cloud provider/hypervisor is being used to run Ignition?
OpenStack
KVM

Desired Feature

Support multi part for user-data so that OpenStack Heat can use Fedora CoreOS 30 (Ignition) along with Heat's SOFTWARE_CONFIG support.

Other Information

OpenStack Magnum is using OpenStack Heat to orchestrate the infra layer resources to deploy Kubernetes Cluster. Now Magnum would like to support Fedora CoreOS 30 as the k8s node operating system but without supporting multi part mime in Ignition making it hard.

[openstacker]

@steveb Please feel free to add your comments in case I missed something. Thank you.

[openstacker]

@strigazi I had a discussion with @steveb. And we think we could be able to use heat-container-agent as long as Ignition can support multi part mime.

[steveb]

As far as I can tell, the openstack provider expects the user_data to be JSON containing ignition format data. What we're requesting is that there is logic to detect whether the user_data is multipart-mime, and if so look for parts with a content-type of application/vnd.coreos.ignition+json to use for the ignition data.

@openstacker has a specific need which comes from how heat supports boot-time config (cloud-init/ignition) with deploy-time config.

I think this is also a feature which can help with transition away from cloud-init. This feature would allow a single multipart payload which contains both cloud-init parts[1] and ignition parts. This would make it easier for end-users to support booting with multiple different cloud operating systems.

I don't know if it is appropriate to do this at the openstack provider level, or elsewhere so other providers can support it too.

[1] https://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive

[lucab]

Thanks everybody for the the report and the additional details.

For context, is this specific to the HTTP metadata endpoint or does this affect the config-drive (or both)?

For the HTTP metadata, as far as I understand there should be no need for this. Ignition HTTP client is already attaching a proper Accept: header with a specific registered mime-type to each request, so that the server knows which content to serve (to the best of knowledge, this is effectively what Openshift4 server does already).

[steveb]

For context, is this specific to the HTTP metadata endpoint or does this affect the config-drive (or both)?

For the openstack provider, config-drive and http://169.254.169.254openstack/latest/user_data will return the same data, and sometimes that data will be in multipart-mime format

For the HTTP metadata, as far as I understand there should be no need for this. Ignition HTTP client is already attaching a proper Accept: header with a specific registered mime-type to each request, so that the server knows which content to serve (to the best of knowledge, this is effectively what Openshift4 server does already).

The openstack nova metadata server just returns whatever user_data is provided, it is not Accept header aware.

Can I suggest that multipart-mime support is added to the openstack provider first, then added to the other providers which might share their configuration data with cloud-init? I think these would be:

• AWS
• Azure
• GCP
• Packet
• DigitalOcean

The other providers have ignition/coreos specific mechanisms which won't need any change

[bgilbert]

The openstack nova metadata server just returns whatever user_data is provided, it is not Accept header aware.

If the metadata server returns whatever userdata it's given, could you elaborate on why multipart support would be useful? Fedora CoreOS will not process anything other than an Ignition config from userdata, and since userdata should be specific to an instance, it seems like there's no reason to include additional parts.

[steveb]

If the metadata server returns whatever userdata it's given, could you elaborate on why multipart support would be useful? Fedora CoreOS will not process anything other than an Ignition config from userdata, and since userdata should be specific to an instance, it seems like there's no reason to include additional parts.

Because userdata is the only mechanism to push config into an instance, it sometimes needs to be shared by more than one agent tool. Heat relies on the multipart support of cloud-init to add parts for its own agent framework (configuration which isn't boot-time). Without similar support in ignition, there would be no way to bootstrap heat's software configuration framework.

[bgilbert]

It appears that the agent framework is in Python? Fedora CoreOS doesn't ship Python and generally doesn't ship agents, so the agent would need to run in a container. Would it make sense to embed the SOFTWARE_CONFIG data in a file written by the Ignition config, which is then bind-mounted into the agent container? That would maintain Ignition as the sole bootstrap mechanism for configuring a Fedora CoreOS instance.

[openstacker]

It appears that the agent framework is in Python? Fedora CoreOS doesn't ship Python and generally doesn't ship agents, so the agent would need to run in a container. Would it make sense to embed the SOFTWARE_CONFIG data in a file written by the Ignition config, which is then bind-mounted into the agent container? That would maintain Ignition as the sole bootstrap mechanism for configuring a Fedora CoreOS instance.

Yes, the heat-container-agent will run as a container (just like it runs on Fedora Atomic now as an atomic system container).

The software deployments are not ONE file, they can be many files, so it's quite hard to include them into one Ignition's .ign json file. And one of the reasons why we need the heat-container-agent is 'breaking' the user-data size limit (in Nova it's 64K).

And I don't think heat-container-agent can parse those script correctly if we put those software deployments (scripts) into Ignition file. @steveb is it?

[strigazi]

Would it make sense to embed the SOFTWARE_CONFIG data in a file written by the Ignition config

This is what we want but, the SOFTWARE_CONFIG data are in multi-part mime format. We (@openstacker 's use-case) can't change this format.

so the agent would need to run in a container.
Yes, the heat-container-agent will run as a container (just like it runs on Fedora Atomic now as an atomic system container).

+1

[steveb]

Would it make sense to embed the SOFTWARE_CONFIG data in a file written by the Ignition config, which is then bind-mounted into the agent container? That would maintain Ignition as the sole bootstrap mechanism for configuring a Fedora CoreOS instance.

The only data in the SOFTWARE_CONFIG payload is the secrets required to bootstrap the polling mechanism (this could be endpoint+credentials, or a webhook). This is injected by heat without visibility to the user, and we chose to do it in a way which didn't rely on cloud-init just in case something replaced it in the future. So it is going to be a hard sell to the heat devs to do this in an ignition specific way.

I would consider multipart capable userdata support as a way for ignition to share userdata with other agents in the same way as the Accept header allows ignition to share an http endpoint

[openstacker]

Hi @bgilbert could you please revisit this issue? As @steveb mentioned above, we can start this work for OpenStack cloud provider only. And @strigazi and I are happy to do the coding work. Thanks.

[dustymabe]

hey @openstacker @steveb @strigazi!

We discussed this in the Fedora CoreOS meeting today. Unfortunately one of our key maintainers was out and will be back next week so we decided to table the discussion until that point as we'd likely have to have the discussion over again.

Would you like to join us next week to discuss?

[steveb]

@openstacker and I are both in New Zealand, so that time slot won't work for us. Thanks for the invite though

[dustymabe]

chiming in here as I thought of something else. In the Fedora Cloud Working group we may consider using Igntion for the cloud base image in the future. We won't be able to move about from cloud-init so we'd need to support both. Would having multi-part mime help with that or is that already covered?

[bgilbert]

I would consider multipart capable userdata support as a way for ignition to share userdata with other agents

Philosophically, Ignition expects to be the single source of truth for the system, so this is a hard sell. However:

This is injected by heat without visibility to the user

So the user provides single-part userdata (e.g. a cloud-config), and Heat automatically converts it to multipart with a SOFTWARE_CONFIG payload? That seems like a different situation. If the user isn't themselves generating the multipart payload, one could argue that the platform just has a complicated way of presenting userdata, which is something we'd normally support. We could limit multipart support to the OpenStack provider, and possibly even fail outright if we see an unexpected part.

In the Fedora Cloud Working group we may consider using Igntion for the cloud base image in the future. We won't be able to move about from cloud-init so we'd need to support both. Would having multi-part mime help with that or is that already covered?

Multipart won't help there, since a particular machine may have an Ignition config or a cloud-config but never both. Ignition v0s2 has code to detect a cloud-config in userdata and automatically enable cloud-init; we dropped that code for v2s3 but could add it back.

[strigazi]

@bgilbert @dustymabe example of user_data generated by heat. https://gist.github.com/strigazi/e014f2d83d75837e921fb31ce8a87628
In this example I passed to heat only the ignition config.

This is the important part for us:

--===============1175957420267015713==
Content-Type: text/x-cfninitdata; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cfn-init-data"

{"os-collect-config": {"cfn": {"stack_name": "strigazi-foo1", "metadata_url": "***", "path": "kube-master.Metadata", "secret_access_key": "***", "access_key_id": "***"}, "collectors": ["ec2", "cfn", "local"]}, "deployments": []}
--===============1175957420267015713==--

[dustymabe]

We discussed this in the Fedora CoreOS meeting today.

  * AGREED: to support the openstack heat use case we can make ignition
    support multi-part mime user-data on the openstack platform only.
    discussion will continue in the ticket on how targeted to heat the
    support will be  (dustymabe, 17:13:35)

[bgilbert]

@strigazi I see an Ignition config in a text/plain part, Heat credentials in a text/x-cfninitdata part, a cloud-config fragment, a boot hook script, and a Python script for parsing out the Heat credentials. Heat injects each of those parts every time?

[ajeddeloh]

Ignition has it's own mime type (application/vnd.coreos.ignition+json;version=<spec version>), is it possible to require that it's set correctly for the Ignition part?

What's the best way to test this? Is there somewhere I can upload images to and test?

[ajeddeloh]

Threw up a PR. It assumes the Content-Type is set correctly. I don't have a way of testing it, however.

[jlebon]

Here's how one could test that PR:

$ cd /srv/fcos
$ cosa init https://github.com/coreos/fedora-coreos-config
$ (cd /path/to/ignition/with/patch && make install DESTDIR=/srv/fcos/overrides/rootfs)
$ cosa build
$ cosa buildextend-openstack

Then throw up that image into your OpenStack instance and you're good to go!

[ajeddeloh]

Then throw up that image into your OpenStack instance and you're good to go!

I don't have an openstack instance?

[steveb]

@strigazi I see an Ignition config in a text/plain part, Heat credentials in a text/x-cfninitdata part, a cloud-config fragment, a boot hook script, and a Python script for parsing out the Heat credentials. Heat injects each of those parts every time?

Heat only injects those parts when the user_data_format is set to SOFTWARE_CONFIG, which is only used when heat's post-boot software configuration tooling is used.

Also, it is fair to assume that the mime-type will be set correctly for any ignition part in the multi-part data. In heat it is possible to build multipart data with specific mime types

[strigazi]

Hi all,
For the latest openstack release, we have patched the platform to support ignition.
I think we can close this for now.

cc @openstacker

Support Ignition for userdata https://review.opendev.org/#/c/685875/

[hardys]

Another potential use-case for multi-part support would be to combine ignition configs from multiple sources - it'd be pretty nice if you could just append ignition mime parts then have ignition automatically add them via append vs client side merging.

That isn't as flexible as the cloud-init merge strategies, but would probably be sufficient in many cases.

[lucab]

For the latest openstack release, we have patched the platform to support ignition.
Support Ignition for userdata https://review.opendev.org/#/c/685875/

That's great, thanks a lot! This is the best possible outcome, which also addresses maintainers' concerns at #849 (comment). Closing.