add a mechanism for containers to configure the host on first boot

[bcressey]

What I'd like:
I'd like a way to extend first boot setup by bringing my own custom logic in a container.

Some use cases:

• creating paths on the host before any orchestrated container starts (provide settings for creating paths on the host #1208)
• generating a setting on boot based on the running environment (Support a separate ENI/Instance-Type Map list for Custom Networking modes #1307)
• applying settings in bulk from a remote source (apiclient: add 'apply' mode for applying settings from URIs #1391)

Any alternatives you've considered:
In #1208 we looked at extending the settings-based API to create directories and files. The goal was to compose this functionality with other settings, such as #1209 for setting up filesystem mounts.

However, this introduces a lot of complexity in the general case. Declaring that a file should be created in a directory on a specific mounted filesystem means that we have to account for the filesystem's devices not always being available. When settings are changed, reconciliation can be challenging. The directory may be declared one way and later moved to a different mount; the directory may contain other, unmanaged host files; a declared file may have its contents modified outside of the API.

We can constrain this with arbitrary rules, such as: files created through the API are immutable; directories will be reconciled at boot, with permissions restored and unmanaged files removed. We can also limit composition: not allowing host files to be related to host directories; or not allowing host directories to be created on ephemeral disks.

In both cases we end up with an API that might suffice for the specific use cases we've heard about, but which is unlikely to generalize. The most powerful solution is to bring a host container with its own tools and logic, and rely on that instead of any host API features.

Host containers as they stand today fall short in a few ways. They run at an unspecified point in the boot sequence, but generally at the same time as orchestrated containers. They're meant to run as long-lived services rather than one-shots. They're also overly privileged for this use case; a super-powered bootstrap container could undermine many of the host protections in subtle ways.

[bcressey]

Tracking the implementation in #1387.