Add root buildpack as interface for app image extension RFC

[jkutner]

Readable

[jkutner]

@nebhale @hone @ekcasey @sclevine while this is a PR to Stephen's OS extensions RFC PR, it's probably easier to digest it by reading the Readable version.

[matthewmcnew]

Could a better name for these buildpacks be stack buildpacks?

[jkutner]

i'm fine with either. I think to most end-users they will just be "buildpacks", so it's probably more a question for implementors of those buildpacks

[matthewmcnew]

I like the idea of buildpacks being the extensibility mechanism for stack extensions!

[matthewmcnew]

Doesn't executing arbitrary "root buildpacks" before a rebase no longer allow rebase to be a safe operation?

[jkutner]

I believe it makes it possible that rebase is unsafe. paging @sclevine

[sclevine]

Sorry I missed this. This is probably clear by now, but the idea is that we would re-play the root buildpacks on the new run image before rebasing.

[jabrown85]

Do root buildpacks participate in detect? Could I write a ruby buildpack to require imagemagick from theapt-get root buildpack if I see a particular gem in use that is known to need it?

[sclevine]

Note from chat on 5/20: would be good to ensure that the app developer doesn't need to pick whether the root buildpack runs on the run image vs. build image vs. both.

[josegonzalez]

Does this mean that root buildpack authors need to know what paths the packages the buildpack installs write to for layer exclusion?

[jkutner]

If they want to slim down the run image, then yes they do. Otherwise they can just ignore this file. This is a good thing to point out because this proposal is tilting toward giving the buildpack authors more responsibility. But with that, I think we keep a highly orthogonal interface (i.e. everything is a buildpack).

[josegonzalez]

Yeah just thinking allowed. I think that might work well for cases where a root buildpack is specific to a particular package - say, installing curl - but not so well if they support arbitrary packages based on files in the application repo?

[jkutner]

@jabrown85 yes, root buildpacks can bin/detect as normal

[ekcasey]

By

the paths will be stored in the cache

do we mean that the snapshotted layer will be stored in the cache?

[jkutner]

Yes, but as @matthewmcnew pointed out, we could choose to store it as an intermediate build image (no need for snapshot in that case I think).

[jkutner]

After more thought, I'm unsure if we can get away from the snapshot layer entirely. We may need it to restrict what changes are preserved (for example, we do not want to preserve changes to the /layers or /workspace dirs)

[ekcasey]

If we are allow root buildpacks to use slices, are we assuming that the app dir is excluded from the snapshot layer?

[jkutner]

yes, the app dir should be excluded from the snapshot layer. I'll call that out.

[ekcasey]

I am confused about how we generate run image layers using the snapshotting mechanism. Wouldn't we need to run the root buildpacks on the run image itself to capture the correct snapshot?

[ekcasey]

Also how does a root buildpack opt into reusing a run-image layer? Typically, we use the presence of a layer.toml file combined with the absence of layer content to signal layer reuse, but that doesn't work if the contents are not being written to the layers dir.

[jkutner]

Yes, the root buildpacks must run twice: once on the build image and once on the run image. I'm not sure about reusing the run-image layer. It might not be possible.

[sclevine]

I think we should cache the previous run image layers locally if they can be reused. They need to be built locally anyways.

[ekcasey]

Would the restorer re-apply the snapshotted build layer? Does that make it hard for the buildpack to remove and rebuild that layer if it is out of date?

[jkutner]

If the buildpack cannot rebuild with the snapshot layer applied, then it is not idempotent and the yet-to-be-named key in the buildpack.toml must specify this.

[natalieparellano]

Should this be detect.buildpacks?

[jkutner]

No, that's not a table in the latest definition of project.toml. So I'm proposing that the root buildpacks be listed in the same list as non-root buildpacks.

[natalieparellano]

Where would information about these layers (such as their digest) be stored? Is that the responsibility of the platform?

[jkutner]

I think the lifecycle would manage these layers (the same way as if you are storing the cache in an image). But I do need to address how they are stored when the cache is a volume. I'll fix this up

[natalieparellano]

Is it possible you could elaborate on this?

[jkutner]

I added "In effect, we are saying that the use of root buildpacks breaks rebase." Does that help?

[sclevine]

@hone @nebhale @ekcasey @jkutner curious if you think the above would be a good compromise? This uses @ekcasey’s idea to meet the requirements I heard, and gives proactive platform maintainers and buildpack authors a degree of control over performance (although it still feels somewhat risky)

[nebhale]

Sounds good to me. I have no skin in this game (I'm not planning on using this feature), and it doesn't seem any more dangerous than any other proposal.

[jkutner]

@sclevine I'm still a little unsettled with:

I still feel strongly that we shouldn't allow root buildpacks to be combined with regular buildpacks in a buildpackage.

I don't think it does a good job of accounting for the python+sqlite case (it pushes the burden onto the end-user, which is the opposite of the buildpacks value-prop). There are many other cases like this (such as Rails ActiveStorage with ffmpeg).

What about this: if we allow root buildpacks to satisfy mixin requirements AND opt-out of running when those packages are already provided by the stack (such as by finding them in the build plan), then it eliminates some of the performance concerns.

With regard to an ecosystem of buildpackages that rampantly include root buildpacks, I'm just not worried about it. Most end-users will use one of the following:

• a single buildpackage
• a buildpack maintained by an organization that knows not to abuse root buildpacks
• Their own custom buildpack

[sclevine]

it pushes the burden onto the end-user

I definitely agree that end-user developers should not have to deal with this. My intent isn't to force developers to specify the root buildpacks they want. I want builder authors / platform maintainers to have control over this instead. E.g., the heroku builder could supply an apt-buildpack in order.toml for every group.

What about this: if we allow root buildpacks to satisfy mixin requirements AND opt-out of running when those packages are already provided by the stack (such as by finding them in the build plan), then it eliminates some of the performance concerns.

How would a platform maintainer prevent developers from installing OS packages that they don't want them to use? They would end up needing to fork meta-buildpacks to remove/replace root buildpacks.

Most end-users will use one of the following:

• a single buildpackage
• a buildpack maintained by an organization that knows not to abuse root buildpacks
• Their own custom buildpack

Aren't we trying to build a large community registry of composable buildpacks in the long run?

[matthewmcnew]

How will root buildpacks contribute mixins in the build plan? Will all mixin validation occur in detect? Will Buildpacks be able to indicate if they provide mixins, giving platforms an early indication if a builder has a chance of satisfying buildpack mixin requirements?

[matthewmcnew]

Could you clarify what the "ephemeral run image" is here? How does it relate to the result of extend phase?

[sclevine]

Confused about this also. Maybe

and an ephemeral run image with additional layers

was supposed to be replaced by the "new extend phase" below?

[dfreilich]

I found the wording a bit confusing also. Is this what you meant?

Suggested change

	All root buildpacks will be sliced out of the list of buildpacks, and run before non-root buildpacks. This creates a version of the current builder with additional layers and an ephemeral run image with additional layers, then does a normal build phase without running the root buildpacks.
	All root buildpacks will be sliced out of the list of buildpacks, and run before non-root buildpacks. This creates a version of the current builder with additional layers and an ephemeral run image with additional layers. `Pack` then does a normal build phase, without running the root buildpacks.

[matthewmcnew]

While I understand the allure of a pack upgrade flow, I feel pretty strongly that this should just be accomplished by a complete rebuild. An "upgrade" that re-executes some but not all buildpacks seems arbitrary and confusing.

[sclevine]

The pack upgrade flow is just supposed to run the root buildpacks, and only on the run image. This does bring up a serious issue though: the app isn't accessible during a rebase.

@jkutner I think root buildpacks will need something like a /bin/rebase executable and/or some way of storing metadata about the app. Otherwise rebase would require the original source code.

Alternatively, we could only give root buildpacks access to metadata in project.toml and not the app directory.

[jromero]

I agree with @matthewmcnew, if I'm understanding the sentiment correctly.

My comment:

I propose that we eliminate the notion of upgrade and still only have the concept of (re)build and rebase.

If you envision the layers as below and then associate the two operations, it becomes easy to understand and eliminates the complexity being introduced by upgrade.

+-------------------------------+   +-------+
|              app              |           |
+-------------------------------+           |
                                            |
+-------------------------------+           |
|    app deps from buildpacks   |           |  (Re)Build
+-------------------------------+           |
                                            |
+-------------------------------+           |
| app deps from root buildpacks |           |
+-------------------------------+   +-------+

+-------------------------------+   +-------+
|                               |           |
|        OS from stack          |           |  Rebase
|                               |           |
+-------------------------------+   +-------+

As an additional (possible) upside, by not providing additional "upgradability" for these set of layers we steer buildpack authors/developers to push recurring dependencies to the stack layers to improve performance.

[dfreilich]

+1 to @jromero and @matthewmcnew . I think adding an upgrade would be a confusion for new consumers, who know they need to do something for their app, but aren't sure what. If there's an explicit need at a later point, great, but adding it preemptively to me sounds like the confusion risks outweigh the benefits.

[sclevine]

I'm starting to worry about @matthewmcnew's point about rebasing being potentially unsafe after root buildpacks are used. Unless we guarantee that root buildpacks only install packages that remain ABI compatible on updates, rebase could change app behavior. This could break apps in production and cause users to lose faith that the rebase process is safe.

If we decide that rebase isn't safe in those cases, then any use of root buildpacks will drastically hurt performance for at-scale security patches. Full rebuilds would be necessary for all images.

We can keep rebase safe with root buildpacks, but only if platforms like kpack can whitelist which root buildpacks are available. If arbitrary meta-buildpacks can contain root buildpacks, then this becomes much harder.

More generally, I feel like root buildpacks make it too easy for buildpack authors to degrade the end-user experience (for both developers and operators) and that we need some sort of control to prevent that from happening. That control shouldn't hurt interoperability (e.g., prevent heroku buildpacks from being used with kpack + many apps).

I'm not strongly opinionated about how we achieve this, but disallowing root buildpacks and regular buildpacks to mix in a buildpackage would be a start. Would you be open to that option if we allowed root buildpacks to be installed as enforced pre-build hooks on builders? Then developers would never need to think about them in the OS package case, even when manually specifying buildpacks. If a buildpack needs a package, either the stack would contain a given package, or the package would get automatically installed.

[josegonzalez]

Is it the case that a buildpack needs a package, or that an app needs a package? As an example, I might write an app that calls out - via subprocess - to headless chrome (installed via some deb package) to do Things™. The buildpack didn't link to chrome for any installation purpose, but it's a runtime requirement.

Asking as I'm not sure if that would impact the end user interface here.

[jkutner]

@josegonzalez i think it's both. Some buildpacks will depend on a package, and some apps will depend on a package. Ideally the former is completely hidden from the end-user (only the buildpack author is aware of it). But I think we're afraid that any solution will force the end-user (app developer) to be aware of it.

[jkutner]

Based on a breakout discussion with @hone @sclevine and @ekcasey, I'm going to try and tweak this RFC such that the following are true:

1. Root buildpacks must be provided by the stack
2. Stack maintainers will include any root buildpacks they support at the time the Builder image is create.
3. End users can only use buildpacks that are provided by the stack in the [[build.buildpacks]] table in project.toml.
4. The run-image will contain any root buildpacks provided by the stack, and those buildpacks will be removed from the final app image (but retained in the run-image so that they can be re-run on upgrade)
5. Buildpackages can only contain root buildpacks or non-root buildpacks (not both)

What is missing, IMO, is a way for an end-user to extend the stack (i.e. to provide their own root buildpacks). A few options I'd like to consider for this are:

• a pack add-buildpack-to-builder (or similarly named command). This would extend the builder with either a root buildpack or a non-root buildpack. For root buildpacks, they would be added to the stack.
• A mechanism that would extend the stack on-the-fly if a root buildpack is used in [[build.buildpacks]] but is not provided by the stack.

[dfreilich]

nit: Spelling

[dfreilich]

Suggested change

As an example, consider the use case of Acme.com. They have three teams that want to contibute privileged layers to every image built at the company. Each of these teams wants to manage their own artifacts and the mechanisms for installing them, but none of these teams are in control of the base image. The teams have special logic for when their layers should be added to an image; in some cases it's the precense of a Node.js app, and in others it's the use of Tomcat, etc. In the past, these teams were trying to contribute their layers by selectively adding them to every `Dockerfile` for every base image in the company, but this doesn't scale well. With Root Buildpacks, each team can manage their own artifacts, release cadence, and add logic to `bin/detect` to ensure the layers are added when needed.

As an example, consider the use case of Acme.com. They have three teams that want to contribute privileged layers to every image built at the company. Each of these teams wants to manage their own artifacts and the mechanisms for installing them, but none of these teams are in control of the base image. The teams have special logic for when their layers should be added to an image; in some cases it's the presence of a Node.js app, and in others it's the use of Tomcat, etc. In the past, these teams were trying to contribute their layers by selectively adding them to every `Dockerfile` for every base image in the company, but this doesn't scale well. With Root Buildpacks, each team can manage their own artifacts, release cadence, and add logic to `bin/detect` to ensure the layers are added when needed.

[dfreilich]

This was correcting the spelling of presence (currently precense).

[dfreilich]

Suggested change

For each root buildpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the buildpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). Alternatively to snapshotting, a platform may store a new stack iamge to cache the changes. All of the captured changes will be included in a single layer produced as output from the buildpack. The `/layers` dir MAY NOT be used to create arbitrary layers.

For each root buildpack, the lifecycle will use [snapshotting](https://github.com/GoogleContainerTools/kaniko/blob/master/docs/designdoc.md#snapshotting-snapshotting) to capture changes made during the buildpack's build phase (excluding `/tmp`, `/cnb`, and `/layers`). As an alternative to snapshotting, the platform may store a new stack image to cache the changes. All of the captured changes will be included in a single layer produced as output from the buildpack. The `/layers` dir MAY NOT be used to create arbitrary layers.

[dfreilich]

What's the rationale behind this constraint? What is the concern it's addressing?

[jkutner]

This addresses a concern that @sclevine had about root buildpacks proliferating through the ecosystem. Like, if we give this tool to people, they will use it too frequently. If we instead restrict its use, it forces buildpacks authors to explore alternative solutions.

I don't necessarily agree with that concern, but I'm willing to accept it in compromise (in part because it's easy to reverse).

[dfreilich]

Why wouldn't this be a positive outcome?

[jkutner]

I think it would be positive, but it's an open question because we aren't defining how it would work.

[jkutner]

Closing in favor of #111 and #112