Add a way to run a script outside of the ostree root after build #96
Conversation
While running a postprocess script inside is sufficient for lots of use case, being able to tun outside permit to avoid polluting the ostree root. One of such use case would be to use ansible outside of the root, to configure things inside the root.
|
My motivation for this is to be able to reuse ansible and my existing playbook to run and install ostree image that i can then deploy on server ( using 'ansible-playbook -c chroot -i "$OSTREE_ROOT," foo.yml, ). While this approach will work ( just requires a wrapper script ), I wonder if being able to give directly a command line wouldn't be better ( but slightly more complex to code ). I pushed the PR for discussion for now, but I forgot the doc, so it is not ready to be merged. And would like some feedback on the way we call the option, if someone has a better name for it ? |
|
After a while, i decided to see if I couldn't directly give a playbook as a argument, so produced a different branch: |
|
The original vision for rpm-ostree was to basically be "yum on the server side", and that's the current state. What that means is that the composed tree is not configured - you have to use something like cloud-init/kickstart to set up authentication, and then a generic config system like Ansible/Puppet per host. What this is doing is blurring all of those domains together with compose-side confmgmt, which is a new direction. Now, I can see some benefits and drawbacks to this. A benefit here is that your system state description becomes stronger - because the Ansible run happened on the server side, when you However, there are disadvantages too. First, OSTree was not designed for carrying secrets (such as the provisioned root password, etc.). You can of course just do all fetches over TLS which provides confidentiality, but it wasn't part of the design today. Another concern I have is that it increases the amount of respinning that's required as the RPMs change. For example, if you end up with multiple trees with different configuration per host type (e.g. you have a "frontend webserver" tree, a "database server" tree) that means that when say the next security errata for bash happens, each of those trees will need updating. That should of course not be difficult to do, and we will be increasing efficiency of the compose process. But it's still a concern. So I guess what I'm getting at is - is this patch an experiment, or are you fairly confident that you want to do pre-configured trees? |
|
I was aiming something in the line of the nixos stateless deployment idea ( http://gfxmonk.net/2015/01/03/nixos-and-stateless-deployment.html ), or at least, have something open enough to make this doable. Respin was not so long ( 10 minutes for a minimal system on my laptop, mostly I/O bound ), even if I guess starting to generalize this would mean having a rather complicated system. So far, that was a experiment. I didn't played enough with pre-configured tree to see, but I am quite sure that this would permit to enable something less minimalistic than * atomic host ( besides servers, one use case I had in mind were internal corporate laptops, where some people seldomly run update by themself, and where we cannot run upgrade in the background as they are not atomic ). Now indeed, the problem of populating /etc properly is open. Another thing is that configuration is sometime tied to runtime. Let's take the case of apache. If I have a apache 2.2 and want to test a upgrade on 2.4, I cannot keep the same configuration, since the format changed for some directive regarding authentication, and some modules changed their names. In practice, I can work around with specific trick on the apache side, but that the exception more than the norm. So if I want to test apache 2.4 and decide to revert, then I have to reconfigure it again. If the configuration is part of the atomic upgrade, then it is also part of the downgrade. |
|
FWIW my conclusion on Nix was: This is a fantastically complex discussion; we haven't even mentioned Docker yet 😄 Or things like NixOS's ability to do installs as non-root (which the RPM world doesn't really do). What I'm thinking on this issue though is that if the goal is to better support merging OS + config on a compose server, it would work best if rpm-ostree was a shared library effectively. And if say you were using Ansible, then you would have a YAML entry like: And a higher level tool would know how to consume the special rpmostree entry and also do |
|
I forgot to mention of course that rpm-ostree has always been declarative, which is fundamentally different from traditional |
|
Concretely in this PR, I'm uncertain about rpm-ostree calling ansible.
Making rpm-ostree a library could even just be breaking up compose tree like: ? |
|
Before I forget, this is a cool thing to experiment with, and thanks for submitting the patch 😄 |
|
Well, I think I didn't explain the PR correctly. I have basically 2 differents code. One is to tell "run this script after the build, but outside of the tree". That's what this PR is about. Providing a wrapper script permit to use more than ansible, without being forced to use any of them. I think that's the most generic solution out there. However, since my goal was to run ansible, I directly merged and made a different branch to play, which is not proposed to be merged. I found it more elegant for my purpose, but obviously, not generic enough. If there was a plugin system, I would have likely tried to add that as a plugin. The idea of having a ostree ansible module seems interesting. Since you can dynamically inject a host in the list of ansible managed host after creating it ( example, ask for a vm on openstack, then start to manage it ), the same could be done with a rpmostree module that create the compose tree, give back the directory, then we could inject it in the list, modify it, and then commit the tree. So your proposal to split rpm-ostree compose in 2 operations would be ok for me. A library would be nice, but calling binary is ok for me. If the rpm-ostree compose tree-prepare command could take a --json flag, and return the tree as a result, it would be perfect for me, but I can do that also in the ansible module. |
|
Ok, sorry for joining this late. I'm looking to use rpm-ostree to create a standard image for our school's desktops. I used to use rpms for configuration, but for F21, we switched to kickstart+ansible, which has been much easier to work with. I think splitting rpm-ostree compose into two operations would make life much easier for me. I can translate my kickstart into a treefile pretty easily, but the ansible stuff is a whole lot more complex and I'd far prefer to just run ansible on the image before committing it. One thing I don't understand is why we need a new commit operation. If rpm-ostree just stopped after preparing the tree, couldn't we just run ostree commit after running ansible? |
|
http://www.ansible.com/blog/immutable-systems is also related to this - the concerns with cloud images are pretty similar to ostree commits. |
|
Since I did broke a few VM because i underspecified them in term of memory ( and yum + selinux tend to consume too much ), immutable images and or offline upgrade ( ie, ostree ) would also help a lot for that. |
|
I'm also interested in this, also coming from a NixOS world. |
|
Let's move this to #471 |
Right now `rpm-ostree compose tree` is very prescriptive about how things work. Trying to add anything that isn't an RPM is absolutely fighting the system. Our postprocessing system *enforces* no network access (good for reproducibilty, but still prescriptive). There's really a logical split between three phases: - "build a rootfs that installs packages" - "run postprocessing" - "commit result" Our primary job is the first two - and we can do *either*. If for example someone wants to use `rpm-ostree compose install`, and then tar up the result as a Docker/OCI image, that becomes a bit easier. Or on the flip side, if someone wants to do a `Dockerfile` style build system, we can make it easier to extract the result of that and commit it into ostree. Related issues/PRs: - coreos#96 - coreos#471
|
Closing this in favor of #1039 Thanks a ton for starting one of the very first interesting PRs here. It wasn't forgotten! |
… commit` Right now `rpm-ostree compose tree` is very prescriptive about how things work. Trying to add anything that isn't an RPM is absolutely fighting the system. Our postprocessing system *enforces* no network access (good for reproducibilty, but still prescriptive). There's really a logical split between three phases: - "build a rootfs that installs packages" - "run postprocessing" - "commit result" Our primary job is the first and last - and we can do *either*. If for example someone wants to use `rpm-ostree compose install`, and then tar up the result as a Docker/OCI image, that becomes a bit easier. Or on the flip side, if someone wants to do a `Dockerfile` style build system, we can make it easier to extract the result of that and commit it into ostree. Related issues/PRs: - coreos#96 - coreos#471
Right now `rpm-ostree compose tree` is very prescriptive about how things work. Trying to add anything that isn't an RPM is absolutely fighting the system. Our postprocessing system *enforces* no network access (good for reproducibilty, but still prescriptive). There's really a logical split between three phases: - install: "build a rootfs that installs packages" - postprocess: "run magical ostree postprocessing like kernel" - commit: "commit result to ostree" So there are two high level flows I'd like to enable here. First is to allow people to do *arbitrary* postprocessing between `install` and `commit`. For example, run Ansible and change `/etc`. This path basically is like what we have today with `postprocess-script.sh`, except the builder can do anything they want with network access enabled. Going much farther, this helps us support a "build with Dockerfile" style flow. We can then provide tooling to extract the container image, and combine `postprocess` and `commit`. Or completely the other way - if for example someone wants to use `rpm-ostree compose install`, they could tar up the result as a Docker/OCI image. That's now easier; an advantage of this flow over e.g. `yum --installroot` is the "change detection" code we have. Related issues/PRs: - coreos#96 - coreos#471 One disadvantage of this approach right now is that if one *does* go for the split approach, we lose the "input hash" metadata for example. And down the line, I'd like to add even more metadata, like the input rpm repos, which could also be rendered on the client side. But, I think we can address that later by e.g. caching the metadata in a file in the install root and picking it back up or something.
Right now `rpm-ostree compose tree` is very prescriptive about how things work. Trying to add anything that isn't an RPM is absolutely fighting the system. Our postprocessing system *enforces* no network access (good for reproducibilty, but still prescriptive). There's really a logical split between three phases: - install: "build a rootfs that installs packages" - postprocess: "run magical ostree postprocessing like kernel" - commit: "commit result to ostree" So there are two high level flows I'd like to enable here. First is to allow people to do *arbitrary* postprocessing between `install` and `commit`. For example, run Ansible and change `/etc`. This path basically is like what we have today with `postprocess-script.sh`, except the builder can do anything they want with network access enabled. Going much farther, this helps us support a "build with Dockerfile" style flow. We can then provide tooling to extract the container image, and combine `postprocess` and `commit`. Or completely the other way - if for example someone wants to use `rpm-ostree compose install`, they could tar up the result as a Docker/OCI image. That's now easier; an advantage of this flow over e.g. `yum --installroot` is the "change detection" code we have. Related issues/PRs: - coreos#96 - coreos#471 One disadvantage of this approach right now is that if one *does* go for the split approach, we lose the "input hash" metadata for example. And down the line, I'd like to add even more metadata, like the input rpm repos, which could also be rendered on the client side. But, I think we can address that later by e.g. caching the metadata in a file in the install root and picking it back up or something.
Right now `rpm-ostree compose tree` is very prescriptive about how things work. Trying to add anything that isn't an RPM is absolutely fighting the system. Our postprocessing system *enforces* no network access (good for reproducibilty, but still prescriptive). There's really a logical split between three phases: - install: "build a rootfs that installs packages" - postprocess: "run magical ostree postprocessing like kernel" - commit: "commit result to ostree" So there are two high level flows I'd like to enable here. First is to allow people to do *arbitrary* postprocessing between `install` and `commit`. For example, run Ansible and change `/etc`. This path basically is like what we have today with `postprocess-script.sh`, except the builder can do anything they want with network access enabled. Going much farther, this helps us support a "build with Dockerfile" style flow. We can then provide tooling to extract the container image, and combine `postprocess` and `commit`. Or completely the other way - if for example someone wants to use `rpm-ostree compose install`, they could tar up the result as a Docker/OCI image. That's now easier; an advantage of this flow over e.g. `yum --installroot` is the "change detection" code we have. Related issues/PRs: - coreos#96 - coreos#471 One disadvantage of this approach right now is that if one *does* go for the split approach, we lose the "input hash" metadata for example. And down the line, I'd like to add even more metadata, like the input rpm repos, which could also be rendered on the client side. But, I think we can address that later by e.g. caching the metadata in a file in the install root and picking it back up or something.
Right now `rpm-ostree compose tree` is very prescriptive about how things work. Trying to add anything that isn't an RPM is absolutely fighting the system. Our postprocessing system *enforces* no network access (good for reproducibilty, but still prescriptive). There's really a logical split between three phases: - install: "build a rootfs that installs packages" - postprocess: "run magical ostree postprocessing like kernel" - commit: "commit result to ostree" So there are two high level flows I'd like to enable here. First is to allow people to do *arbitrary* postprocessing between `install` and `commit`. For example, run Ansible and change `/etc`. This path basically is like what we have today with `postprocess-script.sh`, except the builder can do anything they want with network access enabled. Going much farther, this helps us support a "build with Dockerfile" style flow. We can then provide tooling to extract the container image, and combine `postprocess` and `commit`. Or completely the other way - if for example someone wants to use `rpm-ostree compose install`, they could tar up the result as a Docker/OCI image. That's now easier; an advantage of this flow over e.g. `yum --installroot` is the "change detection" code we have. Related issues/PRs: - #96 - #471 One disadvantage of this approach right now is that if one *does* go for the split approach, we lose the "input hash" metadata for example. And down the line, I'd like to add even more metadata, like the input rpm repos, which could also be rendered on the client side. But, I think we can address that later by e.g. caching the metadata in a file in the install root and picking it back up or something. Closes: #1039 Approved by: jlebon
While running a postprocess script inside is sufficient for
lots of use case, being able to tun outside permit to avoid
polluting the ostree root. One of such use case would
be to use ansible outside of the root, to configure things
inside the root.