New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

demonstrate server-side ansible/chef/etc runs #471

Open
cgwalters opened this Issue Sep 27, 2016 · 3 comments

Comments

Projects
None yet
2 participants
@cgwalters
Member

cgwalters commented Sep 27, 2016

Let's turn #96 into an issue. For people who are doing custom composes, it really makes a lot of sense to take the next step and bake in configuration management. We should make it convenient to pre-execute Ansible/Chef/Puppet type scripts on the compose server side.

An immediate blocker here is going to be the current postprocess-script system (intentionally) turns off networking.

The simplest approach might be exposing an API for fetching the RPMs and doing the initial rootfs preparation, then allowing another tool to do the config management however it wants, then just call ostree commit on that.

@cgwalters

This comment has been minimized.

Show comment
Hide comment
@cgwalters

cgwalters Oct 12, 2016

Member

I'm now thinking this should be a separate project that uses rpm-ostree via some API.

Member

cgwalters commented Oct 12, 2016

I'm now thinking this should be a separate project that uses rpm-ostree via some API.

@vtolstov

This comment has been minimized.

Show comment
Hide comment
@vtolstov

vtolstov Nov 24, 2016

can you provide some example api calls that needs to be done to get run some server config tool?

vtolstov commented Nov 24, 2016

can you provide some example api calls that needs to be done to get run some server config tool?

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Oct 5, 2017

WIP: split `compose tree` into `compose install` and `compose commit`
Right now `rpm-ostree compose tree` is very prescriptive about how
things work.  Trying to add anything that isn't an RPM is absolutely
fighting the system.  Our postprocessing system *enforces* no
network access (good for reproducibilty, but still prescriptive).

There's really a logical split between three phases:

 - "build a rootfs that installs packages"
 - "run postprocessing"
 - "commit result"

Our primary job is the first two - and we can do *either*. If for example
someone wants to use `rpm-ostree compose install`, and then tar up the result as
a Docker/OCI image, that becomes a bit easier. Or on the flip side, if someone
wants to do a `Dockerfile` style build system, we can make it easier to extract
the result of that and commit it into ostree.

Related issues/PRs:

 - projectatomic#96
 - projectatomic#471
@cgwalters

This comment has been minimized.

Show comment
Hide comment
@cgwalters

cgwalters Oct 5, 2017

Member

PR related to this in #1039

Member

cgwalters commented Oct 5, 2017

PR related to this in #1039

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Oct 6, 2017

compose/postprocess: Unlink files in /var as we go
This is prep for compose splitting:
projectatomic#471

We'll need to run some postprocessing types in both phases, and it's a lot
easier to reason about if the postprocessing is "idempotent", i.e. it has no
effect a second time.

rh-atomic-bot added a commit that referenced this issue Oct 6, 2017

compose/postprocess: Unlink files in /var as we go
This is prep for compose splitting:
#471

We'll need to run some postprocessing types in both phases, and it's a lot
easier to reason about if the postprocessing is "idempotent", i.e. it has no
effect a second time.

Closes: #1042
Approved by: jlebon

rh-atomic-bot added a commit that referenced this issue Oct 6, 2017

compose/postprocess: Unlink files in /var as we go
This is prep for compose splitting:
#471

We'll need to run some postprocessing types in both phases, and it's a lot
easier to reason about if the postprocessing is "idempotent", i.e. it has no
effect a second time.

Closes: #1042
Approved by: jlebon

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Oct 16, 2017

bin/compose: Split `compose tree` into `compose install` and `compose…
… commit`

Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - "build a rootfs that installs packages"
 - "run postprocessing"
 - "commit result"

Our primary job is the first and last - and we can do *either*. If for example
someone wants to use `rpm-ostree compose install`, and then tar up the result as
a Docker/OCI image, that becomes a bit easier. Or on the flip side, if someone
wants to do a `Dockerfile` style build system, we can make it easier to extract
the result of that and commit it into ostree.

Related issues/PRs:

 - projectatomic#96
 - projectatomic#471

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Oct 17, 2017

bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - projectatomic#96
 - projectatomic#471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Oct 24, 2017

bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - projectatomic#96
 - projectatomic#471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Oct 24, 2017

bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - projectatomic#96
 - projectatomic#471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

rh-atomic-bot added a commit that referenced this issue Oct 25, 2017

bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - #96
 - #471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

Closes: #1039
Approved by: jlebon
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment