Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildah to use cache, layers for bash builds as well #1292

Open
zvonkok opened this issue Jan 18, 2019 · 8 comments

Comments

@zvonkok
Copy link

commented Jan 18, 2019

Description
Buildah can use cache/layers for building with buildah bud (#767). It would be beneficial if buildah could use a cache/layers for bash builds as well. I am using bash builds extensively as there are more convenient/useful for all of my use-cases.

Dan mentioned that I could use commits in-between and base off the next step of them.
Nice idea but I think this will clutter up the script and the signal-to-noise ratio would be rather high.

BTW using bash scripts to build the images is/was a great idea, this way I can use everything, really everything as a tool that I can install on my Linux and I am not tied to a DSL that I cannot extend or is limited.

Describe the results you received:
Executing the buildah script a second time, buildah executes each step again.

Describe the results you expected:
Skip steps that did not change compared to the last invocation of the script.

Output of rpm -q buildah or apt list buildah:

buildah-1.4-3.gitc8ed967.el7.x86_64

Output of buildah version:

Version:         1.5-dev
Go Version:      go1.10.2
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-alpha1
Git Commit:      c8ed967
Built:           Tue Nov  6 14:01:57 2018
OS/Arch:         linux/amd64
@zvonkok

This comment has been minimized.

Copy link
Author

commented Jan 18, 2019

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2019

This is an awesome suggestion! I wrote a tool which utilizes buildah for building images (and ansible as a frontend) and I had to solve the same issue: caching layers.

It would be awesome if buildah had an API for caching. On the other hand, it will be really tricky because you would need to tell buildah up front what the script is and how are you changing the filesystem and then buildah would need to figure out if there is a matching entry in the cache.

@rhatdan

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

@umohnani8 What do you think? Would this even be possible.
Should we keep all of the buildah commands, we could record the fact that we have already executed them. But anything that modified the content outside of the script we would fail.

We might be able to figure out that

buildah from fedora
buildah run
buildah config
buildah run
buildah mount
yum --installroot We would definitely fail here.
buildah config.

Also how would we know that two buildah from fedora calls were related?

@arthurbarr

This comment has been minimized.

Copy link

commented Jan 24, 2019

I guess that any of those buildah run commands could have a different effect if the bash script has (say) copied some different files into the mounted filesystems. I guess that docker build gets round this by having a fixed "context" directory, where all the source is located. They can then do a hash of that source, or look at time stamps (not sure which they do). So I wonder if buildah could have something which helped users hash and compare particular source files/directories, and skip the step if it hasn't changed. For example:

buildah --skip-if-unchanged=./src:./*.go run blah

@rhatdan

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

@umohnani8 Any update on this issue?

@rhatdan rhatdan assigned QiWang19 and unassigned umohnani8 Apr 12, 2019

@rhatdan

This comment has been minimized.

Copy link
Member

commented Apr 12, 2019

@QiWang19 Can you look into this?

@gcs278

This comment has been minimized.

Copy link

commented Jun 7, 2019

I'm very interested in this concept, great idea

@rhatdan

This comment has been minimized.

Copy link
Member

commented Jun 8, 2019

@zvonkok and @gcs278 PRs would make this move along faster...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.