New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incrementally sending build context #231

Merged
merged 4 commits into from Jun 27, 2017

Conversation

Projects
None yet
8 participants
@tonistiigi
Member

tonistiigi commented Jun 22, 2017

Adds support for incremental context updates to docker build moby/moby#32677 . The feature is behind a --stream=true flag atm, mostly for a testing period and in case there is some reason to use the old method. When we feel confident enough we can just switch the default value and users shouldn't see any changes except things being faster.

I also noticed that it vendors in chrootarchive and reexec package. These are not really needed but require some refactoring. Either splitting the receiver and sender part to a separate package, or passing an archiver instance into the transfer function. I can do that in a follow-up with next update.

@dnephin @vieux

@tonistiigi tonistiigi changed the title from Client session fssession to Incrementally sending build context Jun 22, 2017

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Jun 22, 2017

Codecov Report

Merging #231 into master will decrease coverage by 0.61%.
The diff coverage is 25.44%.

@@            Coverage Diff             @@
##           master     #231      +/-   ##
==========================================
- Coverage   47.19%   46.58%   -0.62%     
==========================================
  Files         171      172       +1     
  Lines       11556    11631      +75     
==========================================
- Hits         5454     5418      -36     
- Misses       5791     5902     +111     
  Partials      311      311

codecov-io commented Jun 22, 2017

Codecov Report

Merging #231 into master will decrease coverage by 0.61%.
The diff coverage is 25.44%.

@@            Coverage Diff             @@
##           master     #231      +/-   ##
==========================================
- Coverage   47.19%   46.58%   -0.62%     
==========================================
  Files         171      172       +1     
  Lines       11556    11631      +75     
==========================================
- Hits         5454     5418      -36     
- Misses       5791     5902     +111     
  Partials      311      311

@vdemeester vdemeester added this to the 17.07.0 milestone Jun 23, 2017

dnephin and others added some commits May 9, 2017

Fix cyclomatic complexity of two formatters
Signed-off-by: Daniel Nephin <dnephin@docker.com>
vendor: update moby
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@dnephin

Some tests would be great.

We don't have any in master yet, but I wrote on in #233 which uses a fakeClient. You could grab my changes to the fake from that PR.

Splitting out some functions might make it possible to test some in isolation as well (like the build shared key functionality).

Show outdated Hide outdated cli/command/cli.go Outdated
@@ -133,6 +146,10 @@ func NewBuildCommand(dockerCli *command.DockerCli) *cobra.Command {
flags.SetAnnotation("squash", "experimental", nil)
flags.SetAnnotation("squash", "version", []string{"1.25"})
flags.BoolVar(&options.stream, "stream", false, "Stream attaches to server to negotiate build context")
flags.SetAnnotation("stream", "experimental", nil)
flags.SetAnnotation("stream", "version", []string{"1.31"})

This comment has been minimized.

@dnephin

dnephin Jun 23, 2017

Collaborator

I think we normally don't set the version until it's out of experimental. I'm not sure how it works if we set both. Maybe it just works?

@dnephin

dnephin Jun 23, 2017

Collaborator

I think we normally don't set the version until it's out of experimental. I'm not sure how it works if we set both. Maybe it just works?

This comment has been minimized.

@tonistiigi

tonistiigi Jun 23, 2017

Member

squash option above seems to set it

@tonistiigi

tonistiigi Jun 23, 2017

Member

squash option above seems to set it

Show outdated Hide outdated cli/command/image/build.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
}
response, err := dockerCli.Client().ImageBuild(ctx, body, buildOptions)
if err != nil {
if options.quiet {
fmt.Fprintf(dockerCli.Err(), "%s", progBuff)
}
cancel()

This comment has been minimized.

@dnephin

dnephin Jun 23, 2017

Collaborator

Isn't this already handled by the defer cancel() ?

@dnephin

dnephin Jun 23, 2017

Collaborator

Isn't this already handled by the defer cancel() ?

This comment has been minimized.

@tonistiigi

tonistiigi Jun 23, 2017

Member

There is a defer that makes sure we have always written out the progress before returning from this function. This cancel is for unblocking that. I'll add a comment.

@tonistiigi

tonistiigi Jun 23, 2017

Member

There is a defer that makes sure we have always written out the progress before returning from this function. This cancel is for unblocking that. I'll add a comment.

Show outdated Hide outdated cli/command/image/build.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
fix build issue with updated moby
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@tonistiigi

This comment has been minimized.

Show comment
Hide comment
@tonistiigi

tonistiigi Jun 23, 2017

Member

@dnephin Updated. Added more comments for the conditions and moved the helper functions to a separate file where possible.

Member

tonistiigi commented Jun 23, 2017

@dnephin Updated. Added more comments for the conditions and moved the helper functions to a separate file where possible.

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Jun 26, 2017

Contributor

LGTM

Contributor

vieux commented Jun 26, 2017

LGTM

Show outdated Hide outdated cli/command/image/build.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
@thaJeztah

left some questions/comments

Show outdated Hide outdated cli/command/image/build.go Outdated
@@ -23,6 +23,7 @@ func TestDiskUsageContextFormatWrite(t *testing.T) {
Images 0 0 0B 0B
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0B 0B

This comment has been minimized.

@thaJeztah

thaJeztah Jun 26, 2017

Member

Wonder if we should have something (-) in the first two columns, but we can tweak that later

@thaJeztah

thaJeztah Jun 26, 2017

Member

Wonder if we should have something (-) in the first two columns, but we can tweak that later

Show outdated Hide outdated cli/command/image/build.go Outdated
const clientSessionRemote = "client-session"
func isSessionSupported(dockerCli *command.DockerCli) bool {

This comment has been minimized.

@thaJeztah

thaJeztah Jun 26, 2017

Member

commit 0c780c8 adds isSessionSupported() and getBuildSharedKey(), which are moved in 9776fb5, perhaps those commits should be squashed, or the change moved from one to the other.

@thaJeztah

thaJeztah Jun 26, 2017

Member

commit 0c780c8 adds isSessionSupported() and getBuildSharedKey(), which are moved in 9776fb5, perhaps those commits should be squashed, or the change moved from one to the other.

This comment has been minimized.

@tonistiigi

tonistiigi Jun 26, 2017

Member

I can squash but that merges session+filesync. The move was requested from review. I think splitting up would take too much time.

@tonistiigi

tonistiigi Jun 26, 2017

Member

I can squash but that merges session+filesync. The move was requested from review. I think splitting up would take too much time.

Show outdated Hide outdated cli/command/image/build_session.go Outdated
Show outdated Hide outdated cli/command/image/build.go Outdated
Use long running session in builder
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Add incremental context send support

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@tonistiigi

This comment has been minimized.

Show comment
Hide comment
@tonistiigi

tonistiigi Jun 26, 2017

Member

@thaJeztah Updated the filename and squashed.

@tiborvass Added comment.

Member

tonistiigi commented Jun 26, 2017

@thaJeztah Updated the filename and squashed.

@tiborvass Added comment.

@thaJeztah

LGTM

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Jun 27, 2017

Contributor

LGTM

Contributor

tiborvass commented Jun 27, 2017

LGTM

@vieux vieux merged commit 0133e13 into docker:master Jun 27, 2017

6 of 7 checks passed

codecov/patch 25.44% of diff hit (target 50%)
Details
ci/circleci: cross Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
ci/circleci: validate Your tests passed on CircleCI!
Details
codecov/project 46.58% (-0.62%) compared to 3648a8b
Details
dco-signed All commits are signed
buildCtx = replaceDockerfileTarWrapper(ctx, buildCtx, relDockerfile, translator, &resolvedTags)
} else if dockerfileCtx != nil {
// if there was not archive context still do the possible replacements in Dockerfile
newDockerfile, _, err := rewriteDockerfileFrom(ctx, dockerfileCtx, translator)

This comment has been minimized.

@dnephin

dnephin Jul 4, 2017

Collaborator

@tonistiigi I notice that resolvedTags is not being assigned here. Isn't this necessary for content trust?

@dnephin

dnephin Jul 4, 2017

Collaborator

@tonistiigi I notice that resolvedTags is not being assigned here. Isn't this necessary for content trust?

case <-ctx.Done():
}
}()
buildBuff = buf

This comment has been minimized.

@dnephin

dnephin Jul 4, 2017

Collaborator

If options.quiet is true then buildBuff is a bytes.Buffer. Later on it is expected that this remains a bytes.Buffer() so that its contents can be printed.

I think this breaks with --quiet

@dnephin

dnephin Jul 4, 2017

Collaborator

If options.quiet is true then buildBuff is a bytes.Buffer. Later on it is expected that this remains a bytes.Buffer() so that its contents can be printed.

I think this breaks with --quiet

This comment has been minimized.

@tonistiigi

tonistiigi Jul 5, 2017

Member

agh, sprintf of io.Writer

imageID = fmt.Sprintf("%s", buildBuff)
🤦‍♂️

@tonistiigi

tonistiigi Jul 5, 2017

Member

agh, sprintf of io.Writer

imageID = fmt.Sprintf("%s", buildBuff)
🤦‍♂️

This comment has been minimized.

@dnephin

dnephin Jul 5, 2017

Collaborator

I have this fixed in #294, it only sets buf if !quiet

@dnephin

dnephin Jul 5, 2017

Collaborator

I have this fixed in #294, it only sets buf if !quiet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment