Support Windows-based images for `build` #739

ameyer-pivotal · 2020-07-10T20:40:54Z

Supports Windows-based images for pack build (under experimental flag)

Resolves #470

ameyer-pivotal · 2020-07-10T20:46:35Z

dfreilich

Overall, this looks great!

One general thought - would it be better to move some of the common variables/Windows changes (i.e. the change from root to ContainerAdministrator) to be injected build-time variables (like we use here), or would that make it more complicated? I'm not recommending necessarily that we use it, I just wonder what people think. (cc @simonjjones )

acceptance/acceptance_test.go

build.go

internal/archive/archive.go

internal/build/container_ops.go

internal/build/build.go

rebase.go

ameyer-pivotal · 2020-07-13T17:56:18Z

Overall, this looks great!

One general thought - would it be better to move some of the common variables/Windows changes (i.e. the change from root to ContainerAdministrator) to be injected build-time variables (like we use here), or would that make it more complicated? I'm not recommending necessarily that we use it, I just wonder what people think. (cc @simonjjones )

@dfreilich It seems to me that the reason for build-time injection would be for values that are subject to change on a build-by-build basis (pack version, etc.). I'm not seeing the value for these particular variables, but open to others' thoughts.

For root vs ContainerAdministrator, it's important to note that the value chosen still needs to be dynamic at runtime, based on the docker host OS being targeted. The actual content of the ContainerAdministrator string could be moved into a build-time variable, but per the above paragraph, I'm not sure what that buys you.

codecov · 2020-07-15T20:53:45Z

Codecov Report

Merging #739 into main will increase coverage by 1.37%.
The diff coverage is 64.93%.

@@            Coverage Diff             @@
##             main     #739      +/-   ##
==========================================
+ Coverage   76.05%   77.41%   +1.37%     
==========================================
  Files          78       78              
  Lines        5301     4443     -858     
==========================================
- Hits         4031     3439     -592     
+ Misses        973      698     -275     
- Partials      297      306       +9

Flag	Coverage Δ
#os_linux	`77.41% <64.93%> (-0.96%)`	⬇️
#os_macos	`73.40% <52.88%> (-0.87%)`	⬇️
#os_windows	`100.00% <ø> (+25.64%)`	⬆️
#unit	`77.41% <64.93%> (+1.37%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

natalieparellano

This is awesome! Given the magnitude of the impact the number of changes is relatively small and it's very clean. Looking forward to seeing the final PR :)

.github/workflows/build.yml

natalieparellano · 2020-07-20T21:37:24Z

acceptance/acceptance_test.go

@@ -901,6 +867,8 @@ func testAcceptance(
 						var buildpackTgz, tempVolume string

 						it.Before(func() {
+							h.SkipIf(t, os.Getenv("DOCKER_HOST") != "", "cannot mount volume when DOCKER_HOST is set")


Do we have a README or something that explains the use of DOCKER_HOST in local development?

Not currently, but since it's a docker usage concern, I wonder if we should link to the docker docs somewhere?

Maybe this could be covered in buildpacks/docs#180?

You might add "pack developer" to the "target audience" list in that issue.

Is that audience covered by buildpacks.io??

It's not and I don't think it should. "pack developers" should be looking at the repo *.md or wiki.

natalieparellano · 2020-07-20T21:40:28Z

acceptance/acceptance_test.go

+							cmd := subjectPack("build", repoName, "-p", filepath.Join("testdata", "mock_app"), "--publish", "--network", "host")
+
+							if dockerHostOS() == "windows" {
+								cmd = subjectPack("build", repoName, "-p", filepath.Join("testdata", "mock_app"), "--publish")


A comment here explaining the underlying limitation might be helpful.

acceptance/acceptance_test.go

natalieparellano · 2020-07-20T21:45:15Z

acceptance/acceptance_test.go

@@ -2028,7 +2046,7 @@ func waitForResponse(t *testing.T, port string, timeout time.Duration) string {
 	for {
 		select {
 		case <-ticker.C:
-			resp, err := h.HTTPGetE("http://localhost:"+port, map[string]string{})
+			resp, err := h.HTTPGetE("http://"+h.RegistryHost(port), map[string]string{})


Rooting for this to be from imgutil one day :)

We created an issue in our backlog to port this to imgutil after this story is complete (that way it's easier to validate the implementation first).

acceptance/testdata/mock_buildpacks/0.2/descriptor-buildpack/bin/detect.bat

acceptance/testdata/mock_buildpacks/0.2/not-in-builder-buildpack/bin/build.bat

internal/builder/lifecycle.go

dfreilich · 2020-07-27T14:41:08Z

For tracking purposes, would you mind also creating a docs issues for documenting building Windows images?

ameyer-pivotal · 2020-08-04T14:35:30Z

For tracking purposes, would you mind also creating a docs issues for documenting building Windows images?

@dfreilich Done. buildpacks/docs#180

jromero

This is great!!! I'm excited to see Windows support!

Didn't get to comb through everything but I thought I'd provide some partial feedback to minimize the feedback loop.

.github/workflows/build.yml

Makefile

acceptance/acceptance_test.go

internal/archive/archive.go

internal/build/container_ops.go

build.go

testhelpers/testhelpers.go

dfreilich

Looks really great! I agree with Javier that it can use a bit of cleaning up to make build.go a bit more elegant, but this is super impressive.

Makefile

acceptance/acceptance_test.go

build.go

build_test.go

build.go

internal/build/phase_config_provider.go

natalieparellano

This is looking great! Added a few comments.

internal/cache/volume_cache_test.go

testhelpers/testhelpers.go

internal/build/phase_config_provider_test.go

acceptance/acceptance_test.go

natalieparellano · 2020-08-10T16:19:57Z

acceptance/acceptance_test.go

@@ -901,6 +867,8 @@ func testAcceptance(
 						var buildpackTgz, tempVolume string

 						it.Before(func() {
+							h.SkipIf(t, os.Getenv("DOCKER_HOST") != "", "cannot mount volume when DOCKER_HOST is set")


You might add "pack developer" to the "target audience" list in that issue.

natalieparellano · 2020-08-10T16:23:39Z

acceptance/acceptance_test.go

@@ -1561,18 +1590,29 @@ include = [ "*.jar", "media/mountain.jpg", "media/person.png" ]
 				var buildRunImage func(string, string, string)

 				it.Before(func() {
+					pack.JustRunSuccessfully("trust-builder", builderName)


I'm guessing this was needed because we don't support untrusted builder workflows on Windows yet (though I'm confused how it relates to rebase, could you explain?).

I'm a bit concerned about this setup because it could potentially pollute other tests (it modifies that pack config file). Is there any way to pass --trust-builder instead?

I understand the confusion. Trusting the builder is needed for the reason you mentioned, but we added this line because when("rebase") is a sibling to the when("build") where this normally happens (under when("default builder is set"), so we duplicated it here. I don't feel strongly one way or the other regarding --trust-builder instead, though the pollution argument would be a problem already (due to the when("build").

Oh, ha! I guess I'm wondering, if you remove the setup, do the rebase tests still pass? I would assume so, but maybe I'm wrong...

@natalieparellano If this were removed, the tests would pass in the Linux container case and fail in the Windows container case

Ohhh... I think I get it now. I was struggling to see how "trusted builder" has anything to do with rebase - but it looks like the test setup has a pack build that would fail if the builder is not trusted. Maybe it's worth adding a comment, so that others don't also get confused. Or you could use --trust-builder just where it's needed.

internal/build/phases.go

build_test.go

build.go

testhelpers/testhelpers.go

Signed-off-by: Andrew Meyer <meyeran@vmware.com> Signed-off-by: Malini Valliath <mvalliath@pivotal.io> Signed-off-by: Micah Young <ymicah@vmware.com> Signed-off-by: Anthony Emengo <aemengo@vmware.com> Signed-off-by: Victoria Henry <vhenry@pivotal.io>

Signed-off-by: Andrew Meyer <meyeran@vmware.com>

jromero

ameyer-pivotal requested review from jromero, natalieparellano and dfreilich July 10, 2020 20:40

dfreilich requested changes Jul 13, 2020

View reviewed changes

natalieparellano reviewed Jul 20, 2020

View reviewed changes

ameyer-pivotal marked this pull request as ready for review August 6, 2020 19:17

ameyer-pivotal requested a review from a team as a code owner August 6, 2020 19:17

ameyer-pivotal requested a review from dfreilich August 6, 2020 21:05

jromero suggested changes Aug 7, 2020

View reviewed changes

jromero linked an issue Aug 7, 2020 that may be closed by this pull request

Rebase generates new app image without platform in Image Config #770

Closed

dfreilich approved these changes Aug 7, 2020

View reviewed changes

jromero mentioned this pull request Aug 7, 2020

Add support for arbitrary writable volume mounts during pack build #778

Merged

2 tasks

natalieparellano reviewed Aug 10, 2020

View reviewed changes

ameyer-pivotal commented Aug 10, 2020

View reviewed changes

testhelpers/testhelpers.go Show resolved Hide resolved

jromero added this to the 0.13.0 milestone Aug 11, 2020

jromero added the type/enhancement Issue that requests a new feature or improvement. label Aug 11, 2020

ameyer-pivotal requested a review from jromero August 11, 2020 16:46

ameyer-pivotal added 2 commits August 11, 2020 09:48

Update build.yml for windows tests

8295786

Signed-off-by: Andrew Meyer <meyeran@vmware.com>

jromero approved these changes Aug 11, 2020

View reviewed changes

jromero merged commit 03503c9 into buildpacks:main Aug 11, 2020

ameyer-pivotal deleted the feature/470-build-windows-images branch August 11, 2020 18:26

This was referenced Aug 13, 2020

Investigate Windows specific reader closing workaround #792

Closed

Expand volume tests to occur on Windows #793

Closed

jromero added the experimental Issue or PR refers to an experimental feature. label Aug 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Windows-based images for `build` #739

Support Windows-based images for `build` #739

ameyer-pivotal commented Jul 10, 2020 •

edited

ameyer-pivotal commented Jul 10, 2020

dfreilich left a comment

ameyer-pivotal commented Jul 13, 2020 •

edited

codecov bot commented Jul 15, 2020 •

edited

natalieparellano left a comment •

edited

natalieparellano Jul 20, 2020

ameyer-pivotal Jul 24, 2020

ameyer-pivotal Aug 4, 2020

natalieparellano Aug 10, 2020

ameyer-pivotal Aug 10, 2020

jromero Aug 11, 2020 •

edited

natalieparellano Jul 20, 2020

natalieparellano Jul 20, 2020

ameyer-pivotal Jul 27, 2020

dfreilich commented Jul 27, 2020

ameyer-pivotal commented Aug 4, 2020 •

edited

jromero left a comment

dfreilich left a comment

natalieparellano left a comment

natalieparellano Aug 10, 2020

natalieparellano Aug 10, 2020

ameyer-pivotal Aug 10, 2020

natalieparellano Aug 11, 2020

ameyer-pivotal Aug 11, 2020

natalieparellano Aug 11, 2020

jromero left a comment

Support Windows-based images for build #739

Support Windows-based images for build #739

Conversation

ameyer-pivotal commented Jul 10, 2020 • edited

ameyer-pivotal commented Jul 10, 2020

dfreilich left a comment

Choose a reason for hiding this comment

ameyer-pivotal commented Jul 13, 2020 • edited

codecov bot commented Jul 15, 2020 • edited

Codecov Report

natalieparellano left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jromero Aug 11, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfreilich commented Jul 27, 2020

ameyer-pivotal commented Aug 4, 2020 • edited

jromero left a comment

Choose a reason for hiding this comment

dfreilich left a comment

Choose a reason for hiding this comment

natalieparellano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jromero left a comment

Choose a reason for hiding this comment

Support Windows-based images for `build` #739

Support Windows-based images for `build` #739

ameyer-pivotal commented Jul 10, 2020 •

edited

ameyer-pivotal commented Jul 13, 2020 •

edited

codecov bot commented Jul 15, 2020 •

edited

natalieparellano left a comment •

edited

jromero Aug 11, 2020 •

edited

ameyer-pivotal commented Aug 4, 2020 •

edited