Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows #6622

Open
laszlocsomor opened this issue Nov 7, 2018 · 6 comments

Comments

Projects
None yet
1 participant
@laszlocsomor
Copy link
Contributor

commented Nov 7, 2018

Description

The option --incompatible_windows_native_test_wrapper enables using the Bash-less test wrapper on Windows. This flag has NO effect on other platforms.

When using bazel test, Bazel does not execute tests directly. Instead Bazel runs a "test wrapper" as a subprocess, which sets up the environment for the test and runs the test.

By default, Bazel uses a Bash script (@bazel_tools//tools/test:test-setup.sh) as the test wrapper, on all platforms. The new flag allows using an alternative test wrapper written in C++ that does not depend on Bash.

When the flag is enabled, Bazel uses the C++ test wrapper from @bazel_tools//tools/test:tw. This test wrapper does not depend on Bash, but it's new and has not yet been battle-tested.

When the flag is disabled, Bazel uses the Bash script test wrapper also used on every other platform, from @bazel_tools//tools/test:test-setup.sh. This script requires Bash. It has always been the way to run tests with Bazel, so it's battle-tested.

Related bug: #5508

Backwards-compatibility

The new test wrapper is designed to be backwards-compatible with the Bash-script based one. We do not expect any incompatibilities.

Migration recipe

None, as of 2019-03-06.

If you do find that this flag breaks your tests, please let us know so we can find a solution.

Rollout plan

Bazel 0.24.0 will not support this flag.
Bazel 0.25.0 is expected to support this flag, with default value being false.
Bazel 0.26.0 is expected to flip this flag to true.
Bazel 0.27.0 is expected to remove this flag.

@laszlocsomor

This comment has been minimized.

Copy link
Contributor Author

commented Nov 7, 2018

Current status:

  • flag is called "incompatible_windows_native_test_wrapper", will be renamed by a PR soon that will reference this issue.
  • remaining logic behind the flag will be implemented in the same PR

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Nov 7, 2018

Windows, test-wrapper: rename flag in Bazel
Rename the "--windows_native_test_wrapper" flag to
"--incompatible_windows_native_test_wrapper", in
order to comply with the "Communicating Breaking
Changes in Bazel" policy (see
https://groups.google.com/forum/#!msg/bazel-dev/DKgObFj6Q-0/HRCm_xJRAAAJ)

See:
bazelbuild#5508
bazelbuild#6622

Change-Id: I7606114ad3e82da662bece63997d03d1d9c2fc8e

bazel-io pushed a commit that referenced this issue Nov 7, 2018

Windows, test-wrapper: rename flag in Bazel
Rename the "--windows_native_test_wrapper" flag to
"--incompatible_windows_native_test_wrapper", in
order to comply with the "Communicating Breaking
Changes in Bazel" policy (see
https://groups.google.com/forum/#!msg/bazel-dev/DKgObFj6Q-0/HRCm_xJRAAAJ)

The flag (and the feature it guards) is not yet in
use so this change should not affect anyone.

See:
#5508
#6622

Change-Id: I7606114ad3e82da662bece63997d03d1d9c2fc8e

Closes #6623.

Change-Id: I7606114ad3e82da662bece63997d03d1d9c2fc8e
PiperOrigin-RevId: 220445813

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Nov 7, 2018

Windows, test-wrapper: write undeclared outputs mf
In this commit:

- implement logic to write the undeclared outputs
  manifest

- add directories to the undeclared outputs zip,
  as Bazel does on Linux

Still missing:

- writing the test output XML file
- running test under a coverage collector

See:
bazelbuild#5508
bazelbuild#6622

Change-Id: I7606114ad3e82da662bece63997d03d1d9c2fc8e

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Nov 7, 2018

Windows, test-wrapper: write undeclared outputs mf
In this commit:

- implement logic to write the undeclared outputs
  manifest

- add directories to the undeclared outputs zip,
  as Bazel does on Linux

Still missing:

- writing the test output XML file
- running test under a coverage collector

See:
bazelbuild#5508
bazelbuild#6622

Change-Id: I7606114ad3e82da662bece63997d03d1d9c2fc8e

@laszlocsomor laszlocsomor changed the title Windows: enable `--incompatible_windows_native_test_wrapper` by default --incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows Feb 18, 2019

@laszlocsomor laszlocsomor changed the title --incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows Feb 18, 2019

@laszlocsomor

This comment has been minimized.

Copy link
Contributor Author

commented Mar 6, 2019

Current status: updated the bug description and added migration and breaking change labels.

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Mar 6, 2019

Windows: add --incompatible_windows_native_test_wrapper
Add the --incompatible_windows_native_test_wrapper
flag (default: false). This flag has no effect on
platforms other than Windows.

When using `bazel test`, Bazel does not execute
tests directly. Instead Bazel runs a "test
wrapper" as a subprocess, which sets up the
environment for the test and runs the test.

By default, Bazel uses a Bash script
(`@bazel_tools//tools/test:test-setup.sh`) as the
test wrapper, on all platforms. The new flag
allows using an alternative test wrapper written
in C++ that does not depend on Bash.

Flag semantics:
- When true: Bazel uses the C++ test wrapper from
  `@bazel_tools//tools/test:tw`. This test wrapper
  does not depend on Bash.
- When false: Bazel uses the Bash script test
  wrapper also used on every other platform, from
  `@bazel_tools//tools/test:test-setup.sh`. This
  script requires Bash.

Incompatible flag: bazelbuild#6622

Related: bazelbuild#5508

RELNOTES[NEW]: Added --incompatible_windows_native_test_wrapper flag: enables using the Bash-less test wrapper on Windows. (No-op on other platforms.)

bazel-io pushed a commit that referenced this issue Mar 7, 2019

Windows: add --incompatible_windows_native_test_wrapper
Add the --incompatible_windows_native_test_wrapper
flag (default: false). This flag has no effect on
platforms other than Windows.

When using `bazel test`, Bazel does not execute
tests directly. Instead Bazel runs a "test
wrapper" as a subprocess, which sets up the
environment for the test and runs the test.

By default, Bazel uses a Bash script
(`@bazel_tools//tools/test:test-setup.sh`) as the
test wrapper, on all platforms. The new flag
allows using an alternative test wrapper written
in C++ that does not depend on Bash.

Flag semantics:
- When true: Bazel uses the C++ test wrapper from
  `@bazel_tools//tools/test:tw`. This test wrapper
  does not depend on Bash.
- When false: Bazel uses the Bash script test
  wrapper also used on every other platform, from
  `@bazel_tools//tools/test:test-setup.sh`. This
  script requires Bash.

Incompatible flag: #6622

Related: #5508

RELNOTES[NEW]: Added --incompatible_windows_native_test_wrapper flag: enables using the Bash-less test wrapper on Windows. (No-op on other platforms.)

Closes #7644.

PiperOrigin-RevId: 237201616

laszlocsomor added a commit that referenced this issue Mar 7, 2019

Windows: add --incompatible_windows_native_test_wrapper
Add the --incompatible_windows_native_test_wrapper
flag (default: false). This flag has no effect on
platforms other than Windows.

When using `bazel test`, Bazel does not execute
tests directly. Instead Bazel runs a "test
wrapper" as a subprocess, which sets up the
environment for the test and runs the test.

By default, Bazel uses a Bash script
(`@bazel_tools//tools/test:test-setup.sh`) as the
test wrapper, on all platforms. The new flag
allows using an alternative test wrapper written
in C++ that does not depend on Bash.

Flag semantics:
- When true: Bazel uses the C++ test wrapper from
  `@bazel_tools//tools/test:tw`. This test wrapper
  does not depend on Bash.
- When false: Bazel uses the Bash script test
  wrapper also used on every other platform, from
  `@bazel_tools//tools/test:test-setup.sh`. This
  script requires Bash.

Incompatible flag: #6622

Related: #5508

RELNOTES[NEW]: Added --incompatible_windows_native_test_wrapper flag: enables using the Bash-less test wrapper on Windows. (No-op on other platforms.)

Closes #7644.

PiperOrigin-RevId: 237201616
@laszlocsomor

This comment has been minimized.

Copy link
Contributor Author

commented Mar 7, 2019

Bazel with downstream projects, with this flag enabled: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/844#77c3bd2d-6828-4949-82f5-f392d0e22cc0

Root causes:

  • test trying to launch .sh file (bazel-skylib)
  • test trying to launch NPM binary that has .sh launcher (rules_nodejs, rules_typescript,
  • test trying to launch binary without file extension (rules_foreign_cc, rules_typescript again, rules_sass)
  • various errors that need individual triaging (buildtools)

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Mar 18, 2019

Windows: fix tests for native test wrapper
All tests work with
`--incompatible_windows_native_test_wrapper`
except for the ones already broken on Windows
(//tests:analysis_test_e2e_test and
//tests:unittest_e2e_test).

See bazelbuild/bazel#6622

@laszlocsomor laszlocsomor changed the title incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows May 2, 2019

@laszlocsomor laszlocsomor changed the title incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows [blocked on #8203] May 2, 2019

kyliau added a commit to bazelbuild/rules_typescript that referenced this issue May 2, 2019

Update rules_nodejs to 0.28.0
This allows testing //internal:tests on Windows
with Bazel 0.25.0 rc8 and
--incompatible_windows_native_test_wrapper (see
bazelbuild/bazel#6622).

irengrig added a commit to irengrig/bazel that referenced this issue May 3, 2019

Windows: fix native test wrapper's arg. escaping
The native test wrapper now correctly escapes the
arguments for the subprocess, using
bazel::launcher::WindowsEscapeArg2 from the native
launcher.

The test_wrapper_test now uses a mock C++ binary
instead of a .bat file to verify the test
arguments. Doing so trades the weird command line
flag parsing logic of cmd.exe (inherent in using a
.bat file) for the saner C++ flag parsing one.

Fixes bazelbuild#7956
Unblocks bazelbuild#6622

Closes bazelbuild#7957.

PiperOrigin-RevId: 242446422

@laszlocsomor laszlocsomor added the P1 label May 16, 2019

@laszlocsomor

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

P1 because I really need to resolve this for Bazel 0.27

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue May 16, 2019

file_test, rule_test: are now sh_test rules
All test rules in
@bazel_tools//tools/build_rule:test_rules.bzl are
now macros around sh_test.

This allows running them on Windows with the
Windows-native test wrapper.

Fixes bazelbuild#8203
Unblocks bazelbuild#6622

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue May 16, 2019

file_test, rule_test: are now sh_test rules
All test rules in
@bazel_tools//tools/build_rule:test_rules.bzl are
now macros around sh_test.

This allows running them on Windows with the
Windows-native test wrapper.

Fixes bazelbuild#8203
Unblocks bazelbuild#6622

laszlocsomor added a commit to laszlocsomor/rules_foreign_cc that referenced this issue May 16, 2019

laszlocsomor added a commit to laszlocsomor/rules_go that referenced this issue May 16, 2019

Windows: future-proof single_output_test.bzl
Fix single_output_test() on Windows with the new native test wrapper.

Bazel 0.26 introduces the
--incompatible_windows_native_test_wrapper flag.

This feature enables a new test wrapper on Windows
that doesn't require Bash. On other platforms this
flag has no effect.

The new test wrapper is a C++ binary and doesn't
use Bash, but therefore test rules whose
executable is a naked shell script also don't
work.

The solution here is to write a ".bat" file,
because that can be executed on every platform
(see inline comments).

See bazelbuild/bazel#6622

laszlocsomor added a commit to laszlocsomor/rules_go that referenced this issue May 16, 2019

Windows: future-proof single_output_test.bzl
Fix single_output_test() on Windows with the new native test wrapper.

Bazel 0.26 introduces the
--incompatible_windows_native_test_wrapper flag.

This feature enables a new test wrapper on Windows
that doesn't require Bash. On other platforms this
flag has no effect.

The new test wrapper is a C++ binary and doesn't
use Bash, but therefore test rules whose
executable is a naked shell script also don't
work.

The solution here is to write a ".bat" file,
because that can be executed on every platform
(see inline comments).

See bazelbuild/bazel#6622

laszlocsomor added a commit to bazelbuild/rules_go that referenced this issue May 16, 2019

Windows: future-proof single_output_test.bzl (#2064)
Fix single_output_test() on Windows with the new
native test wrapper.

Bazel 0.26 introduces the
--incompatible_windows_native_test_wrapper flag.

This feature enables a new test wrapper on Windows
that doesn't require Bash. On other platforms this
flag has no effect.

The new test wrapper is a C++ binary and doesn't
use Bash, but therefore test rules whose
executable is a naked shell script also don't
work.

The solution here is to write a special script
that does nothing and is valid both as .bat and
as .sh.

See bazelbuild/bazel#6622

jayconrod added a commit to bazelbuild/rules_go that referenced this issue May 16, 2019

Windows: future-proof single_output_test.bzl (#2064)
Fix single_output_test() on Windows with the new
native test wrapper.

Bazel 0.26 introduces the
--incompatible_windows_native_test_wrapper flag.

This feature enables a new test wrapper on Windows
that doesn't require Bash. On other platforms this
flag has no effect.

The new test wrapper is a C++ binary and doesn't
use Bash, but therefore test rules whose
executable is a naked shell script also don't
work.

The solution here is to write a special script
that does nothing and is valid both as .bat and
as .sh.

See bazelbuild/bazel#6622

jayconrod added a commit to bazelbuild/rules_go that referenced this issue May 16, 2019

Windows: future-proof single_output_test.bzl (#2064)
Fix single_output_test() on Windows with the new
native test wrapper.

Bazel 0.26 introduces the
--incompatible_windows_native_test_wrapper flag.

This feature enables a new test wrapper on Windows
that doesn't require Bash. On other platforms this
flag has no effect.

The new test wrapper is a C++ binary and doesn't
use Bash, but therefore test rules whose
executable is a naked shell script also don't
work.

The solution here is to write a special script
that does nothing and is valid both as .bat and
as .sh.

See bazelbuild/bazel#6622

bazel-io pushed a commit that referenced this issue May 17, 2019

file_test, rule_test: now as sh_test rules
All test rules in
@bazel_tools//tools/build_rule:test_rules.bzl are
now macros around sh_test.

This allows running them on Windows with the
Windows-native test wrapper.

Fixes #8203
Unblocks #6622

Closes #8352.

PiperOrigin-RevId: 248680587

@laszlocsomor laszlocsomor changed the title incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows [blocked on #8203] incompatible_windows_native_test_wrapper: enables Bash-less test execution on Windows May 17, 2019

@laszlocsomor

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

Last blocker: bazelbuild/rules_foreign_cc#263 -- just needs to get merged, then the project will be fixed. Then I can flip --incompatible_windows_native_test_wrapper at HEAD.

The "Bazel / Bazel@HEAD + Downstream" project on BuildKite uses Bazel from HEAD, so that will have both #8203 and the flipped flag.

Bazel 0.27 will roll out the flipped flag, as well as #8203.

bazel-io pushed a commit that referenced this issue May 17, 2019

Automated rollback of commit 76583ee.
*** Reason for rollback ***

Breaks some of Google's own tests

*** Original change description ***

file_test, rule_test: now as sh_test rules

All test rules in
@bazel_tools//tools/build_rule:test_rules.bzl are
now macros around sh_test.

This allows running them on Windows with the
Windows-native test wrapper.

Fixes #8203
Unblocks #6622

Closes #8352.

PiperOrigin-RevId: 248682341

irengrig added a commit to bazelbuild/rules_foreign_cc that referenced this issue May 17, 2019

Windows: tests now work with native test wrapper (#263)
Windows: tests now work with native test wrapper

See bazelbuild/bazel#6622

bazel-io pushed a commit that referenced this issue May 17, 2019

file_test, rule_test: now as sh_test rules
This commit rolls forward commit 76583ee,
which was rolled back by commit 28f8af7.

Differences:
- test_rules.bzl:success_target and failure_target
  no longer dump 'msg' into a file and print that.
  Instead, they print it as a heredoc.
- The same functions no longer need runfiles.bash.
  The breakage the original commit caused was that
  some Starlark rule used success_target but
  itself didn't depend on the Bash runfiles
  library, and success_target returns a
  DefaultInfo without the runfiles library in it.
  To keep existing success_target and
  failure_target calls intact, I opted to remove
  the dependency on runfiles.bash
- marked all "_impl" rules as testonly=1

Original commit message follows.

All test rules in
@bazel_tools//tools/build_rule:test_rules.bzl are
now macros around sh_test.

This allows running them on Windows with the
Windows-native test wrapper.

Fixes #8203
Unblocks #6622

Closes #8352.

PiperOrigin-RevId: 248691640
@laszlocsomor

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.