New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows: Bash-less test execution #5508

Open
laszlocsomor opened this Issue Jul 3, 2018 · 1 comment

Comments

Projects
None yet
1 participant
@laszlocsomor
Copy link
Contributor

laszlocsomor commented Jul 3, 2018

Description of the problem / feature request:

Design and implement a way to run tests without requiring tools/test/test-setup.sh.

Current functionality includes:

  • setting up environment variables, including making execution-root-relative paths absolute
  • handling signals by post-processing the test's output and transforming into an XML file
  • running the test either in a subshell or directly as a subprocess, and streaming its output to stdout
  • zipping up undeclared output files and writing the undeclared outputs manifest

I think the most complicated item is signal handling: on Windows, Bazel calls TerminateProcess when the user interrupts the test, which kills the test process without a chance to clean up. We may need to run two processes per test on Windows, one running the test and the other doing the cleanup work as soon as the first process is terminated.

Feature requests: what underlying problem are you trying to solve with this feature?

Motivation: #4319

Deliverables: (must be added to / linked from this GH issue):

  • doc explaining current test-setup.sh functionality
  • design doc explaining proposed Windows native approach and how the approach implements all important features of test-setup.sh
  • log of outcome of manual testing on a clean Windows installation without Bash
  • commits implementing this feature nativly on Windows
  • commits adding integration and regression tests for all features of test-setup.sh
  • commit dropping test-setup.sh usage on Windows and using only the new code path

Signals:

  • running the newly added tests
  • manual testing: running "bazel test //foo:bar_test" on a clean Windows installation without Bash

What operating system are you running Bazel on?

Windows 10

What's the output of bazel info release?

release 0.15.0
@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Jul 18, 2018

doc explaining current test-setup.sh functionality

laszlocsomor/proposals@332fa53

I won't merge this branch, instead I'll embed the text in the eventual design doc.

laszlocsomor added a commit to laszlocsomor/proposals that referenced this issue Jul 24, 2018

laszlocsomor added a commit to laszlocsomor/proposals that referenced this issue Jul 24, 2018

laszlocsomor added a commit to laszlocsomor/proposals that referenced this issue Jul 24, 2018

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 7, 2018

Windows: add skeleton for Bash-less test wrapper
In this commit:

- Add the not-yet-documented
  `--windows_native_test_wrapper` flag. This flag
  will guard the new feature and allow the Bazel
  team to roll it out and deprecate the old test
  execution mechanism gradually.

- Add a target for the native implementation of
  the test wrapper.

- Hook up the test action creator logic to use the
  new binary, and add a test to assert this.

See bazelbuild#5508

Change-Id: I980a1a88cfabd04ddcf8a5b26620f8f3255f77ef

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 7, 2018

Windows: add skeleton for Bash-less test wrapper
In this commit:

- Add the not-yet-documented
  `--windows_native_test_wrapper` flag. This flag
  will guard the new feature and allow the Bazel
  team to roll out the feature gradually, as well
  as to deprecate the old test execution mechanism
  gradually.

- Add a target for the native implementation of
  the test wrapper.

- Hook up the test action creator logic to use the
  new binary, and add a test to assert this.

See bazelbuild#5508

Change-Id: I980a1a88cfabd04ddcf8a5b26620f8f3255f77ef

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 7, 2018

Windows: add skeleton for Bash-less test wrapper
In this commit:

- Add the not-yet-documented
  `--windows_native_test_wrapper` flag. This flag
  will guard the new feature and allow the Bazel
  team to roll out the feature gradually, as well
  as to deprecate the old test execution mechanism
  gradually.

- Add a target for the native implementation of
  the test wrapper.

- Hook up the test action creator logic to use the
  new binary, and add a test to assert this.

See bazelbuild#5508

Change-Id: I980a1a88cfabd04ddcf8a5b26620f8f3255f77ef

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 7, 2018

Windows: add skeleton for Bash-less test wrapper
In this commit:

- Add the not-yet-documented
  `--windows_native_test_wrapper` flag. This flag
  will guard the new feature and allow the Bazel
  team to roll out the feature gradually, as well
  as to deprecate the old test execution mechanism
  gradually.

- Add a target for the native implementation of
  the test wrapper.

- Hook up the test action creator logic to use the
  new binary, and add a test to assert this.

See bazelbuild#5508

Change-Id: I980a1a88cfabd04ddcf8a5b26620f8f3255f77ef

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 7, 2018

Create BUILD.tools for //tools/test package.
Motivation: as part of fixing issue bazelbuild#5508, I'll
add a cc_binary to this package. The binary will
be embedded in Bazel as a prebuilt binary (to
avoid requiring a C++ compiler to use it).
Therefore //tools/test/BUILD and
@bazel_tools//tools/test/BUILD will contain
different rules, mandating the BUILD file split.

See bazelbuild#5508

Change-Id: If21bafbc3d83d59e52de498cf3efd2b1b79fa7b6

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 8, 2018

test wrapper: add empty implementation
Add empty implementation of the Windows-native
test wrapper, add it to the embedded tools, and
depend on it from test rules.

See bazelbuild#5508
See https://github.com/laszlocsomor/proposals/blob/win-test-runner/designs/2018-07-18-windows-native-test-runner.md

Change-Id: I15ce28beb6eb98cb9dd60e2158b906c5b0ae9bb9

bazel-io pushed a commit that referenced this issue Aug 9, 2018

Create BUILD.tools for //tools/test package.
Motivation: as part of fixing issue #5508, I'll
add a cc_binary to this package. The binary will
be embedded in Bazel as a prebuilt binary (to
avoid requiring a C++ compiler to use it).
Therefore //tools/test/BUILD and
@bazel_tools//tools/test/BUILD will contain
different rules, mandating the BUILD file split.

See #5508

Change-Id: If21bafbc3d83d59e52de498cf3efd2b1b79fa7b6

Closes #5787.

Change-Id: If21bafbc3d83d59e52de498cf3efd2b1b79fa7b6
PiperOrigin-RevId: 208005382

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 10, 2018

test-wrapper: add empty impl + Bazel flag
This commit adds:

- the skeleton implementation of the Windows
  native test wrapper

- the --windows_native_test_wrapper Bazel flag,
  which is currently a no-op

See bazelbuild#5508

Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 10, 2018

test-wrapper: add empty impl + Bazel flag
This commit adds:

- the skeleton implementation of the Windows
  native test wrapper

- a depenency on the native test wrapper from test
  rules, through the new $test_wrapper rule
  attribute

- the --windows_native_test_wrapper Bazel flag,
  which is currently a no-op

See bazelbuild#5508

Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 10, 2018

test-wrapper: add empty impl + Bazel flag
This commit adds:

- the skeleton implementation of the Windows
  native test wrapper

- a depenency on the native test wrapper from test
  rules, through the new $test_wrapper rule
  attribute

- the --windows_native_test_wrapper Bazel flag,
  which is currently a no-op

See bazelbuild#5508

Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 10, 2018

test-wrapper: add empty impl + Bazel flag
This commit adds:

- the skeleton implementation of the Windows
  native test wrapper

- a depenency on the native test wrapper from test
  rules, through the new $test_wrapper rule
  attribute

- the --windows_native_test_wrapper Bazel flag,
  which is currently a no-op

See bazelbuild#5508

Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 10, 2018

test-wrapper: add empty impl + Bazel flag
This commit adds:

- the skeleton implementation of the Windows
  native test wrapper

- a depenency on the native test wrapper from test
  rules, through the new $test_wrapper rule
  attribute

- the --windows_native_test_wrapper Bazel flag,
  which is currently a no-op

See bazelbuild#5508

Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e

bazel-io pushed a commit that referenced this issue Aug 14, 2018

test-wrapper: add empty impl + Bazel flag
This commit adds:

- the skeleton implementation of the Windows
  native test wrapper

- a depenency on the native test wrapper from test
  rules, through the new $test_wrapper rule
  attribute

- the --windows_native_test_wrapper Bazel flag,
  which is currently a no-op

See #5508

Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e

Closes #5854.

Change-Id: I2ffc78bceec5dd867af775b5878f105fa87c3dba
PiperOrigin-RevId: 208650699

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 17, 2018

test execution: use Windows-native test_wrapper
Use the not-yet-functional test_wrapper on Windows
when --windows_native_test_wrapper is enabled,
otherwise use the test-setup.sh script like other
platforms.

See bazelbuild#5508

Change-Id: I961b5a10b92758045aef4cc6a1fca125a82f7e5b

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Aug 17, 2018

test execution: use Windows-native test_wrapper
Use the not-yet-functional test_wrapper on Windows
when --windows_native_test_wrapper is enabled,
otherwise use the test-setup.sh script like other
platforms.

See bazelbuild#5508

Change-Id: I961b5a10b92758045aef4cc6a1fca125a82f7e5b

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Nov 22, 2018

Windows, test wrapper: impl. split XML generation
With the --experimental_split_xml_generation flag,
Bazel can generate a test xml in case the
test-setup.sh or the test itself failed to do so.

Now this functionality is available on Windows
too, without the aid of generate-xml.sh

See bazelbuild#5508

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Nov 22, 2018

Windows, test wrapper: impl. split XML generation
With the --experimental_split_xml_generation flag,
Bazel can generate a test xml in case the
test-setup.sh or the test itself failed to do so.

Now this functionality is available on Windows
too, without the aid of generate-xml.sh

See bazelbuild#5508

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Nov 22, 2018

Windows, test wrapper: impl. split XML generation
With the --experimental_split_xml_generation flag,
Bazel can generate a test xml in case the
test-setup.sh or the test itself failed to do so.

Now this functionality is available on Windows
too, without the aid of generate-xml.sh

See bazelbuild#5508

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Dec 3, 2018

unittest.bzl: supports Windows
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- add the "is_host_windows" attribute to all
  unittest.make-created test rules, and set its
  value in unittest.end() from a select()
  statement

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  value of "is_host_windows"

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Dec 3, 2018

unittest.bzl: supports Windows
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- add the "is_host_windows" attribute to all
  unittest.make-created test rules, and set its
  value in unittest.end() from a select()
  statement

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  value of "is_host_windows"

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Dec 3, 2018

unittest.bzl: supports Windows
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- add the "is_host_windows" attribute to all
  unittest.make-created test rules, and set its
  value in unittest.end() from a select()
  statement

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  value of "is_host_windows"

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Dec 3, 2018

unittest.bzl: supports Windows
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- define a new toolchain_type and toolchain rules
  for Windows and non-Windows ("default")
  execution platforms

- register the new toolchains in workspace.bzl

- let unittest.make-created test rules require the
  new toolchain_type

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  selected toolchain

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Dec 3, 2018

unittest.bzl: supports Windows
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- define a new toolchain_type and toolchain rules
  for Windows and non-Windows ("default")
  execution platforms

- register the new toolchains in workspace.bzl

- let unittest.make-created test rules require the
  new toolchain_type

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  selected toolchain

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to laszlocsomor/bazel-skylib that referenced this issue Dec 3, 2018

unittest.bzl: supports Windows
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- define a new toolchain_type and toolchain rules
  for Windows and non-Windows ("default")
  execution platforms

- register the new toolchains in workspace.bzl

- let unittest.make-created test rules require the
  new toolchain_type

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  selected toolchain

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to bazelbuild/bazel-skylib that referenced this issue Dec 4, 2018

unittest.bzl: supports Windows (#84)
In this commit:

- change unittest.bzl to declare a named output
  file instead of relying on the deprecated [1]
  default output name (ctx.outputs.executable).

- define a new toolchain_type and toolchain rules
  for cmd.exe and for Bash (basically Windows and
  non-Windows)

- register the new toolchains in workspace.bzl

- let unittest.make-created test rules require the
  new toolchain_type

- write the test output script as a Windows batch
  script or as a Shell script, depending on the
  selected toolchain

This PR enables the Bazel team to break the Bash
dependency (for test execution) on Windows, and
can run Starlark unittests with the new,
Windows-native test wrapper (still under
development).

See bazelbuild/bazel#5508

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Dec 7, 2018

Windows, test wrapper: rename the associated flag
Rename:
--[no]incompatible_windows_native_test_wrapper
to:
--[no]experimental_windows_native_test_wrapper

in order to avoid triggering this flag with
--all_incompatible_changes.

The test wrapper is not yet ready for general use
so turning it on with this flag (e.g. on CI) is
undesirable.

See bazelbuild#5508

bazel-io pushed a commit that referenced this issue Dec 7, 2018

Windows, test wrapper: rename the associated flag
Rename:
--[no]incompatible_windows_native_test_wrapper
to:
--[no]experimental_windows_native_test_wrapper

in order to avoid triggering this flag with
--all_incompatible_changes.

The test wrapper is not yet ready for general use
so turning it on with this flag (e.g. on CI) is
undesirable.

See #5508
Fixes #6859

Closes #6858.

PiperOrigin-RevId: 224490751

laurentlb added a commit that referenced this issue Dec 14, 2018

Windows, test wrapper: rename the associated flag
Rename:
--[no]incompatible_windows_native_test_wrapper
to:
--[no]experimental_windows_native_test_wrapper

in order to avoid triggering this flag with
--all_incompatible_changes.

The test wrapper is not yet ready for general use
so turning it on with this flag (e.g. on CI) is
undesirable.

See #5508
Fixes #6859

Closes #6858.

PiperOrigin-RevId: 224490751

laurentlb added a commit that referenced this issue Dec 17, 2018

Windows, test wrapper: rename the associated flag
Rename:
--[no]incompatible_windows_native_test_wrapper
to:
--[no]experimental_windows_native_test_wrapper

in order to avoid triggering this flag with
--all_incompatible_changes.

The test wrapper is not yet ready for general use
so turning it on with this flag (e.g. on CI) is
undesirable.

See #5508
Fixes #6859

Closes #6858.

PiperOrigin-RevId: 224490751

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jan 7, 2019

Windows, test wrapper: native split-XML generation
Replace generate-xml.sh by a native XML generator
when using --experimental_split_xml_generation.

The XML generator is almost the same binary as the
native test wrapper, just with a different main
function.

See bazelbuild#5508

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jan 7, 2019

Windows, test wrapper: native split-XML generation
Replace generate-xml.sh by a native XML generator
when using --experimental_split_xml_generation.

The XML generator is almost the same binary as the
native test wrapper, just with a different main
function.

See bazelbuild#5508

bazel-io pushed a commit that referenced this issue Jan 10, 2019

Windows, test wrapper: native split-XML generation
Replace generate-xml.sh by a native XML generator
when using --experimental_split_xml_generation
with --experimental_windows_native_test_wrapper
on Windows.

The XML generator is almost the same binary as the
native test wrapper, just with a different main
function.

See #5508

Closes #7048.

PiperOrigin-RevId: 228651495

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jan 14, 2019

Windows: fix rule_size_test for the test wrapper
Fix //src:rule_size_test so it works when using
--experimental_windows_native_test_wrapper.

The test failed with the Windows-native test
wrapper because the test wrapper launches the test
binary directly, and the rule created an empty
.bat file as its executable, and CreateProcessW
apparently cannot run empty .bat files.

Also: log the failed command in the test wrapper
when CreateProcessW fails.

See bazelbuild#5508

bazel-io pushed a commit that referenced this issue Jan 14, 2019

Windows: fix rule_size_test for the test wrapper
Fix //src:rule_size_test so it works when using
--experimental_windows_native_test_wrapper.

The test failed with the Windows-native test
wrapper because the test wrapper launches the test
binary directly, and the rule created an empty
.bat file as its executable, and CreateProcessW
apparently cannot run empty .bat files.

Also: log the failed command in the test wrapper
when CreateProcessW fails.

See #5508

Closes #7112.

PiperOrigin-RevId: 229162712

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jan 15, 2019

Windows: fix tests to work with test wrapper
Fix the shell tests to work with
--experimental_windows_native_test_wrapper (when
using Bazel built at HEAD).

In this change:

- Remove the is_windows function definition from
  testenv.sh, to ensure that tests wanting to
  check the platform will define their own
  is_windows variable (as many tests already do).

- Define the is_windows variable using `uname` in
  every test that needs to know the platform.

- Use the Bash runfiles library in the affected
  tests.

- Every test that calls "bazel shutdown" now uses
  it with try_with_timeout, so tests won't fail
  flakily when bazel cannot immediately be shut
  down.

See bazelbuild#5508

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jan 16, 2019

Windows: fix tests to work with test wrapper
Fix the shell tests to work with
--experimental_windows_native_test_wrapper (when
using Bazel built at HEAD).

In this change:

- Remove the is_windows function definition from
  testenv.sh, to ensure that tests wanting to
  check the platform will define their own
  is_windows variable (as many tests already do).

- Define the is_windows variable using `uname` in
  every test that needs to know the platform.

- Use the Bash runfiles library in the affected
  tests.

- Every test that calls "bazel shutdown" now uses
  it with try_with_timeout, so tests won't fail
  flakily when bazel cannot immediately be shut
  down.

See bazelbuild#5508

bazel-io pushed a commit that referenced this issue Jan 16, 2019

Windows: fix tests to work with test wrapper
Fix the shell tests to work with
--experimental_windows_native_test_wrapper (when
using Bazel built at HEAD).

In this change:

- Remove the is_windows function definition from
  testenv.sh, to ensure that tests wanting to
  check the platform will define their own
  is_windows variable (as many tests already do).

- Define the is_windows variable using `uname` in
  every test that needs to know the platform.

- Use the Bash runfiles library in the affected
  tests.

- Every test that calls "bazel shutdown" now uses
  it with try_with_timeout, so tests won't fail
  flakily when bazel cannot immediately be shut
  down.

See #5508

Closes #7126.

PiperOrigin-RevId: 229539256
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment