Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatible_py2_outputs_are_suffixed: Bazel puts Python 2 targets in a separate output root and Python 3 targets in the "normal" one #7593

Closed
brandjon opened this Issue Feb 28, 2019 · 1 comment

Comments

Projects
None yet
2 participants
@brandjon
Copy link
Member

brandjon commented Feb 28, 2019

Flag: --incompatible_py2_outputs_are_suffixed
Available since: 0.24
Will be flipped in: 0.25
Tracking issue: #6647

It is recommended to enable this flag at the same time as --incompatible_py3_is_default.

Motivation

When the related flag --incompatible_py3_is_default is enabled, targets that do not specifically declare that they are for Python 2, or that are dependencies of such targets, get built in the Python 3 configuration. This includes every target in the build that is not underneath a py_binary or py_test target.

PY2 and PY3 outputs get put in different directories so as to avoid conflicts. Traditionally, PY2 targets are usually reachable via the bazel-bin symlink, while PY3 targets get put in a directory whose name includes the suffix "-py3" and which is not pointed to by bazel-bin. This means that if --incompatible_py3_is_default is enabled, the majority of targets are not reachable under bazel-bin.

Change

When --incompatible_py2_outputs_are_suffixed is enabled, outputs built in the Python 2 configuration are placed in a directory named with a "-py2" suffix, and outputs built for Python 3 do not get a Python-related suffix. This means that bazel-bin will point to PY3 targets instead of PY2.

This change does not affect targets built in the host configuration, which has its own directory.

Migration

Since this only affects the naming of outputs, most use cases should not be affected. However, any automated script that looks at the bazel-bin symlink or a hardcoded output root path may need to be updated. In general, scripts should not rely on these directories having any particular naming convention.

A more robust way to obtain the output root associated with a target's outputs is to use bazel info, e.g.

$ bazel info bazel-bin
[...]/execroot/io_bazel/bazel-out/k8-fastbuild/bin

$ bazel info bazel-bin --python_version=PY3
[...]/execroot/io_bazel/bazel-out/k8-py3-fastbuild/bin

$ bazel info bazel-bin --python_version=PY3 --incompatible_py2_outputs_are_suffixed
[...]/execroot/io_bazel/bazel-out/k8-fastbuild/bin

This requires that you pass the same build args to bazel info as are passed to bazel build (in particular, --incompatible_py2_outputs_are_suffixed), and that you match the value of --python_version=... to the configuration of the target whose outputs you're trying to locate.

@brandjon brandjon self-assigned this Feb 28, 2019

bazel-io pushed a commit that referenced this issue Feb 28, 2019

Add incompatible flag to make PY2 outputs suffixed
This adds --incompatible_py2_outputs_are_suffixed, which makes it so Python 2 targets get built under an output root with the "-py2" suffix, and Python 3 targets get built under a root with no additional suffix. This means that the `bazel-bin` symlink contains Python 3 targets instead of Python 2 targets.

This goes hand-in-hand with --incompatible_py3_is_default, which controls whether targets get built in the Python 2 or Python 3 configuration by default. Enabling just one flag or the other will cause the majority of targets to get built under an output root other than the one pointed to by `bazel-bin`, which is not very convenient.

In general users and scripts are not supposed to care what the exact output path of a target is, but in practice they do, hence why this is an incompatible change.

Work toward #6647. See #7593 for migration info.

RELNOTES[INC]: Added --incompatible_py2_outputs_are_suffixed, for switching the bazel-bin symlink to point to Python 3 outputs instead of Python 2 outputs. See [#7593](#7593).

PiperOrigin-RevId: 236207234
@brandjon

This comment has been minimized.

Copy link
Member Author

brandjon commented Mar 13, 2019

According to downstream presubmit results, looks like we will indeed be flipping this in 0.25.

bazel-io pushed a commit that referenced this issue Mar 26, 2019

Release 0.24.0 (2019-03-26)
Baseline: 235e76b

Cherry picks:

   + badd82e:
     Automated rollback of commit
     1b4c37c.
   + 33e5719:
     Fix the Python version select() mechanism to handle
     PY3-as-default
   + 56366ee:
     Set non-empty values for msvc_env_* when VC not installed
   + 22b3fbf:
     Windows, test wrapper: fix broken integration test
   + f14d447:
     Add whitelist file for starlark transitions
   + d99bc47:
     Update BUILD
   + 3529ad7:
     Rename tools/function_transition_whitelist/BUILD to
     tools/whitelists/function_transition_whitelist/BUILD
   + de0612a:
     Update bazel_toolchains to latest release and add toolchain
     config target for BuildKite CI (rbe_ubuntu1604)
   + 3e660ad:
     Automated rollback of commit
     0877340.
   + 314cf1f:
     Pass -undefined dynamic_lookup to dynamic library linking
     actions on Mac
   + fc586a8:
     Move cc_flags target into @bazel_tools//tools/cpp.
   + ea1703b:
     C++: Fix crash reported in #7721
   + 803801d:
     Pass execution info to xml generating spawn. Fixes #7794

Incompatible changes:

  - Added --incompatible_py3_is_default to test switching the default
    Python version to PY3 for py_binary/py_test targets that do not
    specify a version. See #7359.
  - //tools/cmd_line_differ has been renamed to //tools/aquery_differ
    & can now compare (in addition to command lines) inputs of
    actions given 2 aquery results.
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - The flag --incompatible_list_based_execution_strategy_selection
    was added and is used to ease the migration to the new style of
    specifying
    execution strategy selection and fallback behavior. The
    documentation for
    this flag is here: #7480
  - Added --incompatible_py2_outputs_are_suffixed, for switching the
    bazel-bin symlink to point to Python 3 outputs instead of Python
    2 outputs. See
    [#7593](#7593).

New features:

  - Make actions.args() object chainable.
  - Added --incompatible_windows_style_arg_escaping flag: enables
    correct subprocess argument escaping on Windows. (No-op on other
    platforms.)
  - Added --incompatible_windows_escape_jvm_flags flag: enables
    correct java_binary.jvm_flags and java_test.jvm_flags
    tokenization and escaping on Windows. (No-op on other platforms.)

Important changes:

  - Allow running aquery against the current state of Skyframe
  - Added support for the "navigation" resource directory to Android
    resource processing in Bazel. This is used by the Navigation
    Architecture Component.
  - --incompatible_disable_runtimes_filegroups was flipped
    (#6942)
  - Incompatible flag `--incompatible_linkopts_in_user_link_flags`
    has been flipped (#6826)
  - Incompatible flag `--incompatible_dont_emit_static_libgcc` has
    been flipped (#6825)
  - --incompatible_disallow_filetype is enabled by default.
  - Fixed issue where exceptions and stacktraces were silently
    swallowed in the Android resource processing pipeline.
  - `--incompatible_disable_expand_if_all_available_in_flag_set` has
    been flipped (#7008)
  - --incompatible_disallow_dict_plus is enabled by default
  - Adds --incompatible_disable_objc_library_resources to disable
    resource attributes in objc_library. Please migrate these
    attributes to `data` instead.
  - --incompatible_disallow_old_style_args_add is enabled by default.
  - Using the `native` module in BUILD files is deprecated. It will
    be forbidden with --incompatible_disallow_native_in_build_file.
  - (Python rules) PyRuntimeInfo is exposed to Starlark, making it
    possible for Starlark rules to depend on or imitate `py_runtime`.
    The `files` attribute of `py_runtime` is no longer mandatory.
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured
    targetshttps://github.com//issues/7186.
  - is_using_fission crosstool variable is now exposed in all compile
    actions when fission is active (it used to be exposed only for
    linking actions).
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured
    targetshttps://github.com//issues/7186.
  - `py_runtime` gains a `python_version` attribute for specifying
    whether it represents a Python 2 or 3 interpreter.
  - `--incompatible_java_coverage` is enabled by default.
  - Starlark rules can safely declare attributes named "licenses"
  - When using
    --incompatible_list_based_execution_strategy_selection, Bazel
    will use remote execution by default (if you specify
    --remote_executor), otherwise persistent workers (if the action
    supports it), otherwise sandboxed local execution (if the action
    and platform supports it) and at last unsandboxed local
    execution. The flags --spawn_strategy and --strategy continue to
    work as before - this only sets new defaults for the case where
    you don't specify these flags.
  - Set default value of --incompatible_remap_main_repo to true.
  - Set default value of --incompatible_remap_main_repo to true.

This release contains contributions from many people at Google, as well as Andrew Suffield, Brandon Lico, Chris Eason, Clint Harrison, Ed Schouten, Garrett Hopper, George Gensure, Greg, John Millikin, Julie, Keith Smiley, Laurent Le Brun, Ryan Beasley, Shmuel H, Travis Cline, Vladimir Chebotarev.

katre added a commit that referenced this issue Mar 29, 2019

Release 0.24.0 (2019-03-26)
Baseline: 235e76b

Cherry picks:

   + badd82e:
     Automated rollback of commit
     1b4c37c.
   + 33e5719:
     Fix the Python version select() mechanism to handle
     PY3-as-default
   + 56366ee:
     Set non-empty values for msvc_env_* when VC not installed
   + 22b3fbf:
     Windows, test wrapper: fix broken integration test
   + f14d447:
     Add whitelist file for starlark transitions
   + d99bc47:
     Update BUILD
   + 3529ad7:
     Rename tools/function_transition_whitelist/BUILD to
     tools/whitelists/function_transition_whitelist/BUILD
   + de0612a:
     Update bazel_toolchains to latest release and add toolchain
     config target for BuildKite CI (rbe_ubuntu1604)
   + 3e660ad:
     Automated rollback of commit
     0877340.
   + 314cf1f:
     Pass -undefined dynamic_lookup to dynamic library linking
     actions on Mac
   + fc586a8:
     Move cc_flags target into @bazel_tools//tools/cpp.
   + ea1703b:
     C++: Fix crash reported in #7721
   + 803801d:
     Pass execution info to xml generating spawn. Fixes #7794

Incompatible changes:

  - Added --incompatible_py3_is_default to test switching the default
    Python version to PY3 for py_binary/py_test targets that do not
    specify a version. See #7359.
  - //tools/cmd_line_differ has been renamed to //tools/aquery_differ
    & can now compare (in addition to command lines) inputs of
    actions given 2 aquery results.
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - The flag --incompatible_list_based_execution_strategy_selection
    was added and is used to ease the migration to the new style of
    specifying
    execution strategy selection and fallback behavior. The
    documentation for
    this flag is here: #7480
  - Added --incompatible_py2_outputs_are_suffixed, for switching the
    bazel-bin symlink to point to Python 3 outputs instead of Python
    2 outputs. See
    [#7593](#7593).

New features:

  - Make actions.args() object chainable.
  - Added --incompatible_windows_style_arg_escaping flag: enables
    correct subprocess argument escaping on Windows. (No-op on other
    platforms.)
  - Added --incompatible_windows_escape_jvm_flags flag: enables
    correct java_binary.jvm_flags and java_test.jvm_flags
    tokenization and escaping on Windows. (No-op on other platforms.)

Important changes:

  - Allow running aquery against the current state of Skyframe
  - Added support for the "navigation" resource directory to Android
    resource processing in Bazel. This is used by the Navigation
    Architecture Component.
  - --incompatible_disable_runtimes_filegroups was flipped
    (#6942)
  - Incompatible flag `--incompatible_linkopts_in_user_link_flags`
    has been flipped (#6826)
  - Incompatible flag `--incompatible_dont_emit_static_libgcc` has
    been flipped (#6825)
  - --incompatible_disallow_filetype is enabled by default.
  - Fixed issue where exceptions and stacktraces were silently
    swallowed in the Android resource processing pipeline.
  - `--incompatible_disable_expand_if_all_available_in_flag_set` has
    been flipped (#7008)
  - --incompatible_disallow_dict_plus is enabled by default
  - Adds --incompatible_disable_objc_library_resources to disable
    resource attributes in objc_library. Please migrate these
    attributes to `data` instead.
  - --incompatible_disallow_old_style_args_add is enabled by default.
  - Using the `native` module in BUILD files is deprecated. It will
    be forbidden with --incompatible_disallow_native_in_build_file.
  - (Python rules) PyRuntimeInfo is exposed to Starlark, making it
    possible for Starlark rules to depend on or imitate `py_runtime`.
    The `files` attribute of `py_runtime` is no longer mandatory.
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured
    targetshttps://github.com//issues/7186.
  - is_using_fission crosstool variable is now exposed in all compile
    actions when fission is active (it used to be exposed only for
    linking actions).
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured
    targetshttps://github.com//issues/7186.
  - `py_runtime` gains a `python_version` attribute for specifying
    whether it represents a Python 2 or 3 interpreter.
  - `--incompatible_java_coverage` is enabled by default.
  - Starlark rules can safely declare attributes named "licenses"
  - When using
    --incompatible_list_based_execution_strategy_selection, Bazel
    will use remote execution by default (if you specify
    --remote_executor), otherwise persistent workers (if the action
    supports it), otherwise sandboxed local execution (if the action
    and platform supports it) and at last unsandboxed local
    execution. The flags --spawn_strategy and --strategy continue to
    work as before - this only sets new defaults for the case where
    you don't specify these flags.
  - Set default value of --incompatible_remap_main_repo to true.
  - Set default value of --incompatible_remap_main_repo to true.

This release contains contributions from many people at Google, as well as Andrew Suffield, Brandon Lico, Chris Eason, Clint Harrison, Ed Schouten, Garrett Hopper, George Gensure, Greg, John Millikin, Julie, Keith Smiley, Laurent Le Brun, Ryan Beasley, Shmuel H, Travis Cline, Vladimir Chebotarev.

@bazel-io bazel-io closed this in 5f5e6ad Mar 30, 2019

emusand added a commit to emusand/bazel that referenced this issue Apr 16, 2019

Add incompatible flag to make PY2 outputs suffixed
This adds --incompatible_py2_outputs_are_suffixed, which makes it so Python 2 targets get built under an output root with the "-py2" suffix, and Python 3 targets get built under a root with no additional suffix. This means that the `bazel-bin` symlink contains Python 3 targets instead of Python 2 targets.

This goes hand-in-hand with --incompatible_py3_is_default, which controls whether targets get built in the Python 2 or Python 3 configuration by default. Enabling just one flag or the other will cause the majority of targets to get built under an output root other than the one pointed to by `bazel-bin`, which is not very convenient.

In general users and scripts are not supposed to care what the exact output path of a target is, but in practice they do, hence why this is an incompatible change.

Work toward bazelbuild#6647. See bazelbuild#7593 for migration info.

RELNOTES[INC]: Added --incompatible_py2_outputs_are_suffixed, for switching the bazel-bin symlink to point to Python 3 outputs instead of Python 2 outputs. See [bazelbuild#7593](bazelbuild#7593).

PiperOrigin-RevId: 236207234

emusand added a commit to emusand/bazel that referenced this issue Apr 16, 2019

Release 0.24.0 (2019-03-26)
Baseline: 235e76b

Cherry picks:

   + badd82e:
     Automated rollback of commit
     1b4c37c.
   + 33e5719:
     Fix the Python version select() mechanism to handle
     PY3-as-default
   + 56366ee:
     Set non-empty values for msvc_env_* when VC not installed
   + 22b3fbf:
     Windows, test wrapper: fix broken integration test
   + f14d447:
     Add whitelist file for starlark transitions
   + d99bc47:
     Update BUILD
   + 3529ad7:
     Rename tools/function_transition_whitelist/BUILD to
     tools/whitelists/function_transition_whitelist/BUILD
   + de0612a:
     Update bazel_toolchains to latest release and add toolchain
     config target for BuildKite CI (rbe_ubuntu1604)
   + 3e660ad:
     Automated rollback of commit
     0877340.
   + 314cf1f:
     Pass -undefined dynamic_lookup to dynamic library linking
     actions on Mac
   + fc586a8:
     Move cc_flags target into @bazel_tools//tools/cpp.
   + ea1703b:
     C++: Fix crash reported in bazelbuild#7721
   + 803801d:
     Pass execution info to xml generating spawn. Fixes bazelbuild#7794

Incompatible changes:

  - Added --incompatible_py3_is_default to test switching the default
    Python version to PY3 for py_binary/py_test targets that do not
    specify a version. See bazelbuild#7359.
  - //tools/cmd_line_differ has been renamed to //tools/aquery_differ
    & can now compare (in addition to command lines) inputs of
    actions given 2 aquery results.
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - The flag --incompatible_list_based_execution_strategy_selection
    was added and is used to ease the migration to the new style of
    specifying
    execution strategy selection and fallback behavior. The
    documentation for
    this flag is here: bazelbuild#7480
  - Added --incompatible_py2_outputs_are_suffixed, for switching the
    bazel-bin symlink to point to Python 3 outputs instead of Python
    2 outputs. See
    [bazelbuild#7593](bazelbuild#7593).

New features:

  - Make actions.args() object chainable.
  - Added --incompatible_windows_style_arg_escaping flag: enables
    correct subprocess argument escaping on Windows. (No-op on other
    platforms.)
  - Added --incompatible_windows_escape_jvm_flags flag: enables
    correct java_binary.jvm_flags and java_test.jvm_flags
    tokenization and escaping on Windows. (No-op on other platforms.)

Important changes:

  - Allow running aquery against the current state of Skyframe
  - Added support for the "navigation" resource directory to Android
    resource processing in Bazel. This is used by the Navigation
    Architecture Component.
  - --incompatible_disable_runtimes_filegroups was flipped
    (bazelbuild#6942)
  - Incompatible flag `--incompatible_linkopts_in_user_link_flags`
    has been flipped (bazelbuild#6826)
  - Incompatible flag `--incompatible_dont_emit_static_libgcc` has
    been flipped (bazelbuild#6825)
  - --incompatible_disallow_filetype is enabled by default.
  - Fixed issue where exceptions and stacktraces were silently
    swallowed in the Android resource processing pipeline.
  - `--incompatible_disable_expand_if_all_available_in_flag_set` has
    been flipped (bazelbuild#7008)
  - --incompatible_disallow_dict_plus is enabled by default
  - Adds --incompatible_disable_objc_library_resources to disable
    resource attributes in objc_library. Please migrate these
    attributes to `data` instead.
  - --incompatible_disallow_old_style_args_add is enabled by default.
  - Using the `native` module in BUILD files is deprecated. It will
    be forbidden with --incompatible_disallow_native_in_build_file.
  - (Python rules) PyRuntimeInfo is exposed to Starlark, making it
    possible for Starlark rules to depend on or imitate `py_runtime`.
    The `files` attribute of `py_runtime` is no longer mandatory.
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured
    targetshttps://github.com/bazelbuild/issues/7186.
  - is_using_fission crosstool variable is now exposed in all compile
    actions when fission is active (it used to be exposed only for
    linking actions).
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured
    targetshttps://github.com/bazelbuild/issues/7186.
  - `py_runtime` gains a `python_version` attribute for specifying
    whether it represents a Python 2 or 3 interpreter.
  - `--incompatible_java_coverage` is enabled by default.
  - Starlark rules can safely declare attributes named "licenses"
  - When using
    --incompatible_list_based_execution_strategy_selection, Bazel
    will use remote execution by default (if you specify
    --remote_executor), otherwise persistent workers (if the action
    supports it), otherwise sandboxed local execution (if the action
    and platform supports it) and at last unsandboxed local
    execution. The flags --spawn_strategy and --strategy continue to
    work as before - this only sets new defaults for the case where
    you don't specify these flags.
  - Set default value of --incompatible_remap_main_repo to true.
  - Set default value of --incompatible_remap_main_repo to true.

This release contains contributions from many people at Google, as well as Andrew Suffield, Brandon Lico, Chris Eason, Clint Harrison, Ed Schouten, Garrett Hopper, George Gensure, Greg, John Millikin, Julie, Keith Smiley, Laurent Le Brun, Ryan Beasley, Shmuel H, Travis Cline, Vladimir Chebotarev.

emusand added a commit to emusand/bazel that referenced this issue Apr 16, 2019

Flip default Python version to PY3
This flips --incompatible_py3_is_default and --incompatible_py2_outputs_are_suffixed. We opt out of this change for our own analysis and shell integration tests to avoid migrating them at the moment.

Fixes bazelbuild#7359, fixes bazelbuild#7593, fixes bazelbuild#6647.

RELNOTES[INC]: Python 3 is now the default Python version (for `py_binary` and `py_test` targets that don't specify the `python_version` attribute). Targets that are built for Python 3 will no longer have their output put in a separate `-py3` directory; instead there is now a separate `-py2` directory for Python 2 targets. See bazelbuild#7359 and bazelbuild#7593 for more information.

PiperOrigin-RevId: 241142275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.