Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatible_allow_python_version_transitions: Python version is no longer "sticky" #7307

Open
brandjon opened this Issue Jan 31, 2019 · 4 comments

Comments

Projects
None yet
5 participants
@brandjon
Copy link
Member

brandjon commented Jan 31, 2019

Flag: --incompatible_allow_python_version_transitions
Available since: 0.23
Will be flipped in: either 0.24 or 0.25, depending on migration progress
Tracking issue: #6583
Design doc
Corresponding syntactic change: #7308

Motivation

The Python mode behaved in a confusing and error-prone way, and didn't support important use cases like a Python 3 binary depending on a Python 2 one in its data attribute.

See the design doc for more details.

Note: This incompatible change affects the mechanism that Bazel uses to decide whether a target should be built for Python 2 or 3. However, due to #4815, even when a target is built for Python 3 it may still execute under a Python 2 interpreter. See the workaround using py_runtime / --python_top in #4815 (comment).

Change

This makes the Python version have only two possible states: PY2 and PY3. The version can change whenever an executable Python rule (py_binary or py_test) is encountered, even if the version was previously set by another target or by a top-level flag.

In addition, srcs_version validation is done at the level of the executable target, rather than in each py_library. This makes it so you can do bazel build //... without getting errors from PY2-only or PY3-only libraries.

This change does not remove any syntax. See --incompatible_remove_old_python_version_api for that.

Note: This change does not affect targets built in the host configuration, such as targets in genrule's tools attribute, or user-defined rules that set cfg = host on an attribute. This is a consequence of technical limitations on the host configuration to be addressed in future work. In the meantime, you can set a Python version to use globally for all host-configured targets by setting --host_force_python=[PY2 | PY3].

Migration

If you were relying on --force_python to control your Python mode, that won't work anymore since py_binary will set its own mode. Use the python_version attribute (formerly default_python_version) on py_binary instead.

Using multiple Python versions in the same build can lead to action conflicts if you test the python version by select()-ing on "force_python". To avoid this, you should instead select on @bazel_tools//tools/python:python_version. See here. You can ensure that your build doesn't select on "force_python" by enabling --incompatible_remove_old_python_version_api.

Another type of action conflict can arise due to #7655 when a cc_library is data-depended on by a py_binary. This kind of conflict can be avoided by enabling --incompatible_remove_old_python_version_api. If you're not able to migrate your build to enable that flag yet, you can also avoid this particular conflict by explicitly passing a value for the --force_python flag that matches the value passed for --python_version (or the default value if --python_version is not passed).

If there's a srcs_version conflict in your build, you will no longer see an error at the py_library that introduces the conflict. Instead you'll see it in the py_binary or py_test that transitively depends on the library. The error message will not identify which library introduced the version constraint. To find it, you can run the aspect @bazel_tools//tools/python:srcs_version.bzl%find_requirements as follows:

bazel build <your target> \
    --aspects=@bazel_tools//tools/python:srcs_version.bzl%find_requirements \
    --output_groups=pyversioninfo

This will build a [...]-pyversioninfo.txt file explaining what dependency requires what Python version. You can run it directly on the py_binary or py_test that failed due to a version conflict.

@brandjon brandjon self-assigned this Jan 31, 2019

bazel-io pushed a commit that referenced this issue Feb 1, 2019

Add Py2/3 mode migration flags to --all_incompatible_changes
This makes available --incompatible_allow_python_version_transitions and --incompatible_remove_old_python_version_api, which were previously named "--experimental_...".

See #7307 and #7308 respectively to track the migration for each flag.

Work toward #6583, #7307, and #7308.

RELNOTES[INC]: Two changes to native Python rules: 1) `default_python_version` and `--force_python` are deprecated; use `python_version` and `--python_version` respectively instead. You can preview the removal of the deprecated names with --incompatible_remove_old_python_version_api. See [#7308](#7308). 2) The version flag will no longer override the declared version of a `py_binary` or `py_test` target. You can preview this new behavior with --incompatible_allow_python_version_transitions. See [#7307](#7307).

PiperOrigin-RevId: 231980615

bazel-io pushed a commit that referenced this issue Feb 26, 2019

Release 0.23.0 (2019-02-26)
Baseline: 441fd75

Cherry picks:

   + 6ca7763:
     Fix a typo
   + 2310b1c:
     Ignore SIGCHLD in test setup script
   + f9eb1b5:
     Complete channel initialization in the event loop

Incompatible changes:

  - //src:bazel uses the minimal embedded JDK, if you want to
    avoid the extra steps of minimizing the JDK, use //src:bazel-dev
    instead.
  - //src:bazel uses the minimal embedded JDK, if you want to
    avoid the extra steps of building bazel with the minimized JDK,
    use //src:bazel-dev
    instead.
  - The default value of --host_platform and --platforms will be
      changed to not be dependent on the configuration. This means
    that setting
      --cpu or --host_cpu will not affect the target or host platform.
  - Toolchain resolution for cc rules is now enabled via an
    incompatible flag, --incompatible_enable_cc_toolchain_resolution.
    The previous
    flag, --enabled_toolchain_types, is deprecated and will be
    removed.
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - Python rules will soon reject the legacy "py" struct provider
    (preview by enabling --incompatible_disallow_legacy_py_provider).
    Upgrade rules to use PyInfo instead. See
    [#7298](#7298).
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - Two changes to native Python rules: 1) `default_python_version`
    and `--force_python` are deprecated; use `python_version` and
    `--python_version` respectively instead. You can preview the
    removal of the deprecated names with
    --incompatible_remove_old_python_version_api. See
    [#7308](#7308). 2) The
    version flag will no longer override the declared version of a
    `py_binary` or `py_test` target. You can preview this new
    behavior with --incompatible_allow_python_version_transitions.
    See [#7307](#7307).

Important changes:

  - There is a new flag available
    `--experimental_java_common_create_provider_enabled_packages`
    that acts as a whitelist for usages of
    `java_common.create_provider`. The constructor will be deprecated
    in Bazel 0.23.
  - [#7024] Allow chaining of the same function type in aquery.
  - Introduces --local_{ram,cpu}_resources, which will take the place
    of --local_resources.
  - [#6930] Add documentation for the aquery command.
  - Incompatible flag `--incompatible_dont_emit_static_libgcc` has
    been flipped (#6825)
  - Incompatible flag `--incompatible_linkopts_in_user_link_flags`
    has been flipped (#6826)
  - Flag --incompatible_range_type is removed.
  - Flag --incompatible_disallow_slash_operator is removed.
  - Flag --incompatible_disallow_conflicting_providers is removed.
  - `--incompatible_disallow_data_transition` is now enabled by
    default
  - Allow inclusion of param files in aquery output
  - [#6985] Add test to verify aquery's behavior for Cpp action
    templates.
  - --incompatible_require_feature_configuration_for_pic was flipped
    (#7007).
  - Also ignore module-info.class in multi-version Jars
  - objc_framework has been deleted. Please refer to
    apple_dynamic_framework_import and apple_static_framework_import
    rules available in
    [rules_apple](https://github.com/bazelbuild/rules_apple/blob/maste
    r/doc/rules-general.md)
  - --test_sharding_strategy=experimental_heuristic is no more
  - objc_bundle_library has been removed. Please migrate to
    rules_apple's
    [apple_resource_bundle](https://github.com/bazelbuild/rules_apple/
    blob/master/doc/rules-resources.md#apple_resource_bundle).
  - You can now use the attribute `aapt_version` or the flag
    `--android_aapt` to pick the aapt version for android_local_test
    tests
  - In --keep_going mode, Bazel now correctly returns a non-zero exit
    code when encountering a package loading error during target
    pattern parsing of patterns like "//foo:all" and "//foo/...".
  - The default value for --incompatible_strict_action_env has been
    flipped to 'false' again, as we discovered breakages for local
    execution users. We'll need some more time to figure out the best
    way to make this work for local and remote execution. Follow
    #7026 for more details.
  - Locally-executed spawns tagged "no-cache" no longer upload their
    outputs to the remote cache.
  - Introduces --host_compiler flag to allow setting a compiler for
    host compilation when --host_crosstool_top is specified.
  - --incompatible_expand_directories is enabled by default
  - [aquery] Handle the case of aspect-on-aspect.
  - Fixed a longstanding bug in the http remote cache where the value
    passed to
    --remote_timeout would be interpreted as milliseconds instead of
    seconds.
  - Enable --incompatible_use_jdk10_as_host_javabase by default, see
    #6661
  - Add --incompatible_use_jdk11_as_host_javabase: makes JDK 11 the
    default --host_javabase for remote jdk
    (#7219)
  - Highlight TreeArtifact in aquery text output.
  - Locally-executed spawns tagged "no-cache" no longer upload their
    outputs to the remote cache.
  - java_common APIs now accept JavaToolchainInfo and JavaRuntimeInfo
    instead of configured targets for java_toolchain and java_runtime
  - cc_common.create_cc_toolchain_config_info is stable and available
    for production use
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured targets
    (#7186)
  - --incompatible_strict_argument_ordering is enabled by default.
  - Bazel now supports reading cache hits from a repository cache,
    even if it doesn't have write access to the cache.
  - Adding arm64e to OSX CROSSTOOL.
  - Ignore package-level licenses on config_setting.
  - Add an optional output_source_jar parameter to java_common.compile
  - --incompatible_disable_objc_provider_resources is now enabled by
    default. This disables ObjcProvider's fields related to resource
    processing.
  - Explicitly set https.protocols and exclude TLSv1.3.
  - Bazel now validates that JAVA_HOME points to a valid JDK and
    falls back to auto-detection by looking up the path of `javac`.
  - Upgrade the embedded JDK version to 11.0.2.
  - Added --incompatible_disable_crosstool_file
    (#7320)
  - --incompatible_disable_objc_provider_resources is now enabled by
    default. This disables ObjcProvider's fields related to resource
    processing.
  - --incompatible_disable_tools_defaults_package has been flipped.
  - For tests that do not generate a test.xml, Bazel now uses a
    separate action to generate one; this results in minor
    differences in the generated test.xml, and makes the generation
    more reliable overall.
  - incompatible_generate_javacommon_source_jar: java_common.compile
    now always generates a source jar, see
    #5824.
  - New incompatible flag
    --incompatible_disallow_struct_provider_syntax removes the
    ability for rule implementation functions to return struct. Such
    functions should return a list of providers instead. Migration
    tracking: #7347

This release contains contributions from many people at Google, as well as Benjamin Peterson, Ed Schouten, erenon, George Gensure, Greg Estren, Igal Tabachnik, Ittai Zeidman, Jannis Andrija Schnitzer, John Millikin, Keith Smiley, Kelly Campbell, Max Vorobev, nicolov, Robin Nabel.

excitoon pushed a commit to KasperskyLab/bazel that referenced this issue Feb 27, 2019

Release 0.23.0 (2019-02-26)
Baseline: 441fd75

Cherry picks:

   + 6ca7763:
     Fix a typo
   + 2310b1c:
     Ignore SIGCHLD in test setup script
   + f9eb1b5:
     Complete channel initialization in the event loop

Incompatible changes:

  - //src:bazel uses the minimal embedded JDK, if you want to
    avoid the extra steps of minimizing the JDK, use //src:bazel-dev
    instead.
  - //src:bazel uses the minimal embedded JDK, if you want to
    avoid the extra steps of building bazel with the minimized JDK,
    use //src:bazel-dev
    instead.
  - The default value of --host_platform and --platforms will be
      changed to not be dependent on the configuration. This means
    that setting
      --cpu or --host_cpu will not affect the target or host platform.
  - Toolchain resolution for cc rules is now enabled via an
    incompatible flag, --incompatible_enable_cc_toolchain_resolution.
    The previous
    flag, --enabled_toolchain_types, is deprecated and will be
    removed.
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - Python rules will soon reject the legacy "py" struct provider
    (preview by enabling --incompatible_disallow_legacy_py_provider).
    Upgrade rules to use PyInfo instead. See
    [bazelbuild#7298](bazelbuild#7298).
  - java_(mutable_|)proto_library: removed strict_deps attribute.
  - Two changes to native Python rules: 1) `default_python_version`
    and `--force_python` are deprecated; use `python_version` and
    `--python_version` respectively instead. You can preview the
    removal of the deprecated names with
    --incompatible_remove_old_python_version_api. See
    [bazelbuild#7308](bazelbuild#7308). 2) The
    version flag will no longer override the declared version of a
    `py_binary` or `py_test` target. You can preview this new
    behavior with --incompatible_allow_python_version_transitions.
    See [bazelbuild#7307](bazelbuild#7307).

Important changes:

  - There is a new flag available
    `--experimental_java_common_create_provider_enabled_packages`
    that acts as a whitelist for usages of
    `java_common.create_provider`. The constructor will be deprecated
    in Bazel 0.23.
  - [bazelbuild#7024] Allow chaining of the same function type in aquery.
  - Introduces --local_{ram,cpu}_resources, which will take the place
    of --local_resources.
  - [bazelbuild#6930] Add documentation for the aquery command.
  - Incompatible flag `--incompatible_dont_emit_static_libgcc` has
    been flipped (bazelbuild#6825)
  - Incompatible flag `--incompatible_linkopts_in_user_link_flags`
    has been flipped (bazelbuild#6826)
  - Flag --incompatible_range_type is removed.
  - Flag --incompatible_disallow_slash_operator is removed.
  - Flag --incompatible_disallow_conflicting_providers is removed.
  - `--incompatible_disallow_data_transition` is now enabled by
    default
  - Allow inclusion of param files in aquery output
  - [bazelbuild#6985] Add test to verify aquery's behavior for Cpp action
    templates.
  - --incompatible_require_feature_configuration_for_pic was flipped
    (bazelbuild#7007).
  - Also ignore module-info.class in multi-version Jars
  - objc_framework has been deleted. Please refer to
    apple_dynamic_framework_import and apple_static_framework_import
    rules available in
    [rules_apple](https://github.com/bazelbuild/rules_apple/blob/maste
    r/doc/rules-general.md)
  - --test_sharding_strategy=experimental_heuristic is no more
  - objc_bundle_library has been removed. Please migrate to
    rules_apple's
    [apple_resource_bundle](https://github.com/bazelbuild/rules_apple/
    blob/master/doc/rules-resources.md#apple_resource_bundle).
  - You can now use the attribute `aapt_version` or the flag
    `--android_aapt` to pick the aapt version for android_local_test
    tests
  - In --keep_going mode, Bazel now correctly returns a non-zero exit
    code when encountering a package loading error during target
    pattern parsing of patterns like "//foo:all" and "//foo/...".
  - The default value for --incompatible_strict_action_env has been
    flipped to 'false' again, as we discovered breakages for local
    execution users. We'll need some more time to figure out the best
    way to make this work for local and remote execution. Follow
    bazelbuild#7026 for more details.
  - Locally-executed spawns tagged "no-cache" no longer upload their
    outputs to the remote cache.
  - Introduces --host_compiler flag to allow setting a compiler for
    host compilation when --host_crosstool_top is specified.
  - --incompatible_expand_directories is enabled by default
  - [aquery] Handle the case of aspect-on-aspect.
  - Fixed a longstanding bug in the http remote cache where the value
    passed to
    --remote_timeout would be interpreted as milliseconds instead of
    seconds.
  - Enable --incompatible_use_jdk10_as_host_javabase by default, see
    bazelbuild#6661
  - Add --incompatible_use_jdk11_as_host_javabase: makes JDK 11 the
    default --host_javabase for remote jdk
    (bazelbuild#7219)
  - Highlight TreeArtifact in aquery text output.
  - Locally-executed spawns tagged "no-cache" no longer upload their
    outputs to the remote cache.
  - java_common APIs now accept JavaToolchainInfo and JavaRuntimeInfo
    instead of configured targets for java_toolchain and java_runtime
  - cc_common.create_cc_toolchain_config_info is stable and available
    for production use
  - incompatible_use_toolchain_providers_in_java_common: pass
    JavaToolchainInfo and JavaRuntimeInfo providers to java_common
    APIs instead of configured targets
    (bazelbuild#7186)
  - --incompatible_strict_argument_ordering is enabled by default.
  - Bazel now supports reading cache hits from a repository cache,
    even if it doesn't have write access to the cache.
  - Adding arm64e to OSX CROSSTOOL.
  - Ignore package-level licenses on config_setting.
  - Add an optional output_source_jar parameter to java_common.compile
  - --incompatible_disable_objc_provider_resources is now enabled by
    default. This disables ObjcProvider's fields related to resource
    processing.
  - Explicitly set https.protocols and exclude TLSv1.3.
  - Bazel now validates that JAVA_HOME points to a valid JDK and
    falls back to auto-detection by looking up the path of `javac`.
  - Upgrade the embedded JDK version to 11.0.2.
  - Added --incompatible_disable_crosstool_file
    (bazelbuild#7320)
  - --incompatible_disable_objc_provider_resources is now enabled by
    default. This disables ObjcProvider's fields related to resource
    processing.
  - --incompatible_disable_tools_defaults_package has been flipped.
  - For tests that do not generate a test.xml, Bazel now uses a
    separate action to generate one; this results in minor
    differences in the generated test.xml, and makes the generation
    more reliable overall.
  - incompatible_generate_javacommon_source_jar: java_common.compile
    now always generates a source jar, see
    bazelbuild#5824.
  - New incompatible flag
    --incompatible_disallow_struct_provider_syntax removes the
    ability for rule implementation functions to return struct. Such
    functions should return a list of providers instead. Migration
    tracking: bazelbuild#7347

This release contains contributions from many people at Google, as well as Benjamin Peterson, Ed Schouten, erenon, George Gensure, Greg Estren, Igal Tabachnik, Ittai Zeidman, Jannis Andrija Schnitzer, John Millikin, Keith Smiley, Kelly Campbell, Max Vorobev, nicolov, Robin Nabel.
@katre

This comment has been minimized.

Copy link
Member

katre commented Mar 1, 2019

This issue was tagged as "breaking-change-0.24" but does not appear ready to be flipped in the 0.24.0 release. If this is incorrect please comment on that issue and discuss with me.

@brandjon

This comment has been minimized.

Copy link
Member Author

brandjon commented Mar 1, 2019

Now targeting 0.25.

bazel-io pushed a commit that referenced this issue Mar 4, 2019

Update tests to be compatible with --incompatible_allow_python_versio…
…n_transitions

Work toward #7307.

RELNOTES: None
PiperOrigin-RevId: 236718484
@brandjon

This comment has been minimized.

Copy link
Member Author

brandjon commented Mar 7, 2019

Note the updated paragraph in Migration instructions above to account for #7655.

@brandjon

This comment has been minimized.

Copy link
Member Author

brandjon commented Mar 12, 2019

According to a recent run of the bazelisk CI, the only downstream failures due to --incompatible_allow_python_version_transitions are due to action conflicts in C++ rules, which should go away when --incompatible_remove_old_python_version_api is enabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.