New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatible_range_type: Make range() return a non-list iterable #5264

Closed
brandjon opened this Issue May 25, 2018 · 5 comments

Comments

Projects
None yet
6 participants
@brandjon
Member

brandjon commented May 25, 2018

range() currently returns a list of integers. This takes memory proportional to the size of the range, even though it can easily be represented using just its bounds and step. It's especially bad when used for iteration in a loop that breaks early. We should make range() return a list-like object as in Python 3.

If we take the Python 3 approach, range() would return a completely separate type of sequence from lists and tuples, with observably different behavior on the operations str(), repr(), type(), ==, ordering, +, and slicing.

@c-parsons

This comment has been minimized.

Contributor

c-parsons commented Aug 8, 2018

The issue should remain open until we migrate users onto the flag --incompatible_range_type

@laurentlb

This comment has been minimized.

Contributor

laurentlb commented Aug 20, 2018

Before switching this flag, please make sure we're happy with the exact semantics (str, repr, operators, etc.).

@laurentlb

This comment has been minimized.

Contributor

laurentlb commented Sep 21, 2018

I've tested the implementation. It seems to behave the same as Python3 and https://github.com/google/skylark/.

r = range(7)
print(r)  # range(7)
print(r[2:])  # range(2, 7)
print(r[1:9:2])  # range(1, 7, 2)
print(r[1:5:2])  # range(1, 5, 2)
print(repr(r[1:5:2]))  # range(1, 5, 2)
print(type(r))  # range
# print(r + [6])  # type error (like Python3, unlike Python2)
# print(r * 2)  # type error (like Python3, unlike Python2)

Bug: We should throw an error when comparing range objects:

r < r

Something interesting:

print(range(0, 5, 10) == range(0, 5, 11))

This returns False with the Go implementation; True with both Python3 and Bazel (I suppose it iterates over both ranges?).

Anything else we should test or consider?

@laurentlb

This comment has been minimized.

Contributor

laurentlb commented Oct 24, 2018

cc @ttsugriy

I plan to switch the default of incompatible_range_type for Bazel 0.20 (November), unless there's any concern.

@ttsugriy

This comment has been minimized.

Contributor

ttsugriy commented Oct 24, 2018

@laurentlb, I would be delighted! We've been using it in Buck since it was added to Bazel and haven't had any issues so far :) I hope Bazel users would enjoy it as much as Buck users :)

bazel-io pushed a commit that referenced this issue Oct 26, 2018

Make `range` objects not comparable
This is an incompatible change in the implementation, but it's already guarded by a flag.
Also add a few more tests.

#5264

RELNOTES: None.
PiperOrigin-RevId: 218911186

bazel-io pushed a commit that referenced this issue Oct 26, 2018

Switch the default value of --incompatible_range_type
#5264

RELNOTES: `range()` function now returns a lazy value (`--incompatible_range_type` is now set by default).
PiperOrigin-RevId: 218920956

@dslomov dslomov changed the title from Make range() return a non-list iterable to incompatible_range_type: Make range() return a non-list iterable Oct 31, 2018

@dslomov dslomov added this to the breaking-change-0.20 milestone Oct 31, 2018

@dslomov dslomov removed this from the breaking-change-0.20 milestone Oct 31, 2018

@laurentlb laurentlb closed this Oct 31, 2018

@laurentlb laurentlb assigned laurentlb and unassigned brandjon Oct 31, 2018

bazel-io pushed a commit that referenced this issue Nov 30, 2018

Release 0.20.0 (2018-11-30)
Baseline: 7bf7f03

Cherry picks:

   + fd52341:
     update bazel-toolchains pin to latest release Part of changes to
     allow bazelci to use 0.19.0 configs. RBE toolchain configs at or
     before 0.17.0 are not compatible with bazel 0.19.0 or above.
   + 241f28d:
     Revert "Toggle --incompatible_disable_late_bound_option_defaults
     flag."
   + f7e5aef:
     Add cc_toolchain targets for the new entries in the default
     cc_toolchain_suite.
   + d2920e3:
     Revert "WindowsFileSystem: open files with delete-sharing"

[Breaking changes in 0.20](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+label%3Abreaking-change-0.20)

  - [--incompatible_remove_native_http_archive](#6570).
  - [--incompatible_remove_native_git_repository](#6569).
  - [--incompatible_disable_cc_toolchain_label_from_crosstool_proto](#6434).
  - [--incompatible_disable_depset_in_cc_user_flags](#6384).
  - [--incompatible_disable_cc_configuration_make_variables](#6381).
  - [--incompatible_disallow_conflicting_providers](#5902).
  - [--incompatible_range_type](#5264).

[0.20 is a migration window for the following changes](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+label%3Amigration-0.20)

  - [--incompatible_use_jdk10_as_host_javabase](#6661)
  - [--incompatible_use_remotejdk_as_host_javabase](#6656)
  - [--incompatible_disable_sysroot_from_configuration](#6565)
  - [--incompatible_provide_cc_toolchain_info_from_cc_toolchain_suite](#6537)
  - [--incompatible_disable_depset_in_cc_user_flags](#6383)
  - [--incompatible_package_name_is_a_function](#5827)

[Breaking changes in the next release (0.21)](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+label%3Abreaking-change-0.21)

  - [--incompatible_use_jdk10_as_host_javabase](#6661)
  - [--incompatible_use_remotejdk_as_host_javabase](#6656)
  - [--incompatible_disable_sysroot_from_configuration](#6565)
  - [--incompatible_provide_cc_toolchain_info_from_cc_toolchain_suite](#6537)
  - [--incompatible_disable_depset_in_cc_user_flags](#6383)
  - [--incompatible_disallow_data_transition](#6153)
  - [--incompatible_package_name_is_a_function](#5827)
  - [--incompatible_disallow_slash_operator](#5823)
  - [--incompatible_static_name_resolution](#5637)

Incompatible changes:

  - the --experimental_no_dotd_scanning_with_modules command line
    argument is not supported anymore.
  - The --prune_cpp_modules command line option is not supported
    anymore.
  - the --experimental_prune_cpp_input_discovery command line option
    is not supported anymore.

New features:

  - Added support for Android NDK r18.

Important changes:

  - The 'default' parameter of attr.output and attr.output_list is
    removed. This is controlled by
    --incompatible_no_output_attr_default
  - A number of platform-related Starlark APIs which were previously
    marked "experimental" are now disabled by default, and may be
    enabled via --experimental_platforms_api
  - Make legacy-test-support ("legacy_test-<api-level>") from
    android_sdk_repository neverlink. The legacy test support
    libraries shouldn't be built into test binaries. To make them
    available at runtime, developers should declare them via
    uses-library:
    https://developer.android.com/training/testing/set-up-project#andr
    oid-test-base
  - query remote server Capabilities (per REAPI v2)
  - CppRules: All cc_toolchains depended on from
    cc_toolchain_suite.toolchains are now analyzed when not using
    platforms in order to select the right cc_toolchain.
  - removed obsolete --explicit_jre_deps flag.
  - Incompatible flag
    --incompatible_disable_legacy_cpp_toolchain_skylark_api was
    flipped.
  - Improve error messaging when unsupport proguard options are
    specified at the library level.
  - Incompatible flag
    --incompatible_disable_legacy_cpp_toolchain_skylark_api was
    flipped.
  - Incompatible flag
    --incompatible_disable_legacy_cpp_toolchain_skylark_api was
    flipped.
  - The --incompatible_disable_late_bound_option_defaults flag has
    been flipped (#6384)
  - Incompatible flag
    --incompatible_disable_legacy_flags_cc_toolchain_api was flipped
    (#6434)
  - Fixed issue where ctx.resolve_command created conflicting
    intermediate files when resolve_command was called multiple times
    within the same rule invocation with a long command attribute.
  - Incompatible flag
    --incompatible_disable_cc_configuration_make_variables was
    flipped (#6381)
  - If the --javabase flag is unset, it Bazel locates a JDK using
    the JAVA_HOME environment variable and searching the PATH. If no
    JDK is found --javabase will be empty, and builds targeting Java
    will not
    be supported. Previously Bazel would fall back to using the
    embedded
    JDK as a --javabase, but this is no longer default behaviour. A
    JDK should
    be explicitly installed instead to enable Java development
  - Bazel will now shut down when idle for 5 minutes and the system
    is low on RAM (linux only).
  - CROSSTOOL file is now read from the package of cc_toolchain, not
    from
    the package of cc_toolchain_suite. This is not expected to break
    anybody since
    cc_toolchain_suite and cc_toolchain are commonly in the same
    package.
  - All overrides of Starlark's ctx.new_file function are now
    deprecated.
      Try the `--incompatible_new_actions_api` flag to ensure your
    code is forward-compatible.
  - --incompatible_disable_cc_toolchain_label_from_crosstool_proto
    was flipped.
  - Introduce --(no)shutdown_on_low_sys_mem startup flag to toggle
    idle low-memory shutdown, disabled by default.
  - --incompatible_disable_cc_toolchain_label_from_crosstool_proto
    was flipped.
  - --incompatible_disable_cc_toolchain_label_from_crosstool_proto
    was flipped.
  - CppRules: All cc_toolchains depended on from
    cc_toolchain_suite.toolchains are now analyzed when not using
    platforms in order to select the right cc_toolchain.
  - The function `attr.license` is deprecated and will be removed.
      It can be disabled now with `--incompatible_no_attr_license`.
  - `range()` function now returns a lazy value
    (`--incompatible_range_type` is now set by default).
  - The code coverage report now includes the actual paths to header
    files instead of the ugly,
    Bazel generated, virtual includes path.
  - `--incompatible_disallow_conflicting_providers` has been switched
    to true
  - Add new flag `--incompatible_disable_systool_from_configration` to
    disable loading the systool from CppConfiguration.
  - Add new flag `--incompatible_disable_sysroot_from_configuration`
    to
    disable loading the systool from CppConfiguration.
  - Sorting remote Platform properties for remote execution. May
    affect cache keys!
  - Use different server log files per Bazel server process; java.log
    is
    now a symlink to the latest log.

This release contains contributions from many people at Google, as well as a7g4 <a7g4@a7g4.net>, Alan <alan.agius@betssongroup.com>, Asaf Flescher <asafflesch@gmail.com>, Benjamin Peterson <bp@benjamin.pe>, Ed Schouten <ed.schouten@prodrive-technologies.com>, George Gensure <ggensure@uber.com>, George Kalpakas <kalpakas.g@gmail.com>, Greg <gregestren@users.noreply.github.com>, Irina Iancu <iirina@users.noreply.github.com>, Keith Smiley <keithbsmiley@gmail.com>, Loo Rong Jie <loorongjie@gmail.com>, Mark Zeren <mzeren@vmware.com>, Petros Eskinder <petroseskinder@users.noreply.github.com>, rachcatch <rachelcatchpoole@hotmail.com>, Robert Brown <robert.brown@gmail.com>, Robert Gay <robert.gay@redfin.com>, Salty Egg <2281521+zhouhao@users.noreply.github.com>.

ahirreddy pushed a commit to ahirreddy/bazel that referenced this issue Dec 1, 2018

Release 0.20.0 (2018-11-30)
Baseline: 7bf7f03

Cherry picks:

   + fd52341:
     update bazel-toolchains pin to latest release Part of changes to
     allow bazelci to use 0.19.0 configs. RBE toolchain configs at or
     before 0.17.0 are not compatible with bazel 0.19.0 or above.
   + 241f28d:
     Revert "Toggle --incompatible_disable_late_bound_option_defaults
     flag."
   + f7e5aef:
     Add cc_toolchain targets for the new entries in the default
     cc_toolchain_suite.
   + d2920e3:
     Revert "WindowsFileSystem: open files with delete-sharing"

[Breaking changes in 0.20](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+label%3Abreaking-change-0.20)

  - [--incompatible_remove_native_http_archive](bazelbuild#6570).
  - [--incompatible_remove_native_git_repository](bazelbuild#6569).
  - [--incompatible_disable_cc_toolchain_label_from_crosstool_proto](bazelbuild#6434).
  - [--incompatible_disable_depset_in_cc_user_flags](bazelbuild#6384).
  - [--incompatible_disable_cc_configuration_make_variables](bazelbuild#6381).
  - [--incompatible_disallow_conflicting_providers](bazelbuild#5902).
  - [--incompatible_range_type](bazelbuild#5264).

[0.20 is a migration window for the following changes](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+label%3Amigration-0.20)

  - [--incompatible_use_jdk10_as_host_javabase](bazelbuild#6661)
  - [--incompatible_use_remotejdk_as_host_javabase](bazelbuild#6656)
  - [--incompatible_disable_sysroot_from_configuration](bazelbuild#6565)
  - [--incompatible_provide_cc_toolchain_info_from_cc_toolchain_suite](bazelbuild#6537)
  - [--incompatible_disable_depset_in_cc_user_flags](bazelbuild#6383)
  - [--incompatible_package_name_is_a_function](bazelbuild#5827)

[Breaking changes in the next release (0.21)](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+label%3Abreaking-change-0.21)

  - [--incompatible_use_jdk10_as_host_javabase](bazelbuild#6661)
  - [--incompatible_use_remotejdk_as_host_javabase](bazelbuild#6656)
  - [--incompatible_disable_sysroot_from_configuration](bazelbuild#6565)
  - [--incompatible_provide_cc_toolchain_info_from_cc_toolchain_suite](bazelbuild#6537)
  - [--incompatible_disable_depset_in_cc_user_flags](bazelbuild#6383)
  - [--incompatible_disallow_data_transition](bazelbuild#6153)
  - [--incompatible_package_name_is_a_function](bazelbuild#5827)
  - [--incompatible_disallow_slash_operator](bazelbuild#5823)
  - [--incompatible_static_name_resolution](bazelbuild#5637)

Incompatible changes:

  - the --experimental_no_dotd_scanning_with_modules command line
    argument is not supported anymore.
  - The --prune_cpp_modules command line option is not supported
    anymore.
  - the --experimental_prune_cpp_input_discovery command line option
    is not supported anymore.

New features:

  - Added support for Android NDK r18.

Important changes:

  - The 'default' parameter of attr.output and attr.output_list is
    removed. This is controlled by
    --incompatible_no_output_attr_default
  - A number of platform-related Starlark APIs which were previously
    marked "experimental" are now disabled by default, and may be
    enabled via --experimental_platforms_api
  - Make legacy-test-support ("legacy_test-<api-level>") from
    android_sdk_repository neverlink. The legacy test support
    libraries shouldn't be built into test binaries. To make them
    available at runtime, developers should declare them via
    uses-library:
    https://developer.android.com/training/testing/set-up-project#andr
    oid-test-base
  - query remote server Capabilities (per REAPI v2)
  - CppRules: All cc_toolchains depended on from
    cc_toolchain_suite.toolchains are now analyzed when not using
    platforms in order to select the right cc_toolchain.
  - removed obsolete --explicit_jre_deps flag.
  - Incompatible flag
    --incompatible_disable_legacy_cpp_toolchain_skylark_api was
    flipped.
  - Improve error messaging when unsupport proguard options are
    specified at the library level.
  - Incompatible flag
    --incompatible_disable_legacy_cpp_toolchain_skylark_api was
    flipped.
  - Incompatible flag
    --incompatible_disable_legacy_cpp_toolchain_skylark_api was
    flipped.
  - The --incompatible_disable_late_bound_option_defaults flag has
    been flipped (bazelbuild#6384)
  - Incompatible flag
    --incompatible_disable_legacy_flags_cc_toolchain_api was flipped
    (bazelbuild#6434)
  - Fixed issue where ctx.resolve_command created conflicting
    intermediate files when resolve_command was called multiple times
    within the same rule invocation with a long command attribute.
  - Incompatible flag
    --incompatible_disable_cc_configuration_make_variables was
    flipped (bazelbuild#6381)
  - If the --javabase flag is unset, it Bazel locates a JDK using
    the JAVA_HOME environment variable and searching the PATH. If no
    JDK is found --javabase will be empty, and builds targeting Java
    will not
    be supported. Previously Bazel would fall back to using the
    embedded
    JDK as a --javabase, but this is no longer default behaviour. A
    JDK should
    be explicitly installed instead to enable Java development
  - Bazel will now shut down when idle for 5 minutes and the system
    is low on RAM (linux only).
  - CROSSTOOL file is now read from the package of cc_toolchain, not
    from
    the package of cc_toolchain_suite. This is not expected to break
    anybody since
    cc_toolchain_suite and cc_toolchain are commonly in the same
    package.
  - All overrides of Starlark's ctx.new_file function are now
    deprecated.
      Try the `--incompatible_new_actions_api` flag to ensure your
    code is forward-compatible.
  - --incompatible_disable_cc_toolchain_label_from_crosstool_proto
    was flipped.
  - Introduce --(no)shutdown_on_low_sys_mem startup flag to toggle
    idle low-memory shutdown, disabled by default.
  - --incompatible_disable_cc_toolchain_label_from_crosstool_proto
    was flipped.
  - --incompatible_disable_cc_toolchain_label_from_crosstool_proto
    was flipped.
  - CppRules: All cc_toolchains depended on from
    cc_toolchain_suite.toolchains are now analyzed when not using
    platforms in order to select the right cc_toolchain.
  - The function `attr.license` is deprecated and will be removed.
      It can be disabled now with `--incompatible_no_attr_license`.
  - `range()` function now returns a lazy value
    (`--incompatible_range_type` is now set by default).
  - The code coverage report now includes the actual paths to header
    files instead of the ugly,
    Bazel generated, virtual includes path.
  - `--incompatible_disallow_conflicting_providers` has been switched
    to true
  - Add new flag `--incompatible_disable_systool_from_configration` to
    disable loading the systool from CppConfiguration.
  - Add new flag `--incompatible_disable_sysroot_from_configuration`
    to
    disable loading the systool from CppConfiguration.
  - Sorting remote Platform properties for remote execution. May
    affect cache keys!
  - Use different server log files per Bazel server process; java.log
    is
    now a symlink to the latest log.

This release contains contributions from many people at Google, as well as a7g4 <a7g4@a7g4.net>, Alan <alan.agius@betssongroup.com>, Asaf Flescher <asafflesch@gmail.com>, Benjamin Peterson <bp@benjamin.pe>, Ed Schouten <ed.schouten@prodrive-technologies.com>, George Gensure <ggensure@uber.com>, George Kalpakas <kalpakas.g@gmail.com>, Greg <gregestren@users.noreply.github.com>, Irina Iancu <iirina@users.noreply.github.com>, Keith Smiley <keithbsmiley@gmail.com>, Loo Rong Jie <loorongjie@gmail.com>, Mark Zeren <mzeren@vmware.com>, Petros Eskinder <petroseskinder@users.noreply.github.com>, rachcatch <rachelcatchpoole@hotmail.com>, Robert Brown <robert.brown@gmail.com>, Robert Gay <robert.gay@redfin.com>, Salty Egg <2281521+zhouhao@users.noreply.github.com>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment