Skip to content
Permalink
Branch: master
Commits on Jul 20, 2019
  1. Refactor V2 PEX creation rules (#8080)

    Eric-Arellano committed Jul 20, 2019
    #8063 was originally going to result in creating a `SourcesPex` (in addition to `RequirementsPex`). In preparation for that, the below refactors were made. Even though the approach to #8063 has changed, these are good changes to make in general and may be useful to porting `./pants binary`.
    
    * Extract out `download_pex_bin` rule.
    * Rename `resolve_requirements.py` -> `create_requirements_pex.py` for more explicit name (and avoid name clash with V1 file).
    * Fix shadowing variable name.
    * Use long CLI args.
Commits on Jul 18, 2019
  1. Remove unit test runtime dependencies on resources (#8066)

    Eric-Arellano authored and stuhood committed Jul 18, 2019
    ### Problem
    Currently, `test_base.py`, `jvm_tool_task_test_base.py`, and `interpreter_cache_test_mixin.py` have implicit runtime dependencies on files defined at the build root, specifically `./pants`, `./BUILD.tools`, `3rdparty/BUILD`, and `.pants.d`. 
    
    Because V2 is completely hermetic, these runtime dependencies fail to resolve as the test runner does not know to copy the files into the snapshot. Instead, we need some way to explicitly mark these dependencies via the `BUILD` file or to remove the dependency outright.
    
    #### FYI: why didn't we detect this earlier?
    We've been using V2 to run unit tests in CI for the past two months. Why didn't we detect this? The reason is that V2 runs locally in `.pantsd`, and the logic to get the buildroot traverses ancestors.
    
    This was only detected when running unit tests with remote execution.
    
    ### Solution
    * Remove the dependency on `./pants`, which was not actually needed. 
    * Explicitly depend on `./BUILD.tools` and `3rdparty/BUILD`. 
    * Remove `interpreter_cache_test_mixin.py`, which is inherently non-hermetic and cannot work with the V2 model. Even though this has a slight performance impact, this is offset by the remote caching we will now be able to get + having more sound / hermitic tests.
  2. Use our docker image from #8059 for remote execution (#8068)

    Eric-Arellano committed Jul 18, 2019
    This actually fixes #8057 by using the image created in #8059.
Commits on Jul 17, 2019
  1. Extend Google RBE Docker image to allow building sdist wheels (#8059)

    Eric-Arellano authored and stuhood committed Jul 17, 2019
    Google's Ubuntu image does not have the dependencies necessary to install `psutil`, `scandir`, `pyopenssl`, `setproctitle`, and `python-Levenshtein`, as documented in #8057.
    
    Turns out, all that's necessary to fix this is installing `build-essential`. The fix was confirmed locally by building this image then running `docker run -t remote_execution sh -c 'pip3 install pex; pex setproctitle scandir psutil python-Levenshtein'`.
    
    Will close #8057.
  2. Bump virtualenv to 16.6.2 for local run script (#8061)

    Eric-Arellano committed Jul 17, 2019
    Recently, when running Pants, the below was always printed to stderr.
    
    ```
    /Users/eric/DocsLocal/code/projects/pants/build-support/pants_dev_deps.py36.venv/lib/python3.6/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
      import imp
    ```
    
    This was fixed in virtualenv in [16.3.0](https://virtualenv.pypa.io/en/latest/changes/#id27) and [16.4.0](https://virtualenv.pypa.io/en/latest/changes/#id20)
    
    (Note that you must `rm -rf build-support/pants_dev_deps.py{36,37}.venv` for this change to be picked up locally.)
  3. Replace scandir dependency with stdlib version (#8058)

    Eric-Arellano committed Jul 17, 2019
    The backport wasn't being used now that we don't support Python 2 and it caused an unnecessary sdist to be installed.
Commits on Jul 16, 2019
  1. Set RUST_BACKTRACE in CI for more useful Rust debugging (#8054)

    Eric-Arellano committed Jul 16, 2019
    Because Pants interfaces with the Rust engine so much, Rust will occasionally fail on the non-Rust shards. Proactively setting `RUST_BACKTRACE` makes debugging easier when that happens.
Commits on Jul 4, 2019
  1. Set up pants.remote.ini for remoting Python unit tests (#7990)

    Eric-Arellano committed Jul 4, 2019
    Per #7649, we are working to remote unit tests in CI.
    
    This implements step #2, which allows us to run unit tests locally (several, but not all tests). Here, we set up `pants.remote.ini` so that users only must point to it and provide the authentication token.
  2. Don't use remote store when --no-remote-execution specified (#8010)

    Eric-Arellano committed Jul 4, 2019
    #7991 added the flag `--[no-]remote-exeuction` to toggle on and off remote behavior. There, out of error, it was only used to toggle the remote execution runner, not also the remote store.
    
    This is causing #7990 to continue to fail.
Commits on Jul 3, 2019
  1. Add type checking to several `util` files (#7998)

    Eric-Arellano committed Jul 3, 2019
    The util code—excluding the metaprogramming files—is a good place to start adding type hints to Pants because it is composed of simple pure functions and is used widely throughout the codebase. 
    
    This adds some initial hints to act as a demo for type hints in Pants and get us slightly closer to critical mass.
  2. Fix bad `with_traceback()` change from #7988 (#8004)

    Eric-Arellano committed Jul 3, 2019
    Python does not allow keyword args for several builtins, including `with_traceback`. This was not detected as an issue in #7988 because the test was already flaky, and this only causes an additional error when the test flakes.
  3. Add `--[no-]remote-execution` flag (#7991)

    Eric-Arellano committed Jul 3, 2019
    ### Problem
    We want to allow users to permanently configure all remoting options in their `pants.ini`, without that config triggering Pants always using remoting. Tangibly, we are trying to land #7990 to setup our own usage and can't get green CI because Pants is trying to remote things it shouldn't.
    
    Instead, users should be able to turn on remoting at the per-invocation level, just like we have `--enable-pantsd`. 
    
    ### Solution
    Introduce `--[no-]remote-execution`. For now, this defaults to `False` as this is an alpha feature and it does not work without other remoting config.
    
    ### Result
    Users can have all their remoting config setup and still be able to entirely run locally. Also now more explicit when you are and are not remoting.
Commits on Jul 2, 2019
  1. Use default utf-8 encoding for string `decode()` and `encode()` (#7983)

    Eric-Arellano committed Jul 2, 2019
    Python 3 defaults to using `utf-8` for these two methods: https://docs.python.org/3/library/stdtypes.html#str.encode.
    
    By using the default encoding, we result in less code and less cognitive overload, because the default works and shouldn't require thinking about it. We should only specify the encoding when it's not `utf-8`, like `ascii`.
    
    This change was generated via [this script](https://github.com/foursquare/fsqio/pull/60/files#diff-be67cc89f8af39cc469db25a6d95d105).
  2. Fix most Python 2 issues from #6071 (#7989)

    Eric-Arellano committed Jul 2, 2019
    #6071 was used to keep track of several changes we could only make after dropping Python 2. This PR makes all those changes, minus changing some APIs to return `NotImplemented` in their equality dunder methods.
  3. Remove `future` library (#7988)

    Eric-Arellano committed Jul 2, 2019
Commits on Jul 1, 2019
  1. Update style guide to use f-strings (#7987)

    Eric-Arellano committed Jul 1, 2019
    SKIP_FULL_CI  # Documentation-only change.
    
    We should now use f-strings everywhere possible.
    
    This also rewords some of the other sections to be more concise and clear.
  2. Remove remaining `if PY{2,3}` snippets (#7986)

    Eric-Arellano committed Jul 1, 2019
  3. Remove `six` library (#7985)

    Eric-Arellano committed Jul 1, 2019
    However, `contrib.python.checks.checker` does still need `six`, which was setup in #7982 to be isolated from this change.
  4. No longer use `string_types` (#7980)

    Eric-Arellano committed Jul 1, 2019
    `string_types`, which is the same as `Tuple[bytes, str]`, is a vestige of Python 2's loose mixing of unicode and byte strings. Python 3 intentionally makes you choose which you want to use, which we should be doing too for more sane code.
    
    While this technically violates the deprecation policy for some APIs, this coincides with the even larger breaking change of no longer supporting Python 2. It is almost guaranteed that users who extend Pants will already need to make changes, so it is helpful to land this API change at the same time that people are modernizing their `pants-plugins`. 
    
    Further, several of the below instances do not make sense to support bytes—they would crash were someone to pass bytes and it is solely happenstance that no one has reported trying. We had only kept the support for bytes due to Python 2's frequent use of byte strings.
Commits on Jun 30, 2019
  1. Use dedicated requirements.txt for `contrib.python.checks.checker` (#…

    Eric-Arellano committed Jun 30, 2019
    …7982)
    
    ### Problem
    `contrib.python.checks.checker` must keep support for Python 2, e.g. using `six`, which we will soon drop from the rest of the codebase.
    
    Likewise, the module uses `pycodestyle` and `pyflakes`, neither of which are used anywhere else in the project.
    
    We would like a way to provide requirements for specifically this module and to not pollute the rest of the project in doing so.
    
    ### Solution
    Introduce a dedicated `3rdparty/requirements.txt` for the module, per https://www.pantsbuild.org/3rdparty_py.html#3rdpartypython.
    
    We also indicate `pants_venv` should install these dependencies locally and `release.sh` should build 3rd party wheels for this so that the released PEX continues to work for the contrib package.
Commits on Jun 29, 2019
  1. Replace `six.exec_()` with `exec()` (#7981)

    Eric-Arellano committed Jun 29, 2019
  2. Remove Python 2 handling of unicode (#7973)

    Eric-Arellano committed Jun 29, 2019
  3. Remove `future.moves` and `six.moves` imports (#7975)

    Eric-Arellano committed Jun 29, 2019
Commits on Jun 28, 2019
  1. Remove `text_type` and `binary_type` wrappers (#7970)

    Eric-Arellano committed Jun 28, 2019
    We can now safely use `str` and `bytes` everywhere.
  2. Fix `cargo.sh` to use Python 3 (#7972)

    Eric-Arellano committed Jun 28, 2019
    On many systems `python` resolves to `python2.7`. We cannot run Pants with Python 2.7 anymore, so this results in a confusing error where the Rust linters try to bootstrap Pants with Python 2 on macOS.
  3. Remove Python 2 `hashlib.hexdigest()` decoding (#7965)

    Eric-Arellano committed Jun 28, 2019
    In Python 2, `hashlib.hexdigest()` returned a byte string, but unicode in Py3. We adopted an idiom to always return unicode, which we can now remove.
  4. Remove Python 2 select.select() code (#7964)

    Eric-Arellano authored and blorente committed Jun 28, 2019
    In #7882, we started using the much more efficient selectors module with Python 3 instead of select.select(). We may now remove the Python 2 code that was still using the original select.select().
  5. Simplify tests to no longer check for `u""` character or `unicode` ty…

    Eric-Arellano committed Jun 28, 2019
    …pe (#7966)
    
    When ran with Python 2, Pants would include an `u""` character in front of certain strings. This is no longer a concern thanks to only running via Python 3.
Commits on Jun 27, 2019
  1. Remove subprocess, futures, and faulthandler backports (#7950)

    Eric-Arellano committed Jun 27, 2019
    A followup PR should convert `subprocess` usages to use `subprocess.run()` where possible, which Python [encourages using as the main API](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module).
  2. Remove unittest.mock backport (#7951)

    Eric-Arellano committed Jun 27, 2019
    `mock` became part of the stdlib in Python 3.
Commits on Jun 24, 2019
  1. Add whitelist of folders that must remain Python 2 compatible (#7941)

    Eric-Arellano committed Jun 24, 2019
    Three of our folders may be run with Python 2, so they must keep their compatibility, e.g. `from __future__` imports.
    
    If this impacted more code, we would want to create (restore) formal linters to enforce the compatibility. However, because the code is so small and CI will fail upon dropping Py2 support, we simply add comments to the files explaining this expectation and modify `check_headers.py` to allow these files to not use Py3 headers.
    
    In the process, we refactor `check_headers.py` to use the `pathlib` module for much nicer path handling.
Commits on Jun 23, 2019
  1. Modernize classes to use Python 3 syntax (#7937)

    Eric-Arellano committed Jun 23, 2019
    Python 3 makes it repetitive to subclass `object` and to call the verbose form of `super()`.
    
    We use [this script](https://github.com/foursquare/fsqio/pull/60/files#diff-1597b620408328537d56e23e6d721849) to update the codebase to use the simpler style, excluding the `contrib.python.checks.checker` package. The main part of this script is these two regex substitutions:
    
    ```python
    re.sub(r"super\([a-zA-Z]+, [a-z]+\)", "super()")
    re.sub(r"class (?P<className>[a-zA-Z]*)\(object\):", r"class \g<className>:")
    ```
  2. Remove configparser backport (#7939)

    Eric-Arellano committed Jun 23, 2019
  3. Remove `collections.abc` backport (#7940)

    Eric-Arellano committed Jun 23, 2019
    We can now safely import `collections.abc`, which did not exist in Python 2.
Older
You can’t perform that action at this time.