Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Python 3 #30

Closed
Eric-Arellano opened this issue Mar 11, 2019 · 5 comments

Comments

Projects
None yet
2 participants
@Eric-Arellano
Copy link
Contributor

commented Mar 11, 2019

With Pants soon to be released with Python 3 wheels pantsbuild/pants#7197 and the Python 3 migration nearing completion pantsbuild/pants#6062, we need to modify this setup script to allow Pants end users to consume the upgrade.

Goals for an ideal solution

  1. Decision of which interpreter to use applies to the whole organization.
    • Typically, an individual or team will be responsible for upgrading Pants. They should be the only ones to have to know this is even a decision. Every day users should not be concerned with this upgrade.
    • Thus, solution must be tracked by VCS.
    • Solution should not require something like each individual modifying their .bashrc with their individual preference. It should be an organization-level decision set for the whole repo.
  2. Transparent what is being used.
    • We certainly shouldn't print out to the console every time or anything. But we should make it possible to find, e.g. in a config file.
    • Will help with debugging.
  3. Possible to change the interpreter version.
    • For example, deciding to upgrade from 2.7 -> 3.6 or 3.6 -> 3.7.
    • This decision isn't expected to be very frequent, but it will sometimes happen.
    • When a change happens, must ensure that no one has to do anything beyond the admin. For example, we can't expect people to have to rm -rf their cache.
    • Non-goal: make it extremely easy to change back-and-forth between interpreter versions, a la ./pants2 and ./pants. We are not expecting the average Pants user to worry about which Python interpreter they use to run Pants as both should work equally as well and it is more cognitive load than necessary.
  4. Simple to make the first-time decision.
    • Clearly explain what the options are.
    • Don't overwhelm with too intricate of details.
    • Emphasize Py3 will soon be required for runtime.
    • Maybe interactive?
  5. Backwards compatible.
    • If the user doesn't change their behavior at all, Pants should continue working as before, i.e. continue using Python 2.7.
    • Note this doesn't mean that we can't use the new venv naming scheme. That's okay. Just Pants needs to keep working when the user doesn't change their behavior from before.

Proposed solution

Users should add pants_runtime_python_version: {2.7,3.6,3.7} to their pants.ini [GLOBAL] entry, just as they set pants_version.

We would update the website at https://www.pantsbuild.org/install.html to provide instructions, along the lines of:

Pants may be ran with Python 2.7, 3.6, or 3.7. This does not mean your code has to use this Python
version, as Pants will spawn subprocesses to run with the interpreter you want to use for your own
codebase; rather, this is what Pants uses for its engine.

While Pants defaults to Python 2.7, we recommend pinning the interpreter you prefer in `pants.ini` by
adding an entry like this:

``ini
[GLOBAL]
pants_version: 1.15.0
pants_runtime_python_version: 3.6
``

Any time you want to change the interpreter Pants uses, update this value in your `pants.ini`.

Some notes when choosing an interpreter:
- Ensure you have this Python version accessible on your PATH everywhere Pants is used in your organization.
- Pants will require Python 3.6+ to run as of release 1.17.0.

Tasks

  • Determine naming scheme for the venv folder.
    • Must specify the Py version so that changing interpreters does not require removing the cache.
    • Delete the legacy name folders? I think so, if they happened to already have a folder for that version.
  • Read in the value from pants.ini. Use this to set $PYTHON and determine bootstrap_venv.
  • Add testing.

@Eric-Arellano Eric-Arellano self-assigned this Mar 11, 2019

@cosmicexplorer

This comment has been minimized.

Copy link

commented Mar 11, 2019

For background, pantsbuild/pants#7151 is where we have previously used the .pants-python-version convention.

@cosmicexplorer

This comment has been minimized.

Copy link

commented Mar 11, 2019

And this sounds like a plan!

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Mar 12, 2019

FYI the proposal has changed from the original one of using a .pants-python-version sentinel file to instead read in the value from pants.ini, similar to how we resolve pants_version. See the edit history of the PR description and look for the entry on March 11, 2019 if you'd like to see the original proposal.

@cosmicexplorer

This comment has been minimized.

Copy link

commented Mar 12, 2019

For the record, we could definitely turn:

setup/pants

Lines 69 to 72 in 6dfccab

pants_version=$(
grep -E "^[[:space:]]*pants_version" pants.ini 2>/dev/null | \
cut -f2 -d: | tr -d " "
)
into:

 pants_version=$(
    # use extended regexps, don't print every line
    # when a `pants_version: ` line is found,
    # replace all the text from the start of the line up to `: ` with an empty string, and print
    sed -rne '/\s*pants_version: / s#^.*: ##p' pants.ini 2>/dev/null
    # could add `| head -n1` if there might be multiple such lines
  )

which I personally find easier to read.

Eric-Arellano added a commit to pantsbuild/pants that referenced this issue Mar 13, 2019

Add pants_runtime_python_version global option (#7363)
### Problem
As discussed in pantsbuild/setup#30, we decided to add support for using Python 3 to every day user's setup script `./pants` by having them pin the interpreter version they want to use in `pants.ini`, and then grepping that value to choose the relevant interpreter. However, this currently results in this error:

```
ERROR] Invalid option 'pants_runtime_python_version' under [GLOBAL] in /Users/eric/DocsLocal/code/projects/setup/pants.ini
Exception caught: (<class 'pants.option.config.ConfigValidationError'>)

Exception message: Invalid config entries detected. See log for details on which entries to update or remove.
(Specify --no-verify-config to disable this check.)
```

### Solution
Exactly how we handle `--pants-version`, introduce a new global option `--pants-runtime-python-version` that allows users to set `pants_runtime_python_version` in their `pants.ini`.

Even though Pants code cannot dynamically change the interpreter it runs with—as it is too late to change the interpreter once the process is already running—we can check that Pants is running with the intended interpreter and print a helpful message if it is not.

You can leave off the option, and you'll get no runtime check. Setup scripts are free to implement Python interpreter resolution in another way if they prefer.

### Result
* Including `pants_runtime_python_version` in `pants.ini` no longer results in an error.
* Running `./pants --pants-runtime-python-version=2.6` will error out. Running `./pants2 --pants-runtime-python-version=2.7` will work normally.

Eric-Arellano added a commit that referenced this issue Mar 16, 2019

Add support for running with Python 3.6 and 3.7 (#32)
### Problem
Now that we're releasing Pants with Python 3 wheels (pantsbuild/pants#7197, overall migration tracked by pantsbuild/pants#6062), we need to add a way for every day Pants users to consume the upgrade.

See #30 for more context, including the design goals.

### Solution
Parse the `pants_runtime_python_version` from `pants.ini` to determine which Python interpreter to use. If it's missing, default to Python 2.7.

This requires modifying the venv folders to support multiple Python versions. Rather than using the naming scheme `${pants_version}`, we now consistently use `${pants_version}_py{py_major_minor}`, regardless of if the user is using `pants_runtime_python_version`.

It also requires upgrading virtualenv to avoid a deprecation warning when running with Python 3.

Finally, we add tests to ensure this all works.

### Result
Running `./pants` like normal for the first time will bootstrap Pants again and create the new folder `1.15.dev3_py{27,36,37}`.

Changing the `$PYTHON` env var or the `pants_runtime_python_version` in `pants.ini` will change which Python version is used to run Pants.

When changing interpreters for the first time, it will bootstrap Pants again. After that, you can change without issue.
@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Mar 19, 2019

This was closed by #32.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.