Use more reliable method to run homebrew-pypi-poet

[claui]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same change?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your changes? Here's an example.
• Have you successfully run brew style with your changes locally?
• Have you successfully run brew tests with your changes locally?
• Have you successfully run brew man locally and committed any changes?

---

Update: Replacing the old method instead of adding to it.

---

This replaces the virtualenvwrapper method in the documentation with one based on the pipenv formula, a dependency manager that doubles as a virtualenvwrapper alternative.

A non-brewed installation of virtualenvwrapper is not always reliable.

For me, it fails like this:

$ source /Users/claudia/Library/Python/3.9/bin/virtualenvwrapper.sh
/usr/bin/python: No module named virtualenvwrapper
virtualenvwrapper.sh: There was a problem running the initialization hooks.
If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is
set properly.

It’s therefore useful to document a working alternative based on the pipenv formula, which is actually a dependency manager but doubles as a virtualenvwrapper alternative.

This also adds documentation on how to use poet on software that is not on PyPI.

Thanks again to @Rylan12 for technical help!

[BrewTestBot]

Review period will end on 2020-12-03 at 20:32:43 UTC.

[Rylan12]

Looks good assuming this worked as expected for you (it sounds like it did). I don't actually use pipenv so I can't really judge that part of it.

Just one question, is pipenv needed or can this be accomplished using a standard virtual environment (i.e. without the need to install pipenv).

Here's what seems to work for me:

# Use a temporary directory for the virtual environment
cd "$(mktemp -d)"

# Create and source a new virtual environment in the venv/ directory
python3 -m venv venv
source venv/bin/activate

# Install the package of interest as well as homebrew-pypi-poet
pip install some_package homebrew-pypi-poet
poet some_package

# Destroy the virtual environment
deactivate
rm -rf venv

And similarly (once in the virtual environment):

pip install homebrew-pypi-poet
poet -s requirements.in

---

Edit: I noticed this and commented in Slack, but figured I'd relay it here:

I don't know how common requirements.in files are, but the one that I saw did not contain any version info. This leads to poet always retrieving the latest version of the package which could, potentially, ignore version pinning/rules that exist in requirements.txt. For example, requirements.in may contain:

boltons

While requirements.txt may contain:

boltons==17.2.0

Using method number two, poet will retrieve the latest boltons version (20.2.1 as I'm writing this) instead of version 17.2.0. Because of this, we may not want to encourage people to use this method as it may lead to resource updates that are incorrect.

[claui]

Just one question, is pipenv needed or can this be accomplished using a standard virtual environment (i.e. without the need to install pipenv).

I tried your suggestion and it fails for me:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install simple-rom-manager homebrew-pypi-poet
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
WARNING: You are using pip version 20.2.3; however, version 20.3 is available.
You should consider upgrading via the '/private/var/folders/9k/72bcsqxd55q00scw81pt2f6h0000gp/T/tmp.ZtpH1gAB/venv/bin/python3 -m pip install --upgrade pip' command.

Using method number two, poet will retrieve the latest boltons version (20.2.1 as I'm writing this) instead of version 17.2.0. Because of this, we may not want to encourage people to use this method as it may lead to resource updates that are incorrect.

You’re right. I have removed that section. Thanks for the pointer!

[MikeMcQuaid]

I'd rather we figure out how to fix the original instructions rather than add an "alternatively" that may diverge over time and we're not really sure will work any more consistently.

[claui]

we're not really sure will work any more consistently.

My tests on two different devices and OS versions have established that it does work more consistently than the other two alternatives (i.e. the original one and the one @Rylan12 has suggested).

I’m not sure we’re not overthinking this. Are we supposed to be risk averse with regards to the core software and core formulae? Yes, absolutely. Shall we be risk averse at updating out-of date documentation? I don’t know.

We can always revert later if my variant turns out to be wrong.

that may diverge over time

I wouldn’t mind just replacing the original instructions.

[claui]

I'd rather we figure out how to fix the original instructions

@MikeMcQuaid Done; I replaced the original instructions.
Would appreciate someone other than myself to test this.

Changes addressed.

[MikeMcQuaid]

I’m not sure we’re not overthinking this. Are we supposed to be risk averse with regards to the core software and core formulae? Yes, absolutely. Shall we be risk averse at updating out-of date documentation? I don’t know.

I'm risk averse in adding an "alternatively" instead of either fixing whatever is broken originally.

I wouldn’t mind just replacing the original instructions.

Yup, this seems like a good idea if someone else can verify this. I've removed my objections to this PR being merged but I don't have enough context to comment on the changes here.

[claui]

Yup, this seems like a good idea if someone else can verify this. I've removed my objections to this PR being merged but I don't have enough context to comment on the changes here.

Fair enough. Thanks for your thoughts and for the update @MikeMcQuaid 👍

[Rylan12]

Would appreciate someone other than myself to test this.

I will take a look at this later today. It's worth noting, though, that my testing is not done from a clean install and so my environment may influence the results. Already, it seems that Claudia and I have had different results when trying an alternative solution. See #9391 (review) and #9391 (comment)

[BrewTestBot]

Review period ended.

[Rylan12]

Okay, here are my results. I will test three things:

1. The existing method documented here
2. @claui's method documented in this PR
3. My method documented in Use more reliable method to run homebrew-pypi-poet #9391 (review)

Before each trial, I will reset my environment by running brew reinstall python and ensuring that python3 -m pip list returns the following: (which, according to the python@3.9 formula, is the expected baseline set of packages)

$ python3 -m pip list
Package    Version
---------- -------
pip        20.3
setuptools 50.3.2
wheel      0.35.1

The package that I will be testing with is snakemake. According to the snakemake formula there are several dependencies that should become resource blocks.

Note: I've included shortened versions of the outputs of each command. You can view the entire output here.

1. The existing method from the docs

$ python3 -m pip install virtualenvwrapper
<installs successfully>

$ source $(brew --prefix)/bin/virtualenvwrapper.sh
<runs successfully>

$ mktmpenv
<runs successfully>

$ pip3 install snakemake homebrew-pypi-poet
<installs successfully>

$ poet snakemake
<outputs packages as expected>

$ deactivate
<removed virtual environment as expected>

Verdict: Success

2. @claui's method documented in this PR

$ brew reinstall pipenv
<installs successfully>

$ cd "$(mktemp -d)"
<runs successfully>

$ pipenv install snakemake homebrew-pypi-poet
<installs successfully>

$ pipenv run poet snakemake
<outputs packages as expected>

$ pipenv --rm
<runs successfully>

Verdict: Success

3. My method documented in #9391 (review)

$ cd "$(mktemp -d)"
<runs successfully>

$ python3 -m venv venv
<runs successfully>

$ source venv/bin/activate
<runs successfully>

$ pip3 install snakemake homebrew-pypi-poet
<installs successfully>

$ poet snakemake
<outputs packages as expected>

$ deactivate
<runs successfully>

Verdict: Success

Conclusion

These all seem to be fine with me. However, the pipenv method did not work. The error I got is a known python@3.9 issue, though, that FX is currently working on.

[BrewTestBot]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[claui]

@Rylan12 Does the python@3.9 issue still persist?

[claui]

@MikeMcQuaid This PR isn’t stale.
According to what @Rylan12 wrote, it’s blocked by, and waiting on, a Python 3.9 issue for which a fix is in progress.

[MikeMcQuaid]

@claui The existing method works for @Rylan12, the documented method requires fixes before it will work and @Rylan12 suggested another method. I don't think it's worth progressing with this as a result.

There's been no activity for 25 days beyond it being marked as stale and no activity since then hence my closing.

[claui]

There's been no activity for 25 days beyond it being marked as stale

@MikeMcQuaid What tag should I have used, other than do not merge, to indicate that this is blocked by an unrelated issue?

and no activity since then

I was away from my computer for the holidays.

The existing method works for @Rylan12,

and fails for me.

the documented method requires fixes before it will work

which @fxcoudert has been addressing in the (unrelated) Python 3.9 issue.

and @Rylan12 suggested another method

which fails for me.

I’d like to confirm that the method this PR proposes is the only one that works reliably.

[Rylan12]

Since the last time I looked at this, the Python issue mentioned has been fixed. I went back and re-attempted @claui's proposed fix and it worked fine with the fixed Python 3.9.

I've updated my comment above to reflect this.

[claui]

Thanks @Rylan12 for the update and for re-trying.

And apologies @MikeMcQuaid for my lack of responsiveness.

[claui]

@MikeMcQuaid With the proposed method now confirmed to work for both Rylan and myself, please consider re-opening. Thank you.

[MikeMcQuaid]

@MikeMcQuaid What tag should I have used, other than do not merge, to indicate that this is blocked by an unrelated issue?

I'd have expected a comment from the PR author noting that.

I’d like to confirm that the method this PR proposes is the only one that works reliably.

I'm fine with this being merged as-is.

[claui]

Pinging @dtrodrigues for a third opinion (I just saw #10039.)

[dtrodrigues]

From the existing documentation, the first method of brew update-python-resources <formula> should remain the preferred way, since it will use pipgrip to resolve dependencies, which is what brew bump-formula-pr also uses. If we do need to provide a manual way to get resources, I think what @Rylan12 suggested would be my preferred method since it doesn't require installing another package like pipenv or virtualenvwrapper: #9391 (review). The error from @claui of ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv. implies that the user's global pip conf includes a config option to always do --user installs, which don't work within a venv. This can be checked via pip config list.

It's worth noting that the dependency resolution of pipgrip (PubGrub) differs from the dependency resolution of pip, which is what all manual methods here use. pip does not and will not use PubGrub: pypa/pip#8203 and recently majorly changed their dependency resolution algorithm: https://blog.python.org/2020/11/pip-20-3-release-new-resolver.html

[Rylan12]

Is changing the command to pip --isolated install <packages> a viable solution? Not familiar enough with pip to make that call.

Also, is there a reason to use python3 -m pip over pip or pip3?

In terms of the dependency resolver, is this going to cause issues? As in, should update-python-resources use the same resolver as pip? If we're consistent, it may not be a huge issue. I doubt there are differences that are significant enough to cause problems. That being said, this is the first time I've heard about PubGrub and that stuff, so I'm probably not qualified to say for sure.

[dtrodrigues]

Just --no-user should be sufficient as a pip argument.

Longer-term, the divergence between pipgrip and pip dependency resolution may be an issue, but I'm not aware of any current differences in their output.

[dtrodrigues]

Within a venv, there's no difference between python3 -m pip and pip and pip3 given how venv modifies your path when you activate it.

[Rylan12]

@claui can you try the method I suggested in #9391 (review) again, this time adding --no-user to the pip install command?

[claui]

@Rylan12 Yay – that works! 🎉

Also:

The error from @claui of ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv. implies that the user's global pip conf includes a config option to always do --user installs, which don't work within a venv. This can be checked via pip config list.

@dtrodrigues You’re absolutely right. pip config -v list revealed that I had a ~/Library/'Application Support'/pip/pip.conf lying around, which said:

global.user=1

Trashing the pip.conf fixed the issue for me.
I can still reproduce the problem by running pip3 config set global.user true before the pip install part.

@dtrodrigues @Rylan12 Is it worth including --no-user in the solution? Or is the situation exotic enough that we shouldn’t mind?

[claui]

I think what @Rylan12 suggested would be my preferred method since it doesn't require installing another package like pipenv or virtualenvwrapper

Now that it works for everyone, I absolutely agree.

[claui]

@Rylan12 Do you want to open a PR with your solution (and get proper credit) or would you prefer me to do it?

[Rylan12]

@claui Happy to open a new PR (although I wouldn't have been upset at all if you went ahead and made the changes in here)!

My gut is that it's a pretty specific problem so unless more users report issues, I'm not sure we need to include it. I have no problem including it, though, if we think it's worth it.