Allow environment variable references to be passed through to the Pipfile unredacted

[matteius]

Fixes #2635

[matteius]

I'll add a news fragment before merging once this gets reviewed.

[oz123]

The string method of package.link should redact the authentication according to the code here:
https://github.com/pypa/pip/blob/65680b4bb1a2412e97bc90e49df6999b5f362e40/src/pip/_internal/models/link.py#L93

I suspect the f-string (which calls the repr ) should be also using the string method explicitly.

In other words: package.link should not show auth information.

[matteius]

@oz123 That is the problem -- people are using environment variables for the auth and they get redacted.

[oz123]

I still don't understand what issue is being fixed here.

We should not leak authentication information to Pipefile or Pipefile.lock.
If we merge this, we can expect people to open a CVE for this.

[oz123]

@oz123 That is the problem -- people are using environment variables for the auth and they get redacted.

Redacted from where? From the Pipefile? That's great, pipenv does its job.
If they get redacted from the files pipenv is writing while trying to install a package, then the fix should be some where else.

[matteius]

@oz123 For the case where the user is using environment variables, they don't get expanded, but they do get redacted, which means this doesn't work:

# requirements.txt
git+https://${AUTH_USER}:${AUTH_PASSWORD}@github.com/user/myproject.git@1.2.3#egg=myproject

Then run

> pip install -r requirements.txt

Becasue str(package.link) will redact it to be git+https://${AUTH_USER}:****@github.com/user/myproject.git@1.2.3#egg=myproject

If you have just a user variable, like this: git+https://${AUTH_USER}@github.com/user/myproject.git@1.2.3#egg=myproject
Then that variable will get redacted as well:
git+https://****@github.com/user/myproject.git@1.2.3#egg=myproject

Which creates a not lockable or installable Pipfile. The ask of #2635 is to fix this -- but you are probably correct as well, we need a way to detect if they are environment variables, to not redact them.

[oz123]

OK, I see where is the problem.

These are not environment variables, rather references to them. So if someone was smart and didn't specify the password, pip's own code still think it's a password.

Using _url can be dangerous, we could use split_auth_from_netloc from pip to check for a password or a variable and do something about it.

Something could be:

1. Warn the user.
2. Redact if it is a password
3. Pass it on to Pipfile if the password is the known form for a variable reference, e.g. ${PASSWORD} or ${password} or in regex: ${\w+}.

[matteius]

Changes made to address the concerns and tests added for the various cases.

[matteius]

@oz123 this PR is insightful, because mac os failed on every test pips_to_dep tests regarding requests, possibly because of my new Pipfile tests? I noticed a weird behavior on my mac this past week that took me a while to track down -- even if there is no virtualenv, if there is a Pipfile in a higher level directory, then pipenv prefers to use that over the Pipfile in your current working directory, at least on a mac.