Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redact URL to hide password #6295

Merged
merged 10 commits into from Mar 1, 2019

Conversation

Projects
None yet
3 participants
@expobrain
Copy link
Contributor

commented Feb 23, 2019

When the verbose flag is used in pip install and in ~/.pip/pip.conf you setup the access to you private registry with basic auth, i.e. extra-index-url = https://username:pwd!@pypi.example.com/simple/, when using pip install -v the password is printed in clear to stdout.

This PR redact the URL to hide original password.

@@ -155,7 +166,7 @@ def _get_html_response(url, session):
if _is_url_like_archive(url):
_ensure_html_response(url, session=session)

logger.debug('Getting page %s', url)
logger.debug('Getting page %s', _redacted_url(url))

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 24, 2019

Member

You should use the redact_password_from_url() function which has this purpose.

This comment has been minimized.

Copy link
@expobrain

expobrain Feb 24, 2019

Author Contributor

Good to know!, Fixed in the next commit

@cjerdonek
Copy link
Member

left a comment

A couple more comments.

"Getting page https://user:<pwd>@example.com/simple/initools"
in result.stdout
)
virtualenv.clear()

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 24, 2019

Member

Is this needed? If so, should it be in a finally block?

This comment has been minimized.

Copy link
@expobrain

expobrain Feb 24, 2019

Author Contributor

I used test_command_line_options_override_env_vars as a template and I assumed that you need to cleanup manually. Removed.

assert (
"Getting page https://user:<pwd>@example.com/simple/initools"
in result.stdout
)

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 24, 2019

Member

In addition, you might as well assert that my_password is not in the output. (And I would assign my_password to a variable so that that portion of the test doesn't pass because of a typo.)

This comment has been minimized.

Copy link
@expobrain

expobrain Feb 24, 2019

Author Contributor

Done

"""
password = "my_password"

script.environ['PIP_INDEX_URL'] = (

This comment has been minimized.

Copy link
@xavfernandez

xavfernandez Feb 24, 2019

Contributor

Why the environ options instead of passing '--index-url', 'https://user:{}@example.com/simple/'.format(password) to script.pip ?

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 24, 2019

Member

@xavfernandez Can you confirm if this test hits the network as written? It would be a lot better IMO if it didn't as I think such cases should be reduced to a minimum.

@expobrain It might be better to write a unit test (so in the unit directory rather than functional) of _get_html_response() to avoid hitting the network. You can look at test_get_legacy_build_wheel_path__no_names() and nearby functions for examples of using the caplog fixture (since the code wouldn't be running in a separate subprocess).

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 24, 2019

Member

PS - it looks like there are already a number of unit tests of that function (search for test_get_html_response_), and you can follow the examples there of mocking the session response.

This comment has been minimized.

Copy link
@xavfernandez

xavfernandez Feb 24, 2019

Contributor

Can you confirm if this test hits the network as written? It would be a lot better IMO if it didn't as I think such cases should be reduced to a minimum.

I confirm it should (I was actually thinking about suggesting to add the network marker to the test) and I don't see a way around that for a functional test.
An unit test could indeed also remove the need for network.

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 25, 2019

Member

FYI, here's an example of a test failing because of flakiness due to installing INITools in the test: https://dev.azure.com/pypa/pip/_build/results?buildId=5274&view=logs&jobId=c897e773-a04c-5d18-b461-2c6584d92c5a

This comment has been minimized.

Copy link
@expobrain

expobrain Feb 25, 2019

Author Contributor

@cjerdonek I've moved the test into unit, please have a look at it

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 24, 2019

@expobrain expobrain force-pushed the expobrain:redact_url branch from 59169c2 to a5a5369 Feb 25, 2019

@expobrain expobrain force-pushed the expobrain:redact_url branch from a5a5369 to 70dee95 Feb 25, 2019

@expobrain

This comment has been minimized.

Copy link
Contributor Author

commented Feb 25, 2019

Btw, this needs a news file: https://pip.pypa.io/en/stable/development/contributing/#news-entries

Can I use this PR for the news file or better create a new issue?

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 25, 2019

Can I use this PR for the news file or better create a new issue?

Thanks for asking. You can use this PR.

@@ -118,6 +118,29 @@ def test_get_html_response_no_head(url):
]


@mock.patch("pip._internal.index.logger")
def test_get_html_response_dont_log_clear_text_password(m_logger):

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 25, 2019

Member

You shouldn't need to mock logging. I'll repeat my suggestion from before:

You can look at test_get_legacy_build_wheel_path__no_names() and nearby functions for examples of using the caplog fixture (since the code wouldn't be running in a separate subprocess).

This comment has been minimized.

Copy link
@expobrain

expobrain Feb 27, 2019

Author Contributor

I forgot about it, and fixed in the next commit

@expobrain expobrain force-pushed the expobrain:redact_url branch from 2c929e0 to c33535b Feb 27, 2019

@cjerdonek
Copy link
Member

left a comment

Looking a lot better now. Thanks for sticking it out! Just a couple more minor comments..

def test_get_html_response_dont_log_clear_text_password(caplog):
"""
`_get_html_response()` shouldn't send a HEAD request if the URL does not
look like an archive, only the GET request that retrieves data.

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 27, 2019

Member

Looks like this docstring was copied without updating.


caplog.set_level(logging.DEBUG)

resp = _get_html_response(url_template.format(password), session=session)

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 27, 2019

Member

Instead of using url_template, it would be simpler and more direct just to pass https://user:my_password@example.com/simple/.

record = caplog.records[0]
assert record.levelname == 'DEBUG'
assert record.message.splitlines() == [
"Getting page {}".format(url_template.format("****")),

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 27, 2019

Member

Again, you can just check that actual string instead of using nested format strings, so "Getting page https://user:****@example.com/simple/"... (It's generally better to be more explicit in test code.)

look like an archive, only the GET request that retrieves data.
"""
password = "my_password"
url_template = "https://user:{}@example.com/simple/"

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Feb 27, 2019

Member

You won't need these two variables (see below).

@expobrain expobrain force-pushed the expobrain:redact_url branch 2 times, most recently from 1a2af7b to 9e13fea Mar 1, 2019

@expobrain expobrain force-pushed the expobrain:redact_url branch from 9e13fea to 5bfb47c Mar 1, 2019

@cjerdonek
Copy link
Member

left a comment

A couple wording comments.

@@ -0,0 +1 @@
Redacts the URL of the extra index when `pip -v` to hide the user's password

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Mar 1, 2019

Member

Since the URL isn't being removed but only the password, it would be better to say something like, "Redact the password from the extra index URL when using pip -v." Also, you want to use double backticks before and after "pip -v" rather than single, and end the sentence with a period.

This comment has been minimized.

Copy link
@expobrain

expobrain Mar 1, 2019

Author Contributor

Do you mean like ``pip -v``?

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Mar 1, 2019

Member

Yeah. I couldn't get it to render in GitHub's viewer when I first tried. The reason for double is because the contents are reStructuredText and not, say, Markdown: https://pip.pypa.io/en/stable/development/contributing/#contents-of-a-news-entry

This comment has been minimized.

Copy link
@expobrain

expobrain Mar 1, 2019

Author Contributor

I used a backslash before the backtick to escape it and let Markdown ignore it

@@ -118,6 +118,34 @@ def test_get_html_response_no_head(url):
]


def test_get_html_response_dont_log_clear_text_password(caplog):
"""
`_get_html_response()` shouldn't log the full index's URL includoing the

This comment has been minimized.

Copy link
@cjerdonek

cjerdonek Mar 1, 2019

Member

Typo: including

Also, this could be a little clearer. How about: "_get_html_response() should redact the password from the index URL in its DEBUG log message."

expobrain added some commits Mar 1, 2019

@cjerdonek cjerdonek merged commit 729404d into pypa:master Mar 1, 2019

29 checks passed

Linux Build #20190301.6 succeeded
Details
Linux (Package) Package succeeded
Details
Linux (Test Primary Python27) Test Primary Python27 succeeded
Details
Linux (Test Primary Python36) Test Primary Python36 succeeded
Details
Linux (Test Secondary Python34) Test Secondary Python34 succeeded
Details
Linux (Test Secondary Python35) Test Secondary Python35 succeeded
Details
Linux (Test Secondary Python37) Test Secondary Python37 succeeded
Details
Windows Build #20190301.6 succeeded
Details
Windows (Package) Package succeeded
Details
Windows (Test Primary Python27-x64) Test Primary Python27-x64 succeeded
Details
Windows (Test Primary Python36-x64) Test Primary Python36-x64 succeeded
Details
Windows (Test Secondary Python27-x86) Test Secondary Python27-x86 succeeded
Details
Windows (Test Secondary Python34-x64) Test Secondary Python34-x64 succeeded
Details
Windows (Test Secondary Python34-x86) Test Secondary Python34-x86 succeeded
Details
Windows (Test Secondary Python35-x64) Test Secondary Python35-x64 succeeded
Details
Windows (Test Secondary Python35-x86) Test Secondary Python35-x86 succeeded
Details
Windows (Test Secondary Python36-x86) Test Secondary Python36-x86 succeeded
Details
Windows (Test Secondary Python37-x64) Test Secondary Python37-x64 succeeded
Details
Windows (Test Secondary Python37-x86) Test Secondary Python37-x86 succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
macOS Build #20190301.6 has test failures
Details
macOS (Package) Package succeeded
Details
macOS (Test Primary Python27) Test Primary Python27 succeeded
Details
macOS (Test Primary Python36) Test Primary Python36 succeeded
Details
macOS (Test Secondary Python34) Test Secondary Python34 succeeded
Details
macOS (Test Secondary Python35) Test Secondary Python35 succeeded
Details
macOS (Test Secondary Python37) Test Secondary Python37 succeeded
Details
news-file/pr News files updated and/or change is trivial.
Details
@cjerdonek

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

Thank you for all your work on this, @expobrain!

@expobrain expobrain deleted the expobrain:redact_url branch Mar 1, 2019

@expobrain

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

Thank you @cjerdonek too for the code review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.