Redact URL to hide password #6295

expobrain · 2019-02-23T13:53:03Z

When the verbose flag is used in pip install and in ~/.pip/pip.conf you setup the access to you private registry with basic auth, i.e. extra-index-url = https://username:pwd!@pypi.example.com/simple/, when using pip install -v the password is printed in clear to stdout.

This PR redact the URL to hide original password.

cjerdonek · 2019-02-24T07:25:51Z

src/pip/_internal/index.py

@@ -155,7 +166,7 @@ def _get_html_response(url, session):
    if _is_url_like_archive(url):
        _ensure_html_response(url, session=session)

-    logger.debug('Getting page %s', url)
+    logger.debug('Getting page %s', _redacted_url(url))


You should use the redact_password_from_url() function which has this purpose.

Good to know!, Fixed in the next commit

cjerdonek

A couple more comments.

cjerdonek · 2019-02-24T08:43:59Z

tests/functional/test_install_config.py

+        "Getting page https://user:<pwd>@example.com/simple/initools"
+        in result.stdout
+    )
+    virtualenv.clear()


Is this needed? If so, should it be in a finally block?

I used test_command_line_options_override_env_vars as a template and I assumed that you need to cleanup manually. Removed.

cjerdonek · 2019-02-24T08:45:40Z

tests/functional/test_install_config.py

+    assert (
+        "Getting page https://user:<pwd>@example.com/simple/initools"
+        in result.stdout
+    )


In addition, you might as well assert that my_password is not in the output. (And I would assign my_password to a variable so that that portion of the test doesn't pass because of a typo.)

xavfernandez · 2019-02-24T21:28:39Z

tests/functional/test_install_config.py

+    """
+    password = "my_password"
+
+    script.environ['PIP_INDEX_URL'] = (


Why the environ options instead of passing '--index-url', 'https://user:{}@example.com/simple/'.format(password) to script.pip ?

@xavfernandez Can you confirm if this test hits the network as written? It would be a lot better IMO if it didn't as I think such cases should be reduced to a minimum.

@expobrain It might be better to write a unit test (so in the unit directory rather than functional) of _get_html_response() to avoid hitting the network. You can look at test_get_legacy_build_wheel_path__no_names() and nearby functions for examples of using the caplog fixture (since the code wouldn't be running in a separate subprocess).

PS - it looks like there are already a number of unit tests of that function (search for test_get_html_response_), and you can follow the examples there of mocking the session response.

Can you confirm if this test hits the network as written? It would be a lot better IMO if it didn't as I think such cases should be reduced to a minimum.

I confirm it should (I was actually thinking about suggesting to add the network marker to the test) and I don't see a way around that for a functional test.
An unit test could indeed also remove the need for network.

FYI, here's an example of a test failing because of flakiness due to installing INITools in the test: https://dev.azure.com/pypa/pip/_build/results?buildId=5274&view=logs&jobId=c897e773-a04c-5d18-b461-2c6584d92c5a

@cjerdonek I've moved the test into unit, please have a look at it

cjerdonek · 2019-02-24T21:46:47Z

Btw, this needs a news file: https://pip.pypa.io/en/stable/development/contributing/#news-entries

expobrain · 2019-02-25T22:58:14Z

Btw, this needs a news file: https://pip.pypa.io/en/stable/development/contributing/#news-entries

Can I use this PR for the news file or better create a new issue?

cjerdonek · 2019-02-25T23:03:36Z

Can I use this PR for the news file or better create a new issue?

Thanks for asking. You can use this PR.

cjerdonek · 2019-02-25T23:35:12Z

tests/unit/test_index_html_page.py

@@ -118,6 +118,29 @@ def test_get_html_response_no_head(url):
    ]


+@mock.patch("pip._internal.index.logger")
+def test_get_html_response_dont_log_clear_text_password(m_logger):


You shouldn't need to mock logging. I'll repeat my suggestion from before:

You can look at test_get_legacy_build_wheel_path__no_names() and nearby functions for examples of using the caplog fixture (since the code wouldn't be running in a separate subprocess).

I forgot about it, and fixed in the next commit

cjerdonek

Looking a lot better now. Thanks for sticking it out! Just a couple more minor comments..

cjerdonek · 2019-02-27T11:40:29Z

tests/unit/test_index_html_page.py

+def test_get_html_response_dont_log_clear_text_password(caplog):
+    """
+    `_get_html_response()` shouldn't send a HEAD request if the URL does not
+    look like an archive, only the GET request that retrieves data.


Looks like this docstring was copied without updating.

cjerdonek · 2019-02-27T11:42:03Z

tests/unit/test_index_html_page.py

+
+    caplog.set_level(logging.DEBUG)
+
+    resp = _get_html_response(url_template.format(password), session=session)


Instead of using url_template, it would be simpler and more direct just to pass https://user:my_password@example.com/simple/.

cjerdonek · 2019-02-27T11:43:25Z

tests/unit/test_index_html_page.py

+    record = caplog.records[0]
+    assert record.levelname == 'DEBUG'
+    assert record.message.splitlines() == [
+        "Getting page {}".format(url_template.format("****")),


Again, you can just check that actual string instead of using nested format strings, so "Getting page https://user:****@example.com/simple/"... (It's generally better to be more explicit in test code.)

cjerdonek · 2019-02-27T11:44:21Z

tests/unit/test_index_html_page.py

+    look like an archive, only the GET request that retrieves data.
+    """
+    password = "my_password"
+    url_template = "https://user:{}@example.com/simple/"


You won't need these two variables (see below).

cjerdonek

A couple wording comments.

cjerdonek · 2019-03-01T09:04:17Z

news/6295.bugfix

@@ -0,0 +1 @@
+Redacts the URL of the extra index when `pip -v` to hide the user's password


Since the URL isn't being removed but only the password, it would be better to say something like, "Redact the password from the extra index URL when using pip -v." Also, you want to use double backticks before and after "pip -v" rather than single, and end the sentence with a period.

Do you mean like ``pip -v``?

Yeah. I couldn't get it to render in GitHub's viewer when I first tried. The reason for double is because the contents are reStructuredText and not, say, Markdown: https://pip.pypa.io/en/stable/development/contributing/#contents-of-a-news-entry

I used a backslash before the backtick to escape it and let Markdown ignore it

cjerdonek · 2019-03-01T09:08:03Z

tests/unit/test_index_html_page.py

@@ -118,6 +118,34 @@ def test_get_html_response_no_head(url):
    ]


+def test_get_html_response_dont_log_clear_text_password(caplog):
+    """
+    `_get_html_response()` shouldn't log the full index's URL includoing the


Typo: including

Also, this could be a little clearer. How about: "_get_html_response() should redact the password from the index URL in its DEBUG log message."

cjerdonek · 2019-03-01T12:22:56Z

Thank you for all your work on this, @expobrain!

expobrain · 2019-03-01T12:24:25Z

Thank you @cjerdonek too for the code review!

lock · 2019-05-28T23:10:35Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Redact URL to hide password

00b872f

cjerdonek reviewed Feb 24, 2019

View reviewed changes

Use already existent utility function ot hide password

34b123d

expobrain mentioned this pull request Feb 24, 2019

pip 8.1.1 fails to use credentials specified inside the ~/pypirc file for downloading packages #3643

Closed

cjerdonek reviewed Feb 24, 2019

View reviewed changes

Updated unit test

5c232bc

xavfernandez approved these changes Feb 24, 2019

View reviewed changes

xavfernandez reviewed Feb 24, 2019

View reviewed changes

Pass index url straight to the command

ddb7320

cjerdonek added type: bug A confirmed bug or unintended behavior type: bugfix labels Feb 24, 2019

expobrain force-pushed the redact_url branch from 59169c2 to a5a5369 Compare February 25, 2019 22:37

Moved test into unit

70dee95

expobrain force-pushed the redact_url branch from a5a5369 to 70dee95 Compare February 25, 2019 22:42

cjerdonek reviewed Feb 25, 2019

View reviewed changes

Use caplog instead of mock

c33535b

expobrain force-pushed the redact_url branch from 2c929e0 to c33535b Compare February 27, 2019 11:10

cjerdonek reviewed Feb 27, 2019

View reviewed changes

cjerdonek added the type: security Has potential security implications label Feb 27, 2019

Added NEWS file

1214941

expobrain force-pushed the redact_url branch 2 times, most recently from 1a2af7b to 9e13fea Compare March 1, 2019 08:49

Improved unit test

5bfb47c

expobrain force-pushed the redact_url branch from 9e13fea to 5bfb47c Compare March 1, 2019 09:01

cjerdonek reviewed Mar 1, 2019

View reviewed changes

expobrain added 2 commits March 1, 2019 11:28

Better unit test comment

137a7ab

Better NEWS files content

b37ad3f

cjerdonek approved these changes Mar 1, 2019

View reviewed changes

cjerdonek merged commit 729404d into pypa:master Mar 1, 2019

expobrain deleted the redact_url branch March 1, 2019 12:23

lock bot added the auto-locked Outdated issues that have been locked by automation label May 28, 2019

lock bot locked as resolved and limited conversation to collaborators May 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redact URL to hide password #6295

Redact URL to hide password #6295

expobrain commented Feb 23, 2019

cjerdonek Feb 24, 2019 •

edited

Loading

expobrain Feb 24, 2019

cjerdonek left a comment

cjerdonek Feb 24, 2019

expobrain Feb 24, 2019

cjerdonek Feb 24, 2019 •

edited

Loading

expobrain Feb 24, 2019

xavfernandez Feb 24, 2019

cjerdonek Feb 24, 2019

cjerdonek Feb 24, 2019 •

edited

Loading

xavfernandez Feb 24, 2019

cjerdonek Feb 25, 2019

expobrain Feb 25, 2019

cjerdonek commented Feb 24, 2019

expobrain commented Feb 25, 2019

cjerdonek commented Feb 25, 2019

cjerdonek Feb 25, 2019

expobrain Feb 27, 2019

cjerdonek left a comment

cjerdonek Feb 27, 2019

cjerdonek Feb 27, 2019

cjerdonek Feb 27, 2019

cjerdonek Feb 27, 2019

cjerdonek left a comment

cjerdonek Mar 1, 2019

expobrain Mar 1, 2019 •

edited

Loading

cjerdonek Mar 1, 2019 •

edited

Loading

expobrain Mar 1, 2019

cjerdonek Mar 1, 2019

cjerdonek commented Mar 1, 2019

expobrain commented Mar 1, 2019

lock bot commented May 28, 2019


		caplog.set_level(logging.DEBUG)

		resp = _get_html_response(url_template.format(password), session=session)

		@@ -0,0 +1 @@
		Redacts the URL of the extra index when `pip -v` to hide the user's password

Redact URL to hide password #6295

Redact URL to hide password #6295

Conversation

expobrain commented Feb 23, 2019

cjerdonek Feb 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek Feb 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek Feb 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek commented Feb 24, 2019

expobrain commented Feb 25, 2019

cjerdonek commented Feb 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

expobrain Mar 1, 2019 • edited Loading

Choose a reason for hiding this comment

cjerdonek Mar 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek commented Mar 1, 2019

expobrain commented Mar 1, 2019

lock bot commented May 28, 2019

cjerdonek Feb 24, 2019 •

edited

Loading

cjerdonek Feb 24, 2019 •

edited

Loading

cjerdonek Feb 24, 2019 •

edited

Loading

expobrain Mar 1, 2019 •

edited

Loading

cjerdonek Mar 1, 2019 •

edited

Loading