404 error using github hashable URL

[awbirdsall]

I'd like to include a github repo in requirements.in with a hashable URL. I've been looking at what look like the relevant PRs, like #807, but I'm having a hard time figuring out the correct syntax. I'm trying to follow the suggestion of # Consider using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip and the format of test_generate_hashes_with_url(), but I get a 404 error (see below).

I feel like I must just be misunderstanding the pip-tools/pip syntax here, but I'm having a hard time troubleshooting. Any help would be much appreciated, thanks!

Environment Versions

1. OS Type: Windows 10

2. Python version: $ python -V: Python 3.7.4

3. pip version: $ pip --version: pip 19.2.3

4. pip-tools version: $ pip-compile --version: pip-compile, version 4.1.0

Steps to replicate

> cat test-requirements.in
https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools

2. pip-compile test-requirements.in

Expected result

test-requirements.txt created.

Actual result

HTTP error 404 while getting https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools
Could not install requirement pip-tools from https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools (from -r test-requirements.in (line 1)) because of error 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip
Traceback (most recent call last):
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\operations\prepare.py", line 199, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\download.py", line 1064, in unpack_url
    progress_bar=progress_bar
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\download.py", line 924, in unpack_http_url
    progress_bar)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\download.py", line 1128, in _download_http_url
    resp.raise_for_status()
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_vendor\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\Scripts\pip-compile.exe\__main__.py", line 9, in <module>
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\scripts\compile.py", line 352, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\resolver.py", line 168, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\resolver.py", line 263, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\resolver.py", line 339, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\repositories\local.py", line 70, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\repositories\pypi.py", line 254, in get_dependencies
    download_dir, ireq, wheel_cache
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\piptools\repositories\pypi.py", line 213, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "c:\users\<USER>\appdata\local\continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\operations\prepare.py", line 210, in prepare_linked_requirement
    (req, exc, req.link)
pip._internal.exceptions.InstallationError: Could not install requirement pip-tools from https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools (from -r test-requirements.in (line 1)) because of HTTP error 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip for URL https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools

[atugushev]

Hello @awbirdsall,

Thanks for the issue! I've checked it locally on Windows 10 and couldn't reproduce it. Can you open the link https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip in your browser and download the archive directly?

[awbirdsall]

Thanks @atugushev, yes, the strange thing to me has been I'm able to download the zip archive from the browser...

[atugushev]

@awbirdsall

Can you install it via pip? For example:

$ pip install -v https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools

UPDATE: If you can't — could you attach the output?

[awbirdsall]

@atugushev with your suggestion I think I tracked down the source of my issue!

initially with pip install I got the same 404 error (output below). but I saw that pip is using github credentials I have saved to a ~/.netrc file. I deleted the netrc and tried again, and now both pip install and pip-compile test-requirements.in works.

I don't quite understand why my netrc credentials are causing the GET request to fail, and I'll probably have to figure that out when using pip-compile with a private github repo, but that's outside the scope of this issue...

I think this can probably be closed.

> pip install -v https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools
Created temporary directory: C:\Users\<USER>\AppData\Local\Temp\pip-ephem-wheel-cache-t7x51nzz
Created temporary directory: C:\Users\<USER>\AppData\Local\Temp\pip-req-tracker-9tjtlc62
Created requirements tracker 'C:\\Users\\<USER>\\AppData\\Local\\Temp\\pip-req-tracker-9tjtlc62'
Created temporary directory: C:\Users\<USER>\AppData\Local\Temp\pip-install-kmvxn9l6
Collecting pip-tools from https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools
  Created temporary directory: C:\Users\<USER>\AppData\Local\Temp\pip-unpack-d0km7j72
  Found credentials in netrc for github.com
  Looking up "https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip" in the cache
  No cache entry available
  Starting new HTTPS connection (1): github.com:443
  https://github.com:443 "GET /jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip HTTP/1.1" 404 None
  ERROR: HTTP error 404 while getting https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools
  ERROR: Could not install requirement pip-tools from https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools because of error 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip
Cleaning up...
Removed build tracker 'C:\\Users\\<USER>\\AppData\\Local\\Temp\\pip-req-tracker-9tjtlc62'
ERROR: Could not install requirement pip-tools from https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools because of HTTP error 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip for URL https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools
Exception information:
Traceback (most recent call last):
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\operations\prepare.py", line 199, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\download.py", line 1064, in unpack_url
    progress_bar=progress_bar
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\download.py", line 924, in unpack_http_url
    progress_bar)
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\download.py", line 1128, in _download_http_url
    resp.raise_for_status()
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_vendor\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\cli\base_command.py", line 188, in main
    status = self.run(options, args)
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\commands\install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "C:\Users\<USER>\AppData\Local\Continuum\miniconda3\envs\piptools\lib\site-packages\pip\_internal\operations\prepare.py", line 210, in prepare_linked_requirement
    (req, exc, req.link)
pip._internal.exceptions.InstallationError: Could not install requirement pip-tools from https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools because of HTTP error 404 Client Error: Not Found for url: https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip for URL https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools

[awbirdsall]

In case anyone else runs into similar issues, I'm only noticing the .netrc bad authorization issues when trying to pip install a github repo using the single file artifact format like above (e.g., https://github.com/jazzband/pip-tools/archive/7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip). When I pip install from a repo normally (e.g., git+https://github.com/jazzband/pip-tools.git) there are no issues.

The difference in the pip codebase is the latter is treated as a vcs_url and is pulled down locally with a git clone command. The former is treated as a http_url and attempted to be pulled down with a GET request using code in pip/_vendor/requests. For whatever reason, the .netrc credentials are not working correctly when included in the GET request to github generated by pip. (The .netrc file does work in other contexts.)

[awbirdsall]

Okay, traced down the root cause:

• because a netrc file was detected, the GET request is prepared with a HTTPBasicAuth object attached
• as a consequence the request header includes {"Authorization": "Basic <Basic Auth str>"}, where <Basic Auth str> is generated from b64encode(b':'.join((username, password))).strip().
• this does not conform to how Github wants to see an authorization header: {"Authorization": "token <TOKEN>"}. (see https://developer.github.com/v3/#authentication)

This is all taking place in vendored requests code within pip, in particular _vendor/requests/auth.py -- not sure whether any changes within pip or pip-tools would be able to resolve.

Again, thanks for the help! And this issue can probably be closed.

[atugushev]

I'm glad you've found the root of the issue, so I'll close this then. Thanks!