HTTPError: 400 Client Error: Bad Request for url: {URL}

[RoCCoCo13]

Hi everybody,

I have been experiencing an error since a while already and I cannot figure out what am I doing wrong. I have a pypiserver running in a machine in my local network and every time I try to push a new package to it I'm getting a 400 Client error with no description at all.

The pypiserver runs on a Ubuntu machine using supervisor, and this is the configuration:

[program:my_pypiserver]
command = pypi-server -vvv -p 8000 -P /home/pypiserver/.htaccess /home/pypiserver/packages
user = pypiserver
autostart = true
autorestart = true
starsecs = 10
startretries = 2
redirect_stderr = true
stdout_logfile = /home/pypiserver/logs/stdout.log
stderr_logfile = /home/pypiserver/logs/stderr.log

The pypiserver version is:

ale@quality-agent:~$ pypi-server --version
pypiserver 1.3.2
ale@quality-agent:~$ python3 --version
Python 3.7.7

So it's serving here http://quality-agent:8000/ and I can verify that it's reachable by accessing in the browser:

My .pypirc file looks like this:

[distutils]
index-servers=
    serverpypi

[serverpypi]
repository: http://quality-agent:8000/simple
username: ***
password: ***

For example:

λ twine upload --verbose --repository serverpypi dist\*
Uploading distributions to http://quality-agent:8000/simple
Uploading tavern-1.1.0-py3-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90.8k/90.8k [00:00<00:00, 411kB/s] HTTPError: 400 Client Error: Bad Request for url: http://quality-agent:8000/simple

The PKG-INFO of this upload example is here:
PKG-INFO.txt

But I've seen it with every package I've tried, this is just an example.

My guess is that the problem is somewhere in twine because I cannot see any log trace in my pypiserver, and I'm running it with the -vvv verbose option.

To confirm that the problem is not any of my packages, I'm trying to build and push to my pypiserver a working package available in pypi. In the example we can see that I used tavern.

My pypiserver is running in a ubuntu machine and I'm trying to push the packages from the same Ubuntu machine or even from my Windows machine with the same result.

In my Ubuntu machine I use python 3.7.5 and twine

jenkins@quality-agent:~/workspace$ pipenv run python --version
Python 3.7.5

twine version 3.1.1 (pkginfo: 1.5.0.1, requests: 2.23.0, setuptools: 46.1.3,
requests-toolbelt: 0.9.1, tqdm: 4.46.0)

In my Windows setup I tried with python 3.8.3 and twine:

λ python --version
Python 3.8.3

twine version 3.1.1 (pkginfo: 1.5.0.1, requests: 2.23.0, setuptools: 46.4.0, requests-toolbelt: 0.9.1, tqdm: 4.46.0)

Does anyone knows what the problem could be? I'm running out of ideas about where to look to obtain more info or logs to analyze.

In case it helps, I add here some lines of the pypiserver logs, but as commented before, it throws nothing when executing the twine command to upload any package:

2020-05-23 10:44:32,366|pypiserver.bottle|INFO|140077160449856|Bottle v0.13-dev server starting up (using AutoServ                                                                                      er())...
2020-05-23 10:44:32,366|pypiserver.bottle|INFO|140077160449856|Listening on http://0.0.0.0:8000/
2020-05-23 10:44:32,366|pypiserver.bottle|INFO|140077160449856|Hit Ctrl-C to quit.

/usr/lib/python3/dist-packages/twisted/web/error.py:22: DeprecationWarning: Using or importing the ABCs from 'coll                                                                                      ections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import Sequence
2020-05-23 11:12:18,858|pypiserver._app|INFO|140077049681664|<LocalRequest: GET {URL}:8000/>
/usr/local/lib/python3.7/dist-packages/pypiserver/bottle.py:3640: DeprecationWarning: Flags not at the start of the expression '((?mx)         # ver' (truncated)
  patterns = [re.compile(p % pattern_vars) for p in patterns]
/usr/local/lib/python3.7/dist-packages/pypiserver/bottle.py:3640: DeprecationWarning: Flags not at the start of the expression '\\{\\{((?:((?mx)      ' (truncated)
  patterns = [re.compile(p % pattern_vars) for p in patterns]
2020-05-23 11:12:18,870|pypiserver._app|INFO|140077049681664|200 OK
2020-05-23 11:12:22,869|pypiserver._app|INFO|140077024503552|<LocalRequest: GET {URL}:8000/simple/>
2020-05-23 11:12:22,872|pypiserver._app|INFO|140077024503552|200 OK
2020-05-23 11:12:24,197|pypiserver._app|INFO|140077041288960|<LocalRequest: GET {URL}:8000/simple/ediabas/>
2020-05-23 11:12:24,219|pypiserver._app|INFO|140077041288960|200 OK

Instead we can see the logs of the GET requests when accessing to the HTML representation in the browser to visualize the content of the package folder.

Many thanks

[sigmavirus24]

We don't know:

• What version of python you're using
• What version of twine you're using
• What your twine configuration is like
• What version of pypiserver is running against what version of python
• What the PKG-INFO is like for the package you're trying to upload
• How your network is configured and whether what you're specifying for the URL actually routes to the right pypiserver instance (this is my first concern since any app server worth its salt would log a 400 response if it actually received one)

In short, you ignored the issue template which has information we need, and have given us nothing we can use to actually help you.

Without any of this information, I'm closing the issue.

[RoCCoCo13]

Hi @sigmavirus24,

Thanks for your quick response and sorry for the inconvenience. I have modified my post adding the information missing.

I added all the versions and a description of the setup. I also added an example where we can see that I can reach the package server in the browser with the specified URL. Also, I added the PKG-INFO of the package I'm trying to upload.

I hope that with this new information you can consider to open the issue.

Many thanks for your support and kind regards

[sigmavirus24]

[serverpypi]
repository: http://quality-agent:8000/simple
username: ***
password: ***

Have you tried adding the documented trailing / to your pypirc? That's been known to cause issues with some servers that don't redirect appropriately.

[RoCCoCo13]

Hi @sigmavirus24 ,

Thanks for your quick response. Sure I did, I tried some combinations, with and without the trailing and adding or not the "simple" to the end but the results it's pretty much the same:

λ twine upload --verbose --repository serverpypi dist\* --verbose
Uploading distributions to http://quality-agent:8000
Uploading tavern-1.1.0-py3-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90.8k/90.8k [00:00<00:00, 401kB/s] HTTPError: 400 Client Error: Bad Request for url: http://quality-agent:8000/

The problem is that even adding the --verbose option I cannot see any logs that would indicate me what am I doing wrong. Not here or in the pypiserver side.

Any ideas about where could I obtain more logs or info?

Many thanks

[sigmavirus24]

https://pypiserver.readthedocs.io/en/latest/README.html#uploading-packages-from-sources-remotely Indicates that http://quality-agent:8000/ is the right URL to use. But tracing through that code, there's no path that seems to return what you're seeing.

There's not much else I think we can do to help you solve this. It would almost seem like there's an intermediary between you and the server that's rejecting the POST. Are there proxies or anything else that you can think of which might be involved?

[RoCCoCo13]

Hi @sigmavirus24,

Once again, thank you very much for your help and support. I had a look over the ubuntu firewall to check if was it enabled:

ale@quality-agent:~$ sudo ufw status verbose
Status: inactive

And is not there. I tried to push the package using the localhost in the same machine where the pypiserver is runnning but with another user but I face the same result:

jenkins@quality-agent:~/workspace/LIB - Libraries/ediabas-ci$ pipenv run twine upload --repository serverpypi dist/* --verbose
Uploading distributions to http://localhost:8000/
Uploading ediabas-1.0.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33.3k/33.3k [00:00<00:00, 3.04MB/s]
HTTPError: 400 Client Error: Bad Request for url: http://localhost:8000/

This is a task that was done usually by jenkins but I stopped working at some point. I've been looking at the logs of jenkins of the first time I saw it fail around a month ago and I've discover this:

15:47:52  + pipenv run twine upload --repository serverpypi dist/ediabas-1.0.0-py3-none-any.whl dist/ediabas-1.0.0.tar.gz
15:47:54  Uploading distributions to http://quality-agent:8000/simple
15:47:54  Uploading ediabas-1.0.0-py3-none-any.whl
15:47:54  
  0%|          | 0.00/33.1k [00:00<?, ?B/s]
100%|██████████| 33.1k/33.1k [00:00<00:00, 82.2kB/s]
15:47:54  NOTE: Try --verbose to see response content.
15:47:54  HTTPError: 405 Client Error: Method Not Allowed for url: http://quality-agent:8000/simple

I remember then I did an update of the libraries to see if that could change the problem and the 400 started to show up instead the 405.

I also sniffed the packages from my machine with wireshark to see if I could get any additional info, but I don't see any additional data in the HTTP traces:
twine_upload_bad_request.pcapng.gz
The local IP of the machine where pypiserver is 10.100.2.52 and my local machine IP is 10.1.10.27

I changed the url to http://quality-agent:8000/ as suggested to be aligned with the documentation but I really can't see any difference. I don't know what could be the problem when even trying in the same machine with localhost I see the 400 problem with no description.

Thanks for your support