Occasional 500 instead of 404

[pawamoy]

Sometimes I have to restart pypi-server becauses it sends error 500 instead of 404. For example a tool (PDM or pip) is querying pytest on it, and it doesn't have pytest, so it should answer with a 404, but instead I get this in the logs (it's pypi-server's HTML output):

<h1>Critical error while processing request: /simple/pytest/</h1><h2>Error:</h2>
<pre>
FileNotFoundError(2, &#039;No such file or directory&#039;)
</pre>
<h2>Traceback:</h2>
<pre>
Traceback (most recent call last):
File &quot;/home/pawamoy/.local/pipx/venvs/pypi-insiders/lib/python3.11/site-packages/pypiserver/bottle.py&quot;, line 961, in wsgi
out = self._cast(self._handle(environ))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File &quot;/home/pawamoy/.local/pipx/venvs/pypi-insiders/lib/python3.11/site-packages/pypiserver/bottle.py&quot;, line 914, in _cast
out = self.error_handler.get(out.status_code, self.default_error_handler)(out)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File &quot;/home/pawamoy/.local/pipx/venvs/pypi-insiders/lib/python3.11/site-packages/pypiserver/bottle.py&quot;, line 849, in default_error_handler
return tob(template(ERROR_PAGE_TEMPLATE, e=res))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File &quot;/home/pawamoy/.local/pipx/venvs/pypi-insiders/lib/python3.11/site-packages/pypiserver/bottle.py&quot;, line 3628, in template
TEMPLATES[tplid] = adapter(source=tpl, lookup=lookup, **settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File &quot;/home/pawamoy/.local/pipx/venvs/pypi-insiders/lib/python3.11/site-packages/pypiserver/bottle.py&quot;, line 3229, in __init__
self.lookup = [os.path.abspath(x) for x in lookup]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File &quot;/home/pawamoy/.local/pipx/venvs/pypi-insiders/lib/python3.11/site-packages/pypiserver/bottle.py&quot;, line 3229, in &lt;listcomp&gt;
self.lookup = [os.path.abspath(x) for x in lookup]
^^^^^^^^^^^^^^^^^^
File &quot;&lt;frozen posixpath&gt;&quot;, line 404, in abspath
FileNotFoundError: [Errno 2] No such file or directory
</pre>

If I stop and restart pypi-server, the error goes away.

I'm running the server like this, from Python directly (part of another project):

from pypiserver.__main__ import main as serve

serve(
    [
        "run",
        str(dist_dir),
        "--disable-fallback",
        f"-p{port}",
        "-a.",
        "-P.",
        "-vv",
    ],
)

....where dist_dir is an existing directory.

[dee-me-tree-or-love]

Hey @pawamoy! Thanks for reporting this! Sad to hear about the struggle and sorry for the inconvenience. Looking at the error output, it's linked to the internal workings of the bottle which we still depend on as the default backend. (TLDR; I am noticing a few other issues related to this and I think we should aim to pivot away from it in the near future, but for now all apologies!)

Intuitively, it looks like an internal hiccup with error templates, and the way it performs lookups, with our built-in bottle.py distribution to me 🤔

What would be very interesting: could you somehow log the exact routes that are being called on the pypi-server's side by the tools you mention when you encounter a 500 error please? That'd really help investigate further. Also, do you know approximately how often it occurs?

Thanks and sorry once again ✌️

[pawamoy]

Thank you for your help @dee-me-tree-or-love! And no need to apologize 🙇

If/when it happens again, I'll post the routes that are hit by pip/pdm!
It happens very rarely, I've noticed it maybe 4-5 times since I started using pypi-server a few months ago.
So definitely not a big issue. If you are working towards pivoting to something else than bottle, then do not pay attention to this issue, it will likely be resolved by the pivot 🙂

[dee-me-tree-or-love]

Great, thank you so much @pawamoy. Happy to hear this helps, and thank you for your understanding 🌟

If/when it happens again, I'll post the routes that are hit by pip/pdm! It happens very rarely, I've noticed it maybe 4-5 times since I started using pypi-server a few months ago.

Great! This is good to know, and indeed, if you run into this again, please do let me know.
That'd be very interesting to check out 🔍

So definitely not a big issue. If you are working towards pivoting to something else than bottle, then do not pay attention to this issue, it will likely be resolved by the pivot 🙂

Awesome! I will keep this issue in mind still, it only further supports my intuition that it's about time to start moving towards (replacing or supporting) something other than bottle too! 😁