ValueError when null byte in URL

[stefan-hoelzl]

Hello,

today someone tried a bunch of URLs on my online service (using starlette).
On one URL (/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/etc/passwd%00) starlette (specific the StaticFiles Middleware) raised an ValueError: embedded null byte.

This is the full StackTrace

2024-04-03T12:22:52+02:00	ValueError: embedded null byte
2024-04-03T12:22:52+02:00	    st = os.lstat(newpath)
2024-04-03T12:22:52+02:00	  File "/usr/local/lib/python3.10/posixpath.py", line 431, in _joinrealpath
2024-04-03T12:22:52+02:00	    path, ok = _joinrealpath(filename[:0], filename, strict, {})
2024-04-03T12:22:52+02:00	  File "/usr/local/lib/python3.10/posixpath.py", line 396, in realpath
2024-04-03T12:22:52+02:00	    full_path = os.path.realpath(joined_path)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/staticfiles.py", line 164, in lookup_path
2024-04-03T12:22:52+02:00	    result = context.run(func, *args)
2024-04-03T12:22:52+02:00	  File "/packages/anyio/_backends/_asyncio.py", line 851, in run
2024-04-03T12:22:52+02:00	    return await future
2024-04-03T12:22:52+02:00	  File "/packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
2024-04-03T12:22:52+02:00	    return await get_async_backend().run_sync_in_worker_thread(
2024-04-03T12:22:52+02:00	  File "/packages/anyio/to_thread.py", line 56, in run_sync
2024-04-03T12:22:52+02:00	    full_path, stat_result = await anyio.to_thread.run_sync(
2024-04-03T12:22:52+02:00	  File "/packages/starlette/staticfiles.py", line 122, in get_response
2024-04-03T12:22:52+02:00	    response = await super().get_response(path, scope)
2024-04-03T12:22:52+02:00	  File "/packages/api/app.py", line 122, in get_response
2024-04-03T12:22:52+02:00	    response = await self.get_response(path, scope)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/staticfiles.py", line 103, in __call__
2024-04-03T12:22:52+02:00	    await self.app(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/routing.py", line 487, in handle
2024-04-03T12:22:52+02:00	    await route.handle(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/routing.py", line 778, in app
2024-04-03T12:22:52+02:00	    await self.middleware_stack(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/routing.py", line 758, in __call__
2024-04-03T12:22:52+02:00	    await app(scope, receive, sender)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-04-03T12:22:52+02:00	    raise exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-04-03T12:22:52+02:00	    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-04-03T12:22:52+02:00	    return await old_call(app, scope, new_receive, new_send, **kwargs)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
2024-04-03T12:22:52+02:00	    await old_call(self, scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 256, in _sentry_exceptionmiddleware_call
2024-04-03T12:22:52+02:00	    await self.app(scope, receive_or_disconnect, send_no_error)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 151, in coro
2024-04-03T12:22:52+02:00	    raise app_exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 165, in call_next
2024-04-03T12:22:52+02:00	    return await call_next(request)
2024-04-03T12:22:52+02:00	  File "/packages/api/app.py", line 64, in log_request_duration
2024-04-03T12:22:52+02:00	    response = await self.dispatch_func(request, call_next)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 191, in __call__
2024-04-03T12:22:52+02:00	    raise exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/_utils.py", line 93, in collapse_excgroups
2024-04-03T12:22:52+02:00	    self.gen.throw(typ, value, traceback)
2024-04-03T12:22:52+02:00	  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
2024-04-03T12:22:52+02:00	    with collapse_excgroups():
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 189, in __call__
2024-04-03T12:22:52+02:00	    return await old_call(app, scope, new_receive, new_send, **kwargs)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
2024-04-03T12:22:52+02:00	    await self.app(scope, receive, _send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/errors.py", line 164, in __call__
2024-04-03T12:22:52+02:00	    raise exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/errors.py", line 186, in __call__
2024-04-03T12:22:52+02:00	    return await old_call(app, scope, new_receive, new_send, **kwargs)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
2024-04-03T12:22:52+02:00	    await self.middleware_stack(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/applications.py", line 123, in __call__
2024-04-03T12:22:52+02:00	    return await self.app(
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/asgi.py", line 234, in _run_app
2024-04-03T12:22:52+02:00	    raise exc from None
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/asgi.py", line 241, in _run_app
2024-04-03T12:22:52+02:00	    return await self._run_app(scope, receive, send, asgi_version=3)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/asgi.py", line 146, in _run_asgi3
2024-04-03T12:22:52+02:00	    return await middleware(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 364, in _sentry_patched_asgi_app
2024-04-03T12:22:52+02:00	    await super().__call__(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/fastapi/applications.py", line 1054, in __call__
2024-04-03T12:22:52+02:00	    return await self.app(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
2024-04-03T12:22:52+02:00	    result = await app(  # type: ignore[func-returns-value]
2024-04-03T12:22:52+02:00	  File "/packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
2024-04-03T12:22:52+02:00	Traceback (most recent call last):
2024-04-03T12:22:52+02:00	2024-04-03 10:22:52,088 ERROR    uvicorn.error        Exception in ASGI application
2024-04-03T12:22:52+02:00	2024-04-03 10:22:51,990 INFO     uvicorn.access       GET /%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/etc/passwd%00 HTTP/1.1 500 Internal Server Error

I would expect starlette to handle this in someway instead of raising an exception.

Important

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[Zaczero]

For some more context, this happens when filename contains a null byte (such filenames are invalid). I suppose a fix would be to sanitize filenames or to improve exception handling. I prefer the latter since it will not impact the performance of normal requests. It will also be much simpler in code.

[stefanhoelzl]

This seems to be a cpython issue.
os.path.realpath raises an ValueError when a path contains a null byte.
Unlike other path operations e.g. os.path.exists, os.path.islink, os.fspath.

I created a PR to fix this issue python/cpython#117573