Support static assets where file length may change, e.g. logs

[broccolihighkicks]

This is a bit of an oxymoron.

I am serving a log.txt file for a background process using the Datasette --static CLI. This is useful as I can observe a background process from the web UI to see any errors that occur (instead of spelunking the logs via docker exec/ssh etc).

I get this error, which I think is because Datasette assumes that the size of the content does not change (but appending new log lines means the content length changes).

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 1181, in route_path
    response = await view(request, send)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 305, in inner_static
    await asgi_send_file(send, full_path, chunk_size=chunk_size)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 280, in asgi_send_file
    await send(
  File "/usr/local/lib/python3.9/site-packages/asgi_csrf.py", line 104, in wrapped_send
    await send(event)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 460, in send
    output = self.conn.send(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 501, in send_with_data_passthrough
    writer(event, data_list.append)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 58, in __call__
    self.send_data(event.data, write)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 78, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 1181, in route_path
    response = await view(request, send)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 305, in inner_static
    await asgi_send_file(send, full_path, chunk_size=chunk_size)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 280, in asgi_send_file
    await send(
  File "/usr/local/lib/python3.9/site-packages/asgi_csrf.py", line 104, in wrapped_send
    await send(event)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 460, in send
    output = self.conn.send(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 501, in send_with_data_passthrough
    writer(event, data_list.append)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 58, in __call__
    self.send_data(event.data, write)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 78, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length

Thanks, I am finding Datasette very useful.

[simonw]

Hah, this is certainly unexpected.

It looks like this is the code in question:

datasette/datasette/utils/asgi.py

Lines 259 to 266 in a6ff123

	async def asgi_send_file(
	send, filepath, filename=None, content_type=None, chunk_size=4096, headers=None
	):
	headers = headers or {}
	if filename:
	headers["content-disposition"] = f'attachment; filename="{filename}"'
	first = True
	headers["content-length"] = str((await aiofiles.os.stat(str(filepath))).st_size)

You're right: it assumes that the file it is serving won't change length while it is serving it.

[simonw]

The reason I implemented it like this was to support things like the curl progress bar if users decide to serve up large files using the --static mechanism.

Here's the code that hooks it up to the URL resolver:

datasette/datasette/app.py

Lines 1001 to 1005 in 458f03a

	add_route(
	asgi_static(app_root / "datasette" / "static"), r"/-/static/(?P<path>.*)$"
	)
	for path, dirname in self.static_mounts:
	add_route(asgi_static(dirname), r"/" + path + "/(?P<path>.*)$")

Which uses this function:

datasette/datasette/utils/asgi.py

Lines 285 to 310 in a6ff123

	def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
	root_path = Path(root_path)
	async def inner_static(request, send):
	path = request.scope["url_route"]["kwargs"]["path"]
	try:
	full_path = (root_path / path).resolve().absolute()
	except FileNotFoundError:
	await asgi_send_html(send, "404: Directory not found", 404)
	return
	if full_path.is_dir():
	await asgi_send_html(send, "403: Directory listing is not allowed", 403)
	return
	# Ensure full_path is within root_path to avoid weird "../" tricks
	try:
	full_path.relative_to(root_path.resolve())
	except ValueError:
	await asgi_send_html(send, "404: Path not inside root path", 404)
	return
	try:
	await asgi_send_file(send, full_path, chunk_size=chunk_size)
	except FileNotFoundError:
	await asgi_send_html(send, "404: File not found", 404)
	return
	return inner_static

One option here would be to support a workaround that looks something like this:

http://localhost:8001/my-static/log.txt?_unknown_size=1`

The URL routing code could then look out for that ?_unknown_size=1 option and, if it's present, omit the content-length header entirely.

It's a bit of a cludge, but it would be pretty straight-forward to implement.

Would that work for you @broccolihighkicks?