base_url is omitted in JSON and CSV views

[phubbard]

I have a datasette deployment, using Apache2 to reverse proxy:

   ProxyPass /ged http://thor.phfactor.net:8001
   ProxyPreserveHost On

In settings.json I have

{
    "base_url": "/ged/",
    "trace_debug": 1,
    "template_debug": 1
}

and datasette works correctly. However, if you view a query and then click on the 'This data as json, CSV' both links omit the base_url prefix and are therefore 404.

[simonw]

base_url has been a source of so many bugs like this! I often find them quite hard to replicate, likely because I haven't made myself a good Apache mod_proxy testing environment yet.

[simonw]

Having a live demo running on Cloud Run that proxies through Apache and uses base_url would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and mod_proxy in the same Docker container as Datasette?

[simonw]

I now have a Dockerfile in #1521 (comment) that I can use to run a local Apache 2 with mod_proxy to investigate this class of bugs!

[simonw]

Bug confirmed:

[simonw]

The relevant test is this one:

datasette/tests/test_html.py

Lines 1608 to 1649 in 3025505

	@pytest.mark.parametrize(
	"path",
	[
	"/",
	"/fixtures",
	"/fixtures/compound_three_primary_keys",
	"/fixtures/compound_three_primary_keys/a,a,a",
	"/fixtures/paginated_view",
	"/fixtures/facetable",
	],
	)
	def test_base_url_config(app_client_base_url_prefix, path):
	client = app_client_base_url_prefix
	response = client.get("/prefix/" + path.lstrip("/"))
	soup = Soup(response.body, "html.parser")
	for el in soup.findAll(["a", "link", "script"]):
	if "href" in el.attrs:
	href = el["href"]
	elif "src" in el.attrs:
	href = el["src"]
	else:
	continue # Could be a <script>...</script>
	if (
	not href.startswith("#")
	and href
	not in {
	"https://datasette.io/",
	"https://github.com/simonw/datasette",
	"https://github.com/simonw/datasette/blob/main/LICENSE",
	"https://github.com/simonw/datasette/blob/main/tests/fixtures.py",
	"/login-as-root", # Only used for the latest.datasette.io demo
	}
	and not href.startswith("https://plugin-example.datasette.io/")
	):
	# If this has been made absolute it may start http://localhost/
	if href.startswith("http://localhost/"):
	href = href[len("http://localost/") :]
	assert href.startswith("/prefix/"), {
	"path": path,
	"href_or_src": href,
	"element_parent": str(el.parent),
	}

I modified that test to add "/fixtures/facetable?sql=select+1" as one of the tested paths, and dropped in an assert False to pause it in the debugger:

    @pytest.mark.parametrize(
        "path",
        [
            "/",
            "/fixtures",
            "/fixtures/compound_three_primary_keys",
            "/fixtures/compound_three_primary_keys/a,a,a",
            "/fixtures/paginated_view",
            "/fixtures/facetable",
            "/fixtures?sql=select+1",
        ],
    )
    def test_base_url_config(app_client_base_url_prefix, path):
        client = app_client_base_url_prefix
        response = client.get("/prefix/" + path.lstrip("/"))
        soup = Soup(response.body, "html.parser")
        if path == "/fixtures?sql=select+1":
>           assert False
E           assert False

BUT... in the debugger:

(Pdb) print(soup)
...
<p class="export-links">This data as
  <a href="/prefix/fixtures.json?sql=select+1">json</a>,
  <a href="/prefix/fixtures.testall?sql=select+1">testall</a>,
  <a href="/prefix/fixtures.testnone?sql=select+1">testnone</a>,
  <a href="/prefix/fixtures.testresponse?sql=select+1">testresponse</a>,
  <a href="/prefix/fixtures.csv?sql=select+1&amp;_size=max">CSV</a></p>

Those all have the correct prefix! But that's not what I'm seeing in my Dockerfile reproduction of the issue.

Something very weird is going on here.

[simonw]

I added template_debug in the Dockerfile:

datasette fixtures.db --setting template_debug 1 --setting base_url "/foo/bar/" -p 9000 &\n\

And then hit http://localhost:5000/foo/bar/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1 to view the template context - and it showed the bug, output edited to just show relevant keys:

{
    "edit_sql_url": "/foo/bar/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1",
    "settings": {
        "force_https_urls": false,
        "template_debug": true,
        "trace_debug": false,
        "base_url": "/foo/bar/"
    },
    "show_hide_link": "/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1&_context=1&_hide_sql=1",
    "show_hide_text": "hide",
    "show_hide_hidden": "",
    "renderers": {
        "json": "/fixtures.json?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1"
    },
    "url_csv": "/fixtures.csv?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1&_size=max",
    "url_csv_path": "/fixtures.csv",
    "base_url": "/foo/bar/"
}

This is so strange. edit_sql_url and base_url are correct, but show_hide_link and url_csv and renderers.json are not.

And it's really strange that the bug doesn't show up in the tests.

[simonw]

Here's the code that generates edit_sql_url correctly:

datasette/datasette/views/database.py

Lines 416 to 420 in 8584993

	if allow_execute_sql and is_validated_sql and ":_" not in sql:
	edit_sql_url = (
	self.ds.urls.database(database)
	+ "?"
	+ urlencode(

And here's the code for show_hide_link:

datasette/datasette/views/database.py

Lines 432 to 433 in 8584993

	if bool(params.get("_show_sql")):
	show_hide_link = path_with_removed_args(request, {"_show_sql"})

And for url_csv:

datasette/datasette/views/base.py

Lines 600 to 602 in 8584993

	url_csv = path_with_format(
	request=request, format="csv", extra_qs=url_csv_args
	)

[simonw]

The implementations of path_with_removed_args and path_with_format:

datasette/datasette/utils/__init__.py

Lines 228 to 254 in 8584993

	def path_with_removed_args(request, args, path=None):
	query_string = request.query_string
	if path is None:
	path = request.path
	else:
	if "?" in path:
	bits = path.split("?", 1)
	path, query_string = bits
	# args can be a dict or a set
	current = []
	if isinstance(args, set):
	def should_remove(key, value):
	return key in args
	elif isinstance(args, dict):
	# Must match key AND value
	def should_remove(key, value):
	return args.get(key) == value
	for key, value in urllib.parse.parse_qsl(query_string):
	if not should_remove(key, value):
	current.append((key, value))
	query_string = urllib.parse.urlencode(current)
	if query_string:
	query_string = f"?{query_string}"
	return path + query_string

datasette/datasette/utils/__init__.py

Lines 710 to 729 in 8584993

	def path_with_format(
	*, request=None, path=None, format=None, extra_qs=None, replace_format=None
	):
	qs = extra_qs or {}
	path = request.path if request else path
	if replace_format and path.endswith(f".{replace_format}"):
	path = path[: -(1 + len(replace_format))]
	if "." in path:
	qs["_format"] = format
	else:
	path = f"{path}.{format}"
	if qs:
	extra = urllib.parse.urlencode(sorted(qs.items()))
	if request and request.query_string:
	path = f"{path}?{request.query_string}&{extra}"
	else:
	path = f"{path}?{extra}"
	elif request and request.query_string:
	path = f"{path}?{request.query_string}"
	return path

[simonw]

In the ?_context= debug view the request looks like this:

    "request": "<datasette.utils.asgi.Request object at 0x7faf9fe06200>",

I'm going to add a repr() to it such that it's a bit more useful.

[simonw]

Modified my Dockerfile to do this:

RUN pip install https://github.com/simonw/datasette/archive/ff0dd4da38d48c2fa9250ecf336002c9ed724e36.zip

And now the request in that debug ?_context=1 looks like this:

    "request": "<asgi.Request method=\"GET\" url=\"http://localhost:9000/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1\">"

That explains the bug - that request doesn't maintain the original path prefix of http://localhost:5000/foo/bar/fixtures?sql= (also it's been rewritten to localhost:9000 instead of localhost:5000).