Wildcard support in query parameters

[bsilverm]

I haven't found a way to get the wildcard (%) inserted automatically in to a query parameter. This would be useful for cases the query parameter is followed by a LIKE clause. Wrapping the parameter name using the wildcard character within the metadata file (ie - ...where xyz like %:querystring%) does not seem to work. Can this be made possible? Or if not, can the template be extended to provide a tip to the user that they need to insert the wildcard characters themselves?

[simonw]

This is a little bit fiddly, but it's possible to do it using SQLite string concatenation. Here's an example:

select * from facetable
where neighborhood like "%" || :text || "%";

Try it here: https://latest.datasette.io/fixtures-35b6eb6?sql=select+*+from+facetable+where+neighborhood+like+%22%25%22+%7C%7C+%3Atext+%7C%7C+%22%25%22%3B&text=town

[simonw]

I've added this to the unit tests and the documentation.

Docs: http://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries
Canned query demo: https://latest.datasette.io/fixtures/neighborhood_search?text=town

New unit test:

datasette/tests/test_api.py

Lines 333 to 344 in 3683a6b

	def test_canned_query_with_named_parameter(app_client):
	response = app_client.get(
	"/fixtures/neighborhood_search.json?text=town"
	)
	assert [
	["Corktown", "Detroit", "MI"],
	["Downtown", "Los Angeles", "CA"],
	["Downtown", "Detroit", "MI"],
	["Greektown", "Detroit", "MI"],
	["Koreatown", "Los Angeles", "CA"],
	["Mexicantown", "Detroit", "MI"],
	] == response.json["rows"]

datasette/tests/fixtures.py

Lines 145 to 153 in 3683a6b

	'queries': {
	'pragma_cache_size': 'PRAGMA cache_size;',
	'neighborhood_search': '''
	select neighborhood, facet_cities.name, state
	from facetable
	join facet_cities on facetable.city_id = facet_cities.id
	where neighborhood like '%' || :text || '%'
	order by neighborhood;
	'''

[bsilverm]

Perfect, thank you!!

[bsilverm]

One thing I've noticed with this approach is that the query is executed with no parameters which I do not believe was the case previously. In the case the table contains a lot of data, this adds some time executing the query before the user can enter their input and run it with the parameters they want.

[simonw]

This seems to fix that:

select neighborhood, facet_cities.name, state
from facetable
    join facet_cities on facetable.city_id = facet_cities.id
where :text != '' and neighborhood like '%' || :text || '%'
order by neighborhood;

Compare this (with empty string): https://latest.datasette.io/fixtures-cafd088?sql=select+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+%3Atext+%21%3D+%22%22+and+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B

To this: https://latest.datasette.io/fixtures-cafd088?sql=select+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+%3Atext+%21%3D+%22%22+and+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B&text=town

[bsilverm]

Those queries look identical. How can this be prevented if the queries are in a metadata.json file?

[simonw]

I may have misunderstood your problem here.

I understood that the problem is that when using the "%" || :text || "%" construct the first hit to that page (with an empty string for :text) results in a where neighborhood like "%%" query which is slow because it matches every row in the database.

My fix was to add this to the where clause:

where :text != '' and ...

Which means that when you first load the page the where fails to match any rows and you get no results (and hopefully instant loading times assuming SQLite is smart enough to optimize this away). That's why you don't see any rows returned on this page: https://latest.datasette.io/fixtures-cafd088?sql=select+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+%3Atext+%21%3D+%22%22+and+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B

[bsilverm]

Oh I see.. My issue is that the query executes with an empty string prior to the user submitting the parameters. I'll try adding your workaround to some of my queries. Thanks again,