datatables_view not search if value contains latvian characters

[gatiszeiris]

CKAN version 2.10.1

Describe the bug

A clear and concise description of what the bug is.
Search return results with search value: "datu sis"

Search return results with search value: "datu sistēm"

Steps to reproduce

Steps to reproduce the behavior:

1. dataset resource CSV
   name, number
   "Datu Sistēmas ","4003232323"
2. In datatables_view try serach in field "name" value "Datu Sis" you will get result
3. 2. In datatables_view try serach in field "name" value "Datu Sistēm" you will not get result

Expected behavior

When search text including latvian caracters like ē, ā, ž, ī, ņ etc. search works and return results.

[wardi]

Thank you @gatiszeiris would you check if the same thing happens when using the api, e.g.

ckanapi action datastore_search resource_id=… q="Datu Sistēm"?

[gatiszeiris]

There is result of ckanapi --config="/etc/ckan/default/ckan.ini" action datastore_search resource_id=5f2e918d-5497-47b1-80f8-8201800ef542 q="Datu Sistēm"

{
"_links": {},
"fields": [
{
"id": "_id",
"type": "int"
},
{
"id": "regcode",
"type": "numeric"
},
{
"id": "sepa",
"type": "text"
},
{
"id": "name",
"type": "text"
},
{
"id": "name_before_quotes",
"type": "text"
},
{
"id": "name_in_quotes",
"type": "text"
},
{
"id": "name_after_quotes",
"type": "text"
},
{
"id": "without_quotes",
"type": "numeric"
},
{
"id": "regtype",
"type": "text"
},
{
"id": "regtype_text",
"type": "text"
},
{
"id": "type",
"type": "text"
},
{
"id": "type_text",
"type": "text"
},
{
"id": "registered",
"type": "timestamp"
},
{
"id": "terminated",
"type": "timestamp"
},
{
"id": "closed",
"type": "text"
},
{
"id": "address",
"type": "text"
},
{
"id": "index",
"type": "numeric"
},
{
"id": "addressid",
"type": "numeric"
},
{
"id": "region",
"type": "numeric"
},
{
"id": "city",
"type": "numeric"
},
{
"id": "atvk",
"type": "numeric"
},
{
"id": "reregistration_term",
"type": "text"
},
{
"id": "rank",
"type": "float"
}
],
"include_total": true,
"limit": 100,
"q": "Datu Sistēm",
"records": [],
"records_format": "objects",
"resource_id": "5f2e918d-5497-47b1-80f8-8201800ef542",
"total": 0,
"total_estimation_threshold": null,
"total_was_estimated": false
}

[amercader]

I don't know much about this part of the code, but there is this undocumented config option which might be relevant, and I see that datastore_search supports a language parameter that might be also related.

ckan/ckanext/datastore/backend/postgres.py

Line 570 in 121443c

def _fts_lang(lang: Optional[str] = None) -> str:

[wardi]

There's a language parameter and a ckan.datastore.default_fts_lang config option for datastore_search but the full text search index is created by a trigger

ckan/ckanext/datastore/set_permissions.sql

Lines 76 to 89 in 121443c

	CREATE OR REPLACE FUNCTION populate_full_text_trigger() RETURNS trigger
	AS $body$
	BEGIN
	IF NEW._full_text IS NOT NULL THEN
	RETURN NEW;
	END IF;
	NEW._full_text := (
	SELECT to_tsvector(string_agg(value, ' '))
	FROM json_each_text(row_to_json(NEW.*))
	WHERE key NOT LIKE '\_%');
	RETURN NEW;
	END;
	$body$ LANGUAGE plpgsql;
	ALTER FUNCTION populate_full_text_trigger() OWNER TO {writeuser};

that doesn't include a language