0.5.7

This release makes the evaluator (proxied) path much faster for restricted databases, fixes two PostgreSQL errors with boolean columns, and adds select_normalizer so that select values have the same types on every path.

What's Changed

Faster evaluation on raw data — @dsblank (#7)

• RawDataReader (opt-in). run_query and evaluate_where can read records, and the records they refer to, as raw data instead of building a Gramps object for each one. For a guest in gramps-web-api using RestrictedDb, this makes /query 1.3–13× faster on a 100k-person tree. The default ObjectReader reads through the database's object getters as before, so Gramps proxies and the addons work the same and run at the same speed.
• run_query_with_total returns (rows, total), with the number of matches counted in the same pass. The total ignores after and limit, like compile_count_query. Before, getting a total needed a second full pass.
• run_query no longer goes through GenericFilter. It takes a reader= argument and turns only the final page into objects.
• Related records are cached per query in a 10,000-entry LRU cache.

Behavior change: a related record that can't be read (hidden or missing) is no longer counted by any(...) or len(...). This now matches the SQL path. Before, any(...) and len(...) without a condition still counted a related record that the database hides.

Guest query, 100k-person tree	Before	After	Speedup
gender == Person.FEMALE	28.1 s	6.7 s	4.2×
primary_name.first_name == 'Mary'	27.8 s	4.8 s	5.8×
birth.date.sortval >= Date('Jan 1, 1950')	33.1 s	12.2 s	2.7×
Event like(place.title, '%Texas%')	53.9 s	4.1 s	13.2×

Boolean columns on PostgreSQL — @dsblank (#8)

• Booleans across a relationship. birth.private == False, birth.place.private, father.private and in [...] raised operator does not exist: integer = boolean. They now work, as a top-level private == False already did.
• Keyset cursors. An after cursor holding True/False for a sort column such as private raised integer > boolean. It now works.

These fixes also apply wherever where expressions are compiled to SQL, such as gramps-web-api's MatchesQuery filter rule and the GOQLFilter addon.

select_normalizer — @dsblank (#8)

select_normalizer(spec, select) returns a function that converts a row from compile_query to the values the evaluator path returns:

Selected value	SQLite	PostgreSQL	Normalized
flat boolean (private, birth.private)	1	1	True
JSON boolean (primary_name.private)	1	'true'	True
JSON number (birth.date.sortval)	2389036	'2389036'	2389036
object or list (birth.date)	JSON text	JSON text	parsed
string (primary_name.first_name)	'123'	'123'	'123'

Column types are looked up once when the normalizer is built. Values that already have the right type pass through unchanged.

Tests

811 tests. New tests check that SQLite, PostgreSQL and the evaluator agree on related boolean columns, boolean keyset cursors, and select value types. They also check that the total ignores limit and after, that private parts of visible records are matched as cleaned values with both readers, and that the LRU cache evicts in the right order. The PostgreSQL test helper now creates INTEGER columns as INTEGER. Copying every column as TEXT had hidden the boolean bugs.

Contributors

• @dsblank

Full Changelog: 0.5.6...0.5.7