Improve IAMC endpoint query performance

[meksor]

In use some performance problems have been identified with the iamc endpoints in some production databases.
The culprits seem to be queries like this:

select
	distinct model.name,
	model.created_at,
	model.created_by,
	model.id
from
	model
join run on
	run.model__id = model.id
join iamc_timeseries on
	iamc_timeseries.run__id = run.id
join iamc_datapoint_universal on
	iamc_datapoint_universal.time_series__id = iamc_timeseries.id
order by
	model.id asc

If the iamc_datapoint table is very big this query takes quite a long time. A simple fix i found was to use a subquery instead of a join:

select
	distinct model.name,
	model.created_at,
	model.created_by,
	model.id
from
	model
join run on
	run.model__id = model.id
join iamc_timeseries on
	iamc_timeseries.run__id = run.id
where 
	exists (select * from iamc_datapoint_universal where iamc_datapoint_universal.time_series__id = iamc_timeseries.id  )
order by
	model.id asc

This is about x10 faster on my dataset, but still not ideal. The optimization i went with was the following:
First add some custom logic to the datapoint deletion: When datapoints are deleted, orphaned timeseries rows should be deleted. That way i can completely avoid the join on the iamc_datapoints table and just join the timeseries table:

select
	distinct model.name,
	model.created_at,
	model.created_by,
	model.id
from
	model
join run on
	run.model__id = model.id
join iamc_timeseries on
	iamc_timeseries.run__id = run.id
order by
	model.id asc

Query time is down to 300ms from 10s :)

This PR also adds test for the backend repository count function.

[danielhuppmann]

Looks good to me!

But is there a way to write a test for the new behavior, to ensure that the timeseries-row is indeed deleted (but only if all datapoints are removed).

[meksor]

Good point, i added a check to do_run_datapoints so all the iamc data tests also check if the timeseries get deleted.
Also found a better approach to do the deletion.

[danielhuppmann]

Thanks, looks great!