fix(api): mitigate slow queries

[usu]

We have a bunch of very slow queries, mainly on the /content_nodes endpoint.

Normally, it shouldn't make much difference whether to use WHERE or ON for join statements and the DB would optimize them anyway. But seems, our queries are too complex and this optimization doesn't work.

Hope I caught the biggest offenders. Appreciate if we can test & deploy this as soon as possible.

Timing tests on my local machine (with prod data dump), before PR

Timing tests on my local machine (with pod data dump), after PR

How to check for slow queries

(documentation for the next time)
I followed this guide: https://www.crunchydata.com/blog/tentative-smarter-query-optimization-in-postgres-starts-with-pg_stat_statements

CREATE EXTENSION pg_stat_statements;

And then use the following query:

SELECT
  (total_exec_time / 1000 / 60) as total_min,
  mean_exec_time as avg_ms,
  calls,
  query
FROM pg_stat_statements
ORDER BY 1 DESC
LIMIT 500;

The 4 top entries are all from the /content_nodes endpoint and with average runtime between 8 and 40 seconds (!).

[github-actions]

⛔ Feature branch deployment currently inactive.

If the PR is still open, you can add the deploy! label to this PR to trigger a feature branch deployment.

[carlobeltrame]

I will add a test, we don't yet have good API tests for reading lists of content nodes

[carlobeltrame]

The newly added Api Tests pass with and without the code change in this PR, I checked.

[manuelmeister]

Thanks for investigating! Worked in my manual test on mobile. Did not look at the code though.

[usu]

Normally, it shouldn't make much difference whether to use WHERE or ON for join statements and the DB would optimize them anyway. But seems, our queries are too complex and this optimization doesn't work.

I found out a bit more specifically on this topic: There's indeed a configurable threshold named join_collapse_limit (default set to 8). Once the number of joins exceed this limit, postgresql wouldn't try to reshuffle join statements anymore and they are just executed in the order given by the query. Theoretically, this threshold could be increased, as a side effect, the cost/effort spent for optimizing queries increases. This could again be countered by adjusting geqo_threshold.

Generally, our queries have a lot of joins - for the queries involved for content nodes I counted up to 14 joins. This comes from the eager loading strategy + from the various filters applied. It could make sense to look into this to see if our optimal settings would be different. On the other hand, as long as we can easily optimize the queries from PHP side, I'd prefer not to tinker too much with the DB settings.

Further reading:
https://stackoverflow.com/questions/22339836/postgresql-join-collapse-limit-and-time-for-query-planning
https://www.postgresql.org/docs/15/runtime-config-query.html