Prerun input query in append_pipes

[albireox]

This PR changes the way that append_pipes runs to avoid the query planner making bad decisions when executing the query.

Currently append_pipes joins the pipelines tables to an existing query, extending its select. But in some cases this could make the query planner execute the query inefficiently. With this PR append_to_queries runs the initial query into a temporary table. Then it joins the temporary table to sdssid_to_pipes. This should never be less efficient than doing the whole things as a single query since joining to sdssid_to_pipes doesn't filter down any of the results from the original query, just adds columns.

With this the route

query/carton-program?name=manual_mwm_tess_ob&name_type=carton&observed=true&release=DR17

goes from taking about 13 seconds to 5 seconds. When run with

query/carton-program?name=manual_mwm_tess_ob&name_type=carton&observed=true&release=DR17&limit=50

it takes ~200 milliseconds. Note that this requires not setting

database.execute_sql('SET LOCAL enable_seqscan=false;')

when we are using the limit. I'm not completely sure why that is the case, but it seems that the LIMIT condition does not work well if only index scans are used, and in that case a sequential scan is fast because it will stop as soon as the LIMIT is hit.

I've also checked other queries that use append_pipes in Zora, but more checking would be good.

Fixes #58

[albireox]

@havok2063 Just a quick ping in case you had not seen that this was ready for review.

[havok2063]

This looks good and is promising. Since the zora search form will be the primary entry point, is it feasible to add the limit option to the /query/main endpoint? If we can easily add it to the end, just before the query.dicts() call, that would be ideal. That way, the limit can be used for any query.

I think this is the only other endpoint right now where it'd be useful.

[havok2063]

How long do these temp tables stick around after creation? We're using append_pipes in most, if not all, of the other endpoints. Will we need to worry about the number and size of these tables hanging around the db?

[albireox]

That's a good point. Temporary tables should only stick around while a connection or a transaction are open. But since valis never closes the connection that could be an issue. I think adding an atomic to the Peewee query to force a transaction should delete the temporary table after the query is done, but I'll check that. Worst case I think I can manually delete the table.

[havok2063]

With the removal of the distinct, does this still return a single item per unique sdss_id? Or now multiple items for a given sdss_id?

[albireox]

Let me check. We may still want the distinct. I may have removed while doing tests to try to understand why the query with LIMIT is so much slower.

[havok2063]

if the distinct is needed but slows things down again, I think we can replace it with a group by. I saw similar performance between using group by and distinct to get the unique ids.

[albireox]

OK, the last commit reinstates the .distinct() in carton_program_search(). It also adds some special settings for that query. Not a perfect solution but probably the best we can do generically for all cartons/programs if the query requires multiple joins. I added some notes to remind ourselves what we did here.

[havok2063]

This looks good.