Better query plan viewing experience

[dullbananas]

• Write auto_explain output to log file when running tests
• Don't delete database after running tests so the log can still be seen
• Make some insertions batched so the log is small enough (more insertions should be changed later to improve performance)
• Add SQL comments to some queries so they are labeled and easily searchable in the log
• Disable auto_explain for migrations

[Nutomic]

Good idea! Query batching would also be useful for the sent_activity table`. All federation activities are going through there now, with each activity being written, then read one by one. So it would be good to collect a few seconds of activities and handle them together.

Edit: That would probably solve #4282

[dullbananas]

Instead of showing query plans when running tests, I will add a separate bin crate that inserts a huge amount of stuff and does 1 query specified by cli arg, with auto_explain disabled during insertions.

Small amounts of rows result in query plans that would be inefficient for realistic amounts.

I might keep SQL comments because they can be useful when auto_explain is used in production.

Batched insertions will be kept so they don't take an extremely long time to run.

[dullbananas]

Query batching would also be useful for the sent_activity table`. All federation activities are going through there now, with each activity being written, then read one by one. So it would be good to collect a few seconds of activities and handle them together.

This should not be done with too many rows at a time because one time an error said something like "too large to send to database" when i tried a huge batch insert

[dullbananas]

Things other than PostView can be added later

[phiresky]

The language batch changes are great because they should also reduce the duration the connection is allocated and not available to the pool.

The diesel comment addition is very interesting. Looks like it shouldn't negatively affect anything except for the complexity and maybe the danger of there being a bug in the ast construction for some edge cases?

The post_aggregates_post trigger change I don't fully understand. I'm not sure when the FOR EACH STATEMENT change is relevant since aren't posts always inserted as one post per individual statement anyways? I also see that the change means you had to add more triggers which isn't that great maybe?

[dullbananas]

I'm not sure when the FOR EACH STATEMENT change is relevant since aren't posts always inserted as one post per individual statement anyways?

The db_perf program sets up the posts table way too slowly with FOR EACH ROW triggers

[dessalines]

Nice, this should be much better.

[dessalines]

What is all this for, and what is it doing?

[dullbananas]

I added documentation on ValuesFromSeries to explain it

[Nutomic]

Move this into db_perf crate as its only used there.

[dullbananas]

Done

[Nutomic]

I dont see the purpose of this wrapper, Rust should handle errors from main just fine.

[Nutomic]

Also can you run this code from CI with low number of posts and communities to make sure it doesnt break in the future?

[dullbananas]

The custom error handling makes it possible to show the log path when an error happens. I now changed it to be cleaner and return the correct exit code.

I also made it run in CI.

[dessalines]

Lots of great additions here. @phiresky take a look when you get a chance.