perf(core): Accelerate common execution queries

[ivov]

https://linear.app/n8n/issue/PAY-1609

[krynble]

The way I see it is that we just need 4 indices, and the image below outlines the changes I'm proposing:

The reason for each index has been described in the ticket, but in the end we want all 4 db systems to have only those 4 indices, wdyt?

[ivov]

Thanks, from the story I was under the impression we only wanted to touch these, rather than all of them.

[netroy]

this should only be on the DropIndex class

export class DropIndex extends IndexOperation {
	constructor(
		tableName: string,
		columnNames: string[],
		tablePrefix: string,
		queryRunner: QueryRunner,
		customIndexName?: string,
		protected skipIfMissing = false,
	) {
		super(tableName, columnNames, tablePrefix, queryRunner, customIndexName);
	}

	async execute(queryRunner: QueryRunner) {
		return await queryRunner
			.dropIndex(this.fullTableName, this.customIndexName ?? this.fullIndexName)
			.catch((error) => {
				if (error instanceof Error && error.message.includes('not found') && this.skipIfMissing) {
					return;
				}
				throw error;
			});
	}
}

[netroy]

could we somehow use the IF EXISTS parameter for DBs that support it?

[ivov]

this should only be on the DropIndex class

dd17796

could we somehow use the IF EXISTS parameter for DBs that support it?

We could call queryRunner.query with handcrafted SQL and then fall back to ignoring the error in a catch clause if IF EXISTS is unsupported, but I think the current approach is simpler. Let me know otherwise!

[tomi]

Couple comments, mainly about the structure of the indexes

[tomi]

Are we certain that this is the message returned by all DBs? It's unfortunate that we need to rely on this type of hack just because MySQL doesn't support IF EXISTS. We could also also use the IF EXISTS for those that support it (SQLite, PG) and rely on this for MySQL. WDYT?

[ivov]

They all throw the same error TypeORMError: Supplied index {indexId} was not found in table {table}.

Re: handling differently depending on DB, I replied about this above - my reasoning is it's simpler to have one way of handling this. Once we remove MySQL this will no longer be an issue.

[tomi]

Alright, makes sense 👍 Maybe we can be more specific and expect a TypeORMError instead of any generic Error?

[tomi]

Have we actually done any investigation how much these indexes help and how much they impact write performance? Having multiple indexes (especially composite) is not cheap as they need to be updated with every insert, update and delete.

Looking at the fields, the (status, startedAt) index can probably also serve the ExecutionRepository.findManyByRangeQuery query as the only column missing is workflowId. We could verify this.

For the last two indexes we could create a partial index using WHERE clause to include only rows that have waitTill and stoppedAt as NOT NULL and deletedAt as IS NULL. That way the index size is more limited and performs better as that's how we use them in the queries. Again, MySQL doesn't support partial indexes so in that case we could create the indexes without the WHERE clause.

[ivov]

Have we actually done any investigation how much these indexes help and how much they impact write performance? Having multiple indexes (especially composite) is not cheap as they need to be updated with every insert, update and delete. Looking at the fields, the (status, startedAt) index can probably also serve the ExecutionRepository.findManyByRangeQuery query as the only column missing is workflowId. We could verify this.

I think Omar did but that's now lost. If you have time tomorrow, let's pair on this and you can show me how?

For the last two indexes we could create a partial index using WHERE clause to include only rows that have waitTill and stoppedAt as NOT NULL and deletedAt as IS NULL. That way the index size is more limited and performs better as that's how we use them in the queries. Again, MySQL doesn't support partial indexes so in that case we could create the indexes without the WHERE clause.

Will do tomorrow 👍🏻

[tomi]

Sure, let's do it 👍 Basically the steps are:

1. Have a DB with relevant amount of data
2. Create the indexes
3. Run different queries to see how they perform. Run EXPLAIN ANALYZE to see if the DBMS is utilizing the indexes
4. Compare results

[tomi]

Can we give explicit names to these so they are better identifiable and modifying them later on is easier

[ivov]

These are the names, I find them clear enough, what would you suggest?

[tomi]

I meant name them explicitly, so it's obvious from the code what the index names will be, instead of letting the ORM generate the names. But those are also fine as long as they are stable across all different DBs