Perf/Postgres Index Optimization and Benchmarking

[mohanish2504]

Fixes: #863
/claim #863

Optimization Process

Initial Analysis

• Description: Visiting all queries.
• Tools Used: EXPLAIN ANALYZE, generate_series (for test data generation) both directly in PGAdmin
• Findings:
   1. retreival from schema_definitions uses version and tenant_id as filter. For this query by default postgres use sequential scan.

Index Optimization

• Optimization Strategies: Detail the specific index optimizations implemented.

  • to add index on version of schema_definitions

• Implementation Details:

  • SQL commands for findings 1 => 24c1413

Benchmarking Setup

Environment Details

• Postgres Version: 16
• Operating System: Linux/Ubuntu: 20.04
• Hardware Specifications: [Hardware details]

Benchmarking Methodology

• Queries Tested: All the queries from internal/engines folder.

• Query 1: SELECT name, serialized_definition, version FROM public.schema_definitions WHERE name= :name AND version= :version AND tenant_id= :tenant_id' LIMIT 1;

Benchmark Results

Before Optimization

• Query 1:
  • Average Execution Time: No. of rows
    * 1k : 0.230 ms
    * 10k: 2.9 ms

After Optimization

• Query 1:
  • Average Execution Time: No. of rows
    * 1k : 0.040 ms
    * 10k: 0.060 ms

Analysis and Conclusions

• Performance Improvements: Query speed improved by huge margin.

• Recommendations&Conclusions:

  1. I have added data till 1 million entries. All indexes are already properly placed for concerned tables relation_tuples & attributes.
  2. For table schema I have added new index which covers version for one of the query where name is not used. i.e ReadSchema.
  3. Reads are on point in any case/query. Tried to optimize storage with some combination of indexes, but in any case total_size acheived indexes was same as previous ones.
  4. I don't fill need of adding created_tx_id ,'expired_tx_id' in index since both are covered in Unique Constraint.

• Adttional notes:

  1. Pagination can be added in EntityFilter and SubjectFilter
  2. Some querie in internal/storage can use index optimization since this was out of scope didn't add as commit.
  3. adding brin index on timestamp of transaction. But since query time is every 200 hours and also previous transactions gets deleted, this can be ignored.

Happy New Year!!!!

[tolgaozen]

Can you create and drop indexes using 'CONCURRENTLY'?

[mohanish2504]

@tolgaozen yes for sure

[mohanish2504]

@tolgaozen done

Test:

2024/01/02 18:59:12 OK 20230117143902_initial.sql (18.88ms)
2024/01/02 18:59:12 OK 20230117145140_multi_tenant.sql (11.81ms)
2024/01/02 18:59:12 OK 20230118185622_index_optimization.sql (10.94ms)
2024/01/02 18:59:12 OK 20230729145034_attributes.sql (7.92ms)
2024/01/02 18:59:12 OK 20231122120124_bundles.sql (5.03ms)
2024/01/02 18:59:12 OK 20231230174418_schema_version_index.sql (3.76ms)
2024/01/02 18:59:12 goose: successfully migrated database to version: 20231230174418