Orion v2 / Orion v1 comparison tests and benchmarks

[Lezek123]

This issue covers an idea to test the performance and data consistency of Orion v2 by executing a set of Orion v2 queries/mutations and corresponding Orion v1 / Query node queries/mutations and comparing the results.

Things to prepare:

Queries:

• Full state queries: A query/set of queries that cover as much of the Orion v2 state as possible (ideally - queries for all existing entities w/ all possible fields) + a corresponding query/set of queries that fetch the same data from the QN / Orion v1 (taking into account schema differences): partially done (Estimated time left: ~4h)
• Atlas queries & mutations: Orion v2 version of all current Atlas queries&mutations: done

Scripts:

• Benchmarking script: A script that executes an Orion v2 query/mutation and a corresponding Orion v1 query/mutation (for all queries listed under Queries) multiple times, against provided Orion v2 and Orion v1 graphql server endpoints and calculates the average time it takes to get the results for each query/mutation for both Orion v2 and Orion v1 (optionally also compares those time differences and scores the performance of both). (Estimated time: ~4h)
  • Extra feature: Also supports executing multiple queries/mutations in parallel to simulate more realistic conditions and make sure table locking doesn't drastically decrease performance of Orion v2 (Estimated time: 6h / 1 day)
• Data comparison script: based on the result of queries listed under Queries, compares the data from two sources: Orion v1 (or Query Node) and Orion v2. The script must take into account schema differences and other non-backward-compatible changes as described in Orion v2's CHANGELOG (Estimated time: ~2 days)

Total time to prepare testing setup: ~4-5 days

Tests to run:

Network tests

1. Run ephesus network tests from https://github.com/Joystream/joystream/tree/ephesus (set PERSIST=true to keep the chain alive once tests are done)
2. Run Orion v1 and Orion v2 instances working w/ the test chain (once the tests are done).
3. Use the Benchmarking script to compare performance between Orion v1 and Orion v2.
4. Use the Data comparison script to compare query results between Orion v1 and Orion v2.

Joystream mainnet chain tests:

1. Run Orion v1 and Orion v2 instance processing the current Joystream mainnet chain
2. Use the Benchmarking script to compare performance between Orion v1 and Orion v2.
3. Use the Data comparison script to compare query results between Orion v1 and Orion v2.

Staging networks tests

Optionally: Run tests similar to Joystream mainnet but against other networks like atlas-dev / atlas-next etc.

[Lezek123]

Status update on the work related to this issue:

• State comparison script

  • Has been implemented in: https://github.com/Lezek123/orion/blob/orion-v2/src/tests/compareState.ts
  • Only compares the results of state queries (v1, v2), which cover all the entities usually queried by Atlas. Does not work with all existing Atlas queries, as it would require too much work to prepare such script given the differences between Orion v1 and Orion v2.
  • has been now successfully executed once to compare Orion v2 vs Orion v1 (QN) mainnet chain state
  • It is advised to run the script again using the final version of Orion v2 before the release

• Benchmarking script:

  • Has been fully implemented, but w/o the Extra feature scope mentioned above: https://github.com/Lezek123/orion/tree/orion-v2/src/tests/benchmarks
  • Two rounds of benchmarking have been executed using the script and the results were published in Orion v2 benchmarking results #77