Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Csv export improve #7771

Merged
merged 5 commits into from
Jul 18, 2023
Merged

Csv export improve #7771

merged 5 commits into from
Jul 18, 2023

Conversation

nikitosing
Copy link
Member

@nikitosing nikitosing commented Jun 22, 2023

Close #5009

Changelog

  • Change all the paginated quries to query with limit 10_000

  • Get rid of useless DB requests on CSV export

  • Add new index for blocks table:
    plan on goerli after index:

      ```
            Result  (cost=4.13..4.14 rows=1 width=8) (actual time=0.037..0.038 rows=1 loops=1)
                    InitPlan 1 (returns $0)
                      ->  Limit  (cost=0.44..4.13 rows=1 width=8) (actual time=0.035..0.035 rows=0 loops=1)
                            ->  Index Scan Backward using blocks_date on blocks b0  (cost=0.44..169589.80 rows=45868 width=8) (actual time=0.034..0.035 rows=0 loops=1)
                                  Index Cond: ((date("timestamp") = to_date('2019-01-14'::text, 'YYYY-MM-DD'::text)) AND (number IS NOT NULL))
                  Planning Time: 0.195 ms
                  Execution Time: 0.056 ms
      ```
    

    plan on goerli before index:

      ```
      Result  (cost=26.48..26.49 rows=1 width=8) (actual time=4767.006..4767.007 rows=1 loops=1)
        InitPlan 1 (returns $0)
          ->  Limit  (cost=0.43..26.48 rows=1 width=8) (actual time=4767.002..4767.003 rows=1 loops=1)
                ->  Index Scan Backward using blocks_number_index on blocks b0  (cost=0.43..1196276.63 rows=45923 width=8) (actual time=4767.001..4767.001 rows=1 loops=1)
                      Index Cond: (number IS NOT NULL)
                      Filter: (date("timestamp") = to_date('2023-05-14'::text, 'YYYY-MM-DD'::text))
                      Rows Removed by Filter: 170549
      Planning Time: 0.463 ms
      Execution Time: 4767.027 ms
      
      ```
    

Checklist for your Pull Request (PR)

@nikitosing nikitosing force-pushed the csv-export-improve branch 3 times, most recently from 4ed3105 to bc6640d Compare June 23, 2023 20:25
@vbaranov
Copy link
Member

@nikitosing please rebase PR to master branch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nikitosing just for clarification, which query execution is improved with this index?

Copy link
Member Author

@nikitosing nikitosing Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This index improves this request which is used here

@nikitosing nikitosing force-pushed the csv-export-improve branch 2 times, most recently from 4bc6be4 to 6db1f1f Compare July 13, 2023 10:38
@vbaranov vbaranov merged commit 97b512d into master Jul 18, 2023
15 checks passed
@vbaranov vbaranov deleted the csv-export-improve branch July 18, 2023 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CSV export: Error 504 when trying to export transactions
3 participants