Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-246] dag_stats endpoint has a terrible query #1610

Closed
wants to merge 1 commit into from
Closed

[AIRFLOW-246] dag_stats endpoint has a terrible query #1610

wants to merge 1 commit into from

Conversation

sekikn
Copy link
Contributor

@sekikn sekikn commented Jun 20, 2016

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:

Testing Done:

  • all 43 core unit tests passed
  • accessed /dag_stats using curl and confirmed
    • it worked with both SQLite and MySQL backends
    • the endpoint returned the same result before and after this modification
    • the endpoint returned x3 faster in turnaround time after this modification, w/ MySQL, 230k rows
    • the dispatched query was as expected (confirmed with MySQL general log)

@sekikn
[AIRFLOW-246] dag_stats endpoint has a terrible query
8bf45cf 
For now, accessing /dag_stats can take a relatively long time
(e.g. over 20 seconds with less than a million rows on some environment).
This patch replaces multiple LEFT OUTER JOINs with INNER JOINs and UNION ALL
and improves that process x3-5 faster.
@codecov-io
Copy link

Current coverage is 63.83%

Merging #1610 into master will increase coverage by 0.02%

@@             master      #1610   diff @@
==========================================
  Files           120        120          
  Lines          8449       8452     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           5391       5395     +4   
+ Misses         3058       3057     -1   
  Partials          0          0

Powered by Codecov. Last updated by 7c0f837...8bf45cf

@asfgit asfgit closed this in a2ed55f Jun 29, 2016
perryao pushed a commit to astronomer/airflow that referenced this pull request Feb 16, 2017
@sekikn @collado-mike
[AIRFLOW-246] Improve dag_stats endpoint query
5e51922 
For now, accessing /dag_stats can take a relatively long time
(e.g. over 20 seconds with less than a million rows on some environment).
This patch replaces multiple LEFT OUTER JOINs with INNER JOINs and UNION ALL
and improves that process by making it 3-5x faster.

Closes apache#1610 from sekikn/AIRFLOW-246
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@sekikn @codecov-io