Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-6239] Filter dags return by last_dagruns #6804

Merged
merged 1 commit into from
Dec 13, 2019

Conversation

robinedwards
Copy link
Contributor

Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.

Jira

Description

  • Here are some details about my PR, including screenshots of any UI changes:

The last_dagruns endpoint returns all dags by default. This can result in an extremely slow response time when you have a lot of dags (In our case 1500+ takes 8 seconds).

The accompanying pull request adds a dag_ids get parameter to the last_dagruns end point which is populated by the dags present on the page.

Please see related and merged issue for task_stats: https://issues.apache.org/jira/browse/AIRFLOW-6095

Tests

  • My PR adds the following unit tests

(see test_views)

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

NA

Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.
tests/www/test_views.py Show resolved Hide resolved
@codecov-io
Copy link

codecov-io commented Dec 12, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@0f9983f). Click here to learn what that means.
The diff coverage is 88.88%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #6804   +/-   ##
=========================================
  Coverage          ?   84.59%           
=========================================
  Files             ?      672           
  Lines             ?    38225           
  Branches          ?        0           
=========================================
  Hits              ?    32338           
  Misses            ?     5887           
  Partials          ?        0
Impacted Files Coverage Δ
airflow/www/views.py 76.08% <88.88%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f9983f...07b3f7b. Read the comment docs.

@ashb ashb merged commit bdcd7cd into apache:master Dec 13, 2019
ashb pushed a commit to ashb/airflow that referenced this pull request Dec 18, 2019
Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.

(cherry picked from commit bdcd7cd)
ashb pushed a commit that referenced this pull request Dec 18, 2019
Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.

(cherry picked from commit bdcd7cd)
ashb pushed a commit that referenced this pull request Dec 19, 2019
Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.

(cherry picked from commit bdcd7cd)
kaxil pushed a commit that referenced this pull request Dec 19, 2019
Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.

(cherry picked from commit bdcd7cd)
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants