Skip to content

Comments

Move TestDagFileProcessorQueriesCount to quarantine#9119

Merged
mik-laj merged 1 commit intoapache:masterfrom
PolideaInternal:add-quarantine
Jun 3, 2020
Merged

Move TestDagFileProcessorQueriesCount to quarantine#9119
mik-laj merged 1 commit intoapache:masterfrom
PolideaInternal:add-quarantine

Conversation

@mik-laj
Copy link
Member

@mik-laj mik-laj commented Jun 3, 2020

I was able to repeat this problem locally, but I still need some time to investigate the cause.It may be related to this line:

if run.execution_date > timezone.utcnow() and not dag.allow_future_exec_dates:

When I do the following changes in this file, this issue doesn't happent.

        # update the state of the previously active dag runs
        active_dag_runs = 0
        task_instances_list = []
-       for run in dag_runs:
+       for run in sorted(dag_runs, key=lambda d: d.execution_date):
            self.log.info("Examining DAG run %s", run)
            # don't consider runs that are executed in the future unless
            # specified by config and schedule_interval is None

I am not sure if this is a good solution.


Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Jun 3, 2020
@turbaszek
Copy link
Member

@mik-laj this "time-related" problem is something I observed too

@mik-laj mik-laj requested review from ashb, kaxil, potiuk and turbaszek June 3, 2020 14:04
@mik-laj mik-laj merged commit 2b45d8f into apache:master Jun 3, 2020
@mik-laj mik-laj deleted the add-quarantine branch June 3, 2020 14:23
@ashb
Copy link
Member

ashb commented Jun 3, 2020

Seems odd at first glance that sorting a list in python would change the number of queries... 🤔 I wonder what's going on here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants