Remove side effects from TestCliDags and TestCliTasks

[turbaszek]

Apache Airflow version:
2.0.0dev

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• breeze used locally, Python 3.8

What happened:
It seems that TestCliDags and TestCliTasks depends on side effects. Both works on CI (see builds for #12704) but do not work locally when run separately from all other tests.

Running pytest tests/cli/commands/test_dag_command.py:

FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_backfill - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it failed to p...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_dag_runs - airflow.exceptions.AirflowException: Dag id example_bash_operator not found
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_dags - AssertionError: 'airflow/example_dags/example_complex.py' not found in "- dag_id: clear_subdag_test_dag\n  filepath: test_clear_...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_jobs_with_args - airflow.exceptions.AirflowException: Dag id example_bash_operator not found
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_report - AssertionError: 'airflow/example_dags/example_complex.py' not found in '[{"file": "/test_example_bash_operator.py", "duration": "0:...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_dag_state - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it failed to ...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_next_execution - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it faile...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_show_dag_dave - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it failed...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_show_dag_imgcat - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it fail...
FAILED tests/cli/commands/test_dag_command.py::TestCliDags::test_show_dag_print - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it faile...
==================================================================================== 10 failed, 9 passed, 4 warnings in 19.64s =====================================================================================

and running pytest tests/cli/commands/test_task_command.py

============================================================================================= short test summary info ==============================================================================================
SKIPPED [1] tests/conftest.py:311: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_local_run>
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_list_tasks - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it fai...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it failed to ...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_0__ignore_all_dependencies - AssertionError: "Option --raw does not work with some of the other options on this com...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_1__ignore_depends_on_past - AssertionError: "Option --raw does not work with some of the other options on this comm...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_2__ignore_dependencies - AssertionError: "Option --raw does not work with some of the other options on this command...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_3__force - AssertionError: "Option --raw does not work with some of the other options on this command." does not ma...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_mutually_exclusive - AssertionError: "Option --raw and --local are mutually exclusive." does not match "dag_id could not be found: exa...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_test - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it failed to...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_test_with_env_vars - airflow.exceptions.AirflowException: dag_id could not be found: example_passing_params_via_test_command. Either the d...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_test_with_params - airflow.exceptions.AirflowException: dag_id could not be found: example_passing_params_via_test_command. Either the dag...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_parentdag_downstream_clear - airflow.exceptions.AirflowException: dag_id could not be found: example_subdag_operator.section-1. Either the dag...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_subdag_clear - airflow.exceptions.AirflowException: dag_id could not be found: example_subdag_operator. Either the dag did not exist or it fai...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_task_state - airflow.exceptions.AirflowException: dag_id could not be found: example_bash_operator. Either the dag did not exist or it failed ...
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_task_states_for_dag_run - KeyError: 'example_python_operator'
FAILED tests/cli/commands/test_task_command.py::TestCliTasks::test_test - airflow.exceptions.AirflowException: dag_id could not be found: example_python_operator. Either the dag did not exist or it failed to p...
FAILED tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_logging_with_run_task - AssertionError: 2 != 1
===================================================================================== 16 failed, 6 passed, 1 skipped in 39.03s =====================================================================================

What you expected to happen:

Each test case should be runnable on its own.

[eladkal]

Seems to be fixed:
pytest tests/cli/commands/test_dag_command.py:

tests/cli/commands/test_dag_command.py::TestCliDags::test_backfill PASSED                                                                                                                          [  5%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_backfill_depends_on_past PASSED                                                                                                      [ 10%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_backfill_depends_on_past_backwards PASSED                                                                                            [ 15%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_dag_runs PASSED                                                                                                                 [ 20%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_dags PASSED                                                                                                                     [ 25%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_jobs PASSED                                                                                                                     [ 30%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_list_jobs_with_args PASSED                                                                                                           [ 35%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_cli_report PASSED                                                                                                                        [ 40%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_dag_state PASSED                                                                                                                         [ 45%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_dag_test PASSED                                                                                                                          [ 50%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_dag_test_show_dag PASSED                                                                                                                 [ 55%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_delete_dag PASSED                                                                                                                        [ 60%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_delete_dag_existing_file PASSED                                                                                                          [ 65%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_next_execution PASSED                                                                                                                    [ 70%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_pause PASSED                                                                                                                             [ 75%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_show_dag_dave PASSED                                                                                                                     [ 80%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_show_dag_imgcat PASSED                                                                                                                   [ 85%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_show_dag_print PASSED                                                                                                                    [ 90%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_trigger_dag PASSED                                                                                                                       [ 95%]
tests/cli/commands/test_dag_command.py::TestCliDags::test_trigger_dag_invalid_conf PASSED                                                                                                          [100%]
==================================================================================== 20 passed, 53 warnings in 35.12s ====================================================================================

And

pytest tests/cli/commands/test_task_command.py:

tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_list_tasks PASSED                                                                                                                  [  3%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_0__ignore_all_dependencies PASSED                                                                           [  7%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_1__ignore_depends_on_past PASSED                                                                            [ 11%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_2__ignore_dependencies PASSED                                                                               [ 15%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_invalid_raw_option_3__force PASSED                                                                                             [ 19%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_mutually_exclusive PASSED                                                                                                      [ 23%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_run_when_pickle_and_dag_cli_method_selected PASSED                                                                                 [ 26%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_test_with_env_vars PASSED                                                                                                          [ 30%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_cli_test_with_params PASSED                                                                                                            [ 34%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_parentdag_downstream_clear PASSED                                                                                                      [ 38%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_run_raises_when_theres_no_dagrun PASSED                                                                                                [ 42%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_run_with_existing_dag_run_id PASSED                                                                                                    [ 46%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_subdag_clear PASSED                                                                                                                    [ 50%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_task_render PASSED                                                                                                                     [ 53%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_task_state PASSED                                                                                                                      [ 57%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_task_states_for_dag_run PASSED                                                                                                         [ 61%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_task_states_for_dag_run_when_dag_run_not_exists PASSED                                                                                 [ 65%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_test PASSED                                                                                                                            [ 69%]
tests/cli/commands/test_task_command.py::TestCliTasks::test_test_with_existing_dag_run PASSED                                                                                                      [ 73%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_disable_handler_modifying SKIPPED (The test is skipped because it has quarantined marker. And --include-quarantined ...) [ 76%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_external_executor_id_present_for_fork_run_task SKIPPED (The test is skipped because it has quarantined marker. And -...) [ 80%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_external_executor_id_present_for_process_run_task SKIPPED (The test is skipped because it has quarantined marker. An...) [ 84%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_log_file_template_with_run_task SKIPPED (The test is skipped because it has quarantined marker. And --include-quaran...) [ 88%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_logging_with_run_task SKIPPED (The test is skipped because it has quarantined marker. And --include-quarantined flag...) [ 92%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_logging_with_run_task_subprocess SKIPPED (The test is skipped because it has quarantined marker. And --include-quara...) [ 96%]
tests/cli/commands/test_task_command.py::TestLogsfromTaskRunCommand::test_root_logger_restored SKIPPED (The test is skipped because it has quarantined marker. And --include-quarantined flag ...) [100%]
======================================================================================== short test summary info =========================================================================================
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_disable_handler_modifying>
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_external_executor_id_present_for_fork_run_task>
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_external_executor_id_present_for_process_run_task>
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_log_file_template_with_run_task>
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_logging_with_run_task>
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_logging_with_run_task_subprocess>
SKIPPED [1] tests/conftest.py:306: The test is skipped because it has quarantined marker. And --include-quarantined flag is passed to pytest. <TestCaseFunction test_root_logger_restored>
============================================================================== 19 passed, 7 skipped, 55 warnings in 55.27s ===============================================================================