Task instance details page blows up when no dagrun

[nadflinn]

Apache Airflow version:
1.10.9 although I think it would apply to 1.10.10, 1.10.8, 1.10.7, and 1.10.6

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration: aws
• OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
• Kernel (e.g. uname -a): Linux 567ff872fc2d 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:
If you have a DAG that has not been turned on before and you click on that DAG in the admin page and then in the DAG graph click on one of the tasks and then click on Task Instance Details you get the "Ooops" explosion error page. The error is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/dist-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/flask_login/utils.py", line 258, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/airflow/www/utils.py", line 290, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 1041, in task
    dep_context=dep_context)]
  File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 1039, in <listcomp>
    failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in
  File "/usr/local/lib/python3.7/dist-packages/airflow/models/taskinstance.py", line 668, in get_failed_dep_statuses
    dep_context):
  File "/usr/local/lib/python3.7/dist-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
    for dep_status in self._get_dep_statuses(ti, session, dep_context):
  File "/usr/local/lib/python3.7/dist-packages/airflow/ti_deps/deps/dagrun_id_dep.py", line 51, in _get_dep_statuses
    if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id):
AttributeError: 'NoneType' object has no attribute 'run_id'

What you expected to happen:
No error page. It should have loaded the standard task instance details page.

I believe the reason that this is happening is because there is no check to see if a dagrun exists in DagrunIdDep:
https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/dagrun_id_dep.py#L50

This is a bit of an edge case because there isn't really any reason someone should be checking out task instance details for a DAG that has never been run. That said we should probably avoid the error page.

How to reproduce it:
If you have a DAG that has not been turned on before and you click on that DAG in the admin page and then in the DAG graph click on one of the tasks and then click on Task Instance Details you get the "Ooops" explosion error page.

Anything else we need to know:

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[atsalolikhin-spokeo]

Hello,

I just installed Airflow for the first time, following the Quick Start guide.

I ran the bash example operator (still following the guide):

# run your first task instance
airflow run example_bash_operator runme_0 2015-01-01

And then, when I went to see how the run went, I got the error page with the mushroom cloud and the error.

Now, this is on Apache Airflow v1.10.11 (latest stable), which I see was released on July 10, 2020. And this pull request was merged on April 15th. And this issue is closed. But I'm running into the same broken behavior.

Here is my error page:

Ooops.

                          ____/ (  (    )   )  \___
                         /( (  (  )   _    ))  )   )\
                       ((     (   )(    )  )   (   )  )
                     ((/  ( _(   )   (   _) ) (  () )  )
                    ( (  ( (_)   ((    (   )  .((_ ) .  )_
                   ( (  )    (      (  )    )   ) . ) (   )
                  (  (   (  (   ) (  _  ( _) ).  ) . ) ) ( )
                  ( (  (   ) (  )   (  ))     ) _)(   )  )  )
                 ( (  ( \ ) (    (_  ( ) ( )  )   ) )  )) ( )
                  (  (   (  (   (_ ( ) ( _    )  ) (  )  )   )
                 ( (  ( (  (  )     (_  )  ) )  _)   ) _( ( )
                  ((  (   )(    (     _    )   _) _(_ (  (_ )
                   (_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
                   ((__)        \\||lll|l||///          \_))
                            (   /(/ (  )  ) )\   )
                          (    ( ( ( | | ) ) )\   )
                           (   /(| / ( )) ) ) )) )
                         (     ( ((((_(|)_)))))     )
                          (      ||\(|(|)|/||     )
                        (        |(||(||)||||        )
                          (     //|/l|||)|\\ \     )
                        (/ / //  /|//||||\\  \ \  \ _)
-------------------------------------------------------------------------------
Node: ip-172-20-2-227.us-west-2.compute.internal
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib64/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/utils.py", line 286, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1055, in task
    dep_context=dep_context)]
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1053, in <listcomp>
    failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses
    dep_context):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
    for dep_status in self._get_dep_statuses(ti, session, dep_context):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/dagrun_id_dep.py", line 51, in _get_dep_statuses
    if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id):
AttributeError: 'NoneType' object has no attribute 'run_id'

[atsalolikhin-spokeo]

I don't have the rights to re-open this issue. Is it enough for me to comment on it here?

[atsalolikhin-spokeo]

I ran into this in AWS, running Amazon Linux 2.

uname -a shows:

Linux ip-172-20-2-227.us-west-2.compute.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[mik-laj]

This fix may not have been officially released. Can you check to see if the problem is in the master version?

[atsalolikhin-spokeo]

Thank you for re-opening this issue. I've installed the master version:

[airflow@ip-172-20-2-227 airflow]$ airflow version
2.0.0.dev0
[airflow@ip-172-20-2-227 airflow]$

I can't login through the Web UI, though. It's asking me for username and password.

When I started the webserver, it said:

[2020-08-11 18:25:27,417] {manager.py:719} WARNING - No user yet created, use flask fab command to do it.

I tried, but it didn't work. Help?

[airflow@ip-172-20-2-227 airflow]$ flask fab create-user
Role [Public]:
Username: admin
User first name: admin
User last name: admin
Email: admin@example.com
Password:
Repeat for confirmation:
Usage: flask fab create-user [OPTIONS]

Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.
[airflow@ip-172-20-2-227 airflow]$

[mik-laj]

Here is docs about your problem: https://github.com/apache/airflow/blob/master/UPDATING.md#drop-legacy-ui-in-favor-of-fab-rbac-ui

[atsalolikhin-spokeo]

Thank you! I figured it out, actually.

Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.
[airflow@ip-172-20-2-227 airflow]$ find . -name wsgi.py
[airflow@ip-172-20-2-227 airflow]$ find . -name app.py
./airflow/www/app.py
[airflow@ip-172-20-2-227 airflow]$ cd airflow/www
[airflow@ip-172-20-2-227 www]$ flask fab create-user
Role [Public]: Admin
Username: admin
User first name: admin
User last name: admin
Email: admin@example.com
Password:
Repeat for confirmation:
[2020-08-11 18:34:09,566] {dagbag.py:409} INFO - Filling up the DagBag from /home/airflow/airflow/dags
Please make sure to build the frontend in static/ directory and restart the server
[2020-08-11 18:34:09,735] {manager.py:719} WARNING - No user yet created, use flask fab command to do it.
User admin created.
[airflow@ip-172-20-2-227 www]$

(I did read the documentation section you linked to, thank you; just after I already created an admin user and logged in.)

Don't laugh, but I can't figure out how to unpause the bash example operator. Instead of a sliding toggle switch, there is a checkbox.

I installed airflow by running "pip3 install --user ." in the top directory of the top of master.

[atsalolikhin-spokeo]

Here is a screenshot, showing there is no toggle switch like there was in 1.10.11

[atsalolikhin-spokeo]

I see this on my console:

Please make sure to build the frontend in static/ directory and restart the server

[mik-laj]

You should compile JS files now, but this is a problematic process.
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#installing-yarn-and-its-packages

Another solution: can you try to unpause the DAG with CLI and then go directly to the URL?

[atsalolikhin-spokeo]

Okay. I used sqlite3 to change the setting in airflow.db via CLI as you suggested.

Now the UI is showing all 22 DAGs are unpaused.

update dag
set is_paused = '0';

[atsalolikhin-spokeo]

I've installed JS using yarn as per the linked instructions. Now Airflow is showing 18 DAGs (before it was 22). The bash example operator DAG is gone.

[mik-laj]

Is the issue discussed in this ticket still present in this version?

[atsalolikhin-spokeo]

I'm sorry, I can't tell. I don't seem to have a fully functional Airflow installation.

The console (CLI) keeps telling me:

Please make sure to build the frontend in static/ directory and restart the server

but I don't know how to do that.

And the example bash operator disappeared from the UI after I did the "yarn prod run".

[atsalolikhin-spokeo]

I triggered the "tutorial" DAG instead (from the UI), and it looks like it ran.

I can look at the task details. No mushroom cloud. So I would say this issue is fixed in master.

[mdube9]

I'm using 1.10.9 and seeing the same error for task details on SLA misses. Can someone comment if this has been fixed or not

[jahanxb]

I have the same issue with my airflow, I am using version 1.10.12.

I cannot see the logs till the operation completes and even if the operation completes , i get this error. Why do we have to spend our time on tools with little or no support ? I will appreciate it, if there is a quick fix available as it appears a lot of people are facing the same problem.

[mik-laj]

This should be fixed in the next release. See: #11328
@turbaszek Can you confirm it?

[turbaszek]

@mik-laj yes, #11328 should fix it

[kaxil]

Closed by #11343

This will be released in 1.10.13