Clicking Edit on TaskInstance Causes Crash

[jrd281]

Apache Airflow version: 2.0.0b2

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration: Ubuntu 20.04 Guest/Windows Server 2012 R2 Host
• OS (e.g. from /etc/os-release): Ubuntu 20.04.1 LTS
• Kernel (e.g. uname -a): Linux 5.4.0-1031-azure A simple plugin system for Airflow #32-Ubuntu SMPx86_64 GNU/Linux
• Install tools: Pip 20.3b1, Ansible Template - https://github.com/jrd281/ansible-role-airflow/tree/airflowdbg-1
• Others: Python 3.8

What happened: I was looking at the taskinstance list and then I clicked the edit icon. After clicking, an error page was displayed.

What you expected to happen: I expected to be whisked to an editing UI

How to reproduce it:

1. Visit your installation's home page
2. Click the success bucket under the Recent Tasks heading
   
3. Click the Edit button for one of your tasks
   
4. Watch the system throw an error.
   

Anything else we need to know:

UI Error Page

Something bad has happened. Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.5
Airflow version: 2.0.0b2
Node: REDACTED

Traceback (most recent call last):
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
return f(self, *args, **kwargs)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/views.py", line 606, in edit
return self.render_template(
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 280, in render_template
return render_template(
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/templating.py", line 137, in render_template
return _render(
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html", line 2, in top-level template code
{% import 'appbuilder/general/lib.html' as lib %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/base.html", line 1, in top-level template code
{% extends base_template %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/airflow/www/templates/airflow/master.html", line 20, in top-level template code
{% extends 'appbuilder/baselayout.html' %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 46, in top-level template code
{% block body %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 19, in block "body"
{% block content %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html", line 23, in block "content"
{% block edit_form %}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html", line 25, in block "edit_form"
{{ widgets.get('edit')(form_action=form_action)|safe }}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/widgets.py", line 37, in call
return template.render(args)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/widgets/form.html", line 45, in top-level template code
{{ lib.render_field(field, begin_sep_label, end_sep_label, begin_sep_field, end_sep_field) }}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/jinja2/runtime.py", line 679, in _invoke
rv = self._func(*arguments)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/lib.html", line 237, in template
{{ field(**kwargs)|safe }}
File "/var/lib/airflow/venv/lib/python3.8/site-packages/wtforms/fields/core.py", line 160, in call
return self.meta.render_field(self, kwargs)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/wtforms/meta.py", line 56, in render_field
return field.widget(field, **render_kw)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/fieldwidgets.py", line 176, in call
return super(Select2ManyWidget, self).call(field, **kwargs)
File "/var/lib/airflow/venv/lib/python3.8/site-packages/wtforms/widgets/core.py", line 300, in call
for val, label, selected in field.iter_choices():
File "/var/lib/airflow/venv/lib/python3.8/site-packages/flask_appbuilder/fields.py", line 209, in iter_choices
yield (pk, self.get_label(obj), obj in self.data)
TypeError: argument of type 'DagRun' is not iterable

How often does this problem occur?: Every time I click the Edit icon

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[zachliu]

@jrd281 lol, you beat me to it 😂
let me add that this 🐛 is also in 2.0.0b3

Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.6
Airflow version: 2.0.0b3
Node: <Of Course I Still Love You>
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/views.py", line 606, in edit
    return self.render_template(
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 280, in render_template
    return render_template(
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 137, in render_template
    return _render(
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 120, in _render
    rv = template.render(context)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html", line 2, in top-level template code
    {% import 'appbuilder/general/lib.html' as lib %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/base.html", line 1, in top-level template code
    {% extends base_template %}
  File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/master.html", line 20, in top-level template code
    {% extends 'appbuilder/baselayout.html' %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 46, in top-level template code
    {% block body %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 19, in block "body"
    {% block content %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html", line 23, in block "content"
    {% block edit_form %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/model/edit.html", line 25, in block "edit_form"
    {{ widgets.get('edit')(form_action=form_action)|safe }}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/widgets.py", line 37, in __call__
    return template.render(args)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/widgets/form.html", line 45, in top-level template code
    {{ lib.render_field(field, begin_sep_label, end_sep_label, begin_sep_field, end_sep_field) }}
  File "/usr/local/lib/python3.8/site-packages/jinja2/runtime.py", line 679, in _invoke
    rv = self._func(*arguments)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/general/lib.html", line 237, in template
    {{ field(**kwargs)|safe }}
  File "/usr/local/lib/python3.8/site-packages/wtforms/fields/core.py", line 160, in __call__
    return self.meta.render_field(self, kwargs)
  File "/usr/local/lib/python3.8/site-packages/wtforms/meta.py", line 56, in render_field
    return field.widget(field, **render_kw)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/fieldwidgets.py", line 176, in __call__
    return super(Select2ManyWidget, self).__call__(field, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/wtforms/widgets/core.py", line 300, in __call__
    for val, label, selected in field.iter_choices():
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/fields.py", line 209, in iter_choices
    yield (pk, self.get_label(obj), obj in self.data)
TypeError: argument of type 'DagRun' is not iterable

[patchandpray]

I'd like to work on this one.

[mik-laj]

@patchandpray Assigned 🐈

[ryw]

any progress on this @patchandpray?

[patchandpray]

I managed to identify where the problem is. but I did not have the time to look into the actual fix. I should have time for that tomorrow :)

[ryw]

ok we'd like this bugfix in for Airflow 2.0rc1 so let us know if you get stuck / need help

[patchandpray]

Hi, I think I need some help.

I have been able to deduce that the issue arises upon form autogeneration for a taskinstance by flask-appbuilder. When editing a taskinstance. A taskinstance appears to have a one on one relationship with a DagRun and when generating the edit form flask_appbuilder uses QuerySelectMultipleField.iter_choices() which gets called with: <DagRun example_bash_operator @ 2020-11-29 00:00:00+00:00: scheduled__2020-11-29T00:00:00+00:00, externally triggered: False>. Since this is a non-iterable object the exception is raised.

When using QuerySelectField.iter_choices() with this object the form does generate but I am stuck finding a way to force this behavior from airflow code. An option might be to not autogenerate a form and specify a form for taskinstance edit so we can explicitly define which fields to use but I am unsure if this is the correct way to implement a fix for this.

Also, how is determined which fields are generated for this form?

Furthermore when forcing the generation from flask_appbuilder and generating a form:

when trying to save a modification for a taskinstance I am presented with this error. I want to create a new bug report for this but it might be related to this bug.

[kaxil]

@patchandpray #12770 fixes the issue.

We are in process of fixing all the pending bugs for 2.0 since we are just a week away from cutting RCs -- so I went ahead and created the PR to fix this.

[patchandpray]

I understand completely. 🙂