WebUI broke when enable_xcom_pickling Airflow2.1

[saurasingh]

Apache Airflow version: 2.1

Kubernetes version (if you are using kubernetes) (use kubectl version):
AWS EKS, AWS ECS
Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian GNU/Linux 10
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

I have a python operator where python_callable just returns a data frame. And I was getting this error

[2021-07-06 15:02:08,889] {xcom.py:229} ERROR - Could not serialize the XCom value into JSON. If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config.
[2021-07-06 15:02:08,890] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1344, in _execute_task
    self.xcom_push(key=XCOM_RETURN_KEY, value=result)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1925, in xcom_push
    session=session,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line 79, in set
    value = XCom.serialize_value(value)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line 226, in serialize_value
    return json.dumps(value).encode('UTF-8')
  File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DataFrame is not JSON serializable

Then I set enable_xcom_pickling: True in airflow.cfg as per suggestion in the error. It did work and DAG was successful but then my XCOM's UI broke with the following error.

Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.7.10
Airflow version: 2.1.0
Node: airflow-webserver-7d7f64fbc4-zl8zk
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/views.py", line 553, in list
    self.list_template, title=self.list_title, widgets=widgets
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 288, in render_template
    template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
    ctx.app,
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
    rv = template.render(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/list.html", line 2, in top-level template code
    {% import 'appbuilder/general/lib.html' as lib %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/base.html", line 1, in top-level template code
    {% extends base_template %}
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/templates/airflow/main.html", line 20, in top-level template code
    {% extends 'appbuilder/baselayout.html' %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 37, in top-level template code
    {% block body %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 19, in block "body"
    {% block content %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/list.html", line 13, in block "content"
    {% block list_list scoped %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/list.html", line 15, in block "list_list"
    {{ widgets.get('list')()|safe }}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/widgets.py", line 37, in __call__
    return template.render(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/templates/airflow/model_list.html", line 21, in top-level template code
    {% extends 'appbuilder/general/widgets/base_list.html' %}
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/widgets/base_list.html", line 23, in top-level template code
    {% block begin_loop_values %}
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/templates/airflow/model_list.html", line 80, in block "begin_loop_values"
    {% elif item[value] != None %}
  File "/home/airflow/.local/lib/python3.7/site-packages/pandas/core/generic.py", line 1443, in __nonzero__
    f"The truth value of a {type(self).__name__} is ambiguous. "
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

What you expected to happen:

I expected to see the XCOM return_value in the xcom list as it use to in previous versions.

How to reproduce it:

Anything else we need to know:

[samgans]

We need to change the != None to is not None as the DF cannot evaluate to True or False as a whole. I can make a PR and fix this in close time. Any ideas if it can break something?

[saurasingh]

@samgans I believe this was already addressed in the past #12315, but really do not know if @turbaszek changed this !=None to is not None. Also, I am too sure if this is going to break anything.

[samgans]

Yes, the decision was not committed to the main. I've just tested and it seems like changing this to is not none fixes the issue. I think I will make a PR to address this and make an issue that will describe another problem revealed in #12315.

UPD: The trouble with deserialization was fixed, but the change for the template was not merged, so anyway I will make a PR tomorrow with the changes.

[uranusjr]

Not sure I understand the issue (not familiar with Pandas); since #16893 is merged, should this be closed?

[samgans]

Hi @uranusjr. Yes, I think it can be closed.