Some DAGs don't open in the UI using MSSQL as the backend

[fphhotchips]

Airflow on Ubuntu 14.04 using Python 2.7.10 and PyMSSQL 2.1.1. MSSQL is 2016 CTP 2. Downgrading PyMSSQL to a 1.x version solves the problem.

I'd suggest this is probably an issue upstream, but I'm not familiar enough with Airflow to be sure.

Trying to access the example Python DAG gives the following traceback (same for bash example):

2015-08-31 22:00:15,383 - root - INFO - Importing /home/nic_000/anaconda/lib/python2.7/site-packages/airflow/example_dags/example_python_operator.py
2015-08-31 22:00:15,385 - root - INFO - Loaded DAG <DAG: example_python_operator>
2015-08-31 22:00:15,389 - airflow.www.app - ERROR - Exception on /admin/airflow/graph [GET]
Traceback (most recent call last):
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask_admin/base.py", line 68, in inner
    return self._run_view(f, *args, **kwargs)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/flask_admin/base.py", line 359, in _run_view
    return fn(self, *args, **kwargs)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/airflow/www/utils.py", line 96, in view_func
    return f(*args, **kwargs)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/airflow/www/app.py", line 1137, in graph
    dttm = dag.latest_execution_date or datetime.now().date()
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/airflow/models.py", line 1619, in latest_execution_date
    TI.task_id.in_(self.task_ids)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2504, in scalar
    ret = self.one()
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2473, in one
    ret = list(self)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1344, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/nic_000/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "pymssql.pyx", line 417, in pymssql.Cursor.execute (pymssql.c:6057)
  File "_mssql.pyx", line 943, in _mssql.MSSQLConnection.execute_query (_mssql.c:9858)
  File "_mssql.pyx", line 974, in _mssql.MSSQLConnection.execute_query (_mssql.c:9734)
  File "_mssql.pyx", line 1091, in _mssql.MSSQLConnection.format_and_run_query (_mssql.c:10814)
  File "_mssql.pyx", line 1113, in _mssql.MSSQLConnection.format_sql_command (_mssql.c:11042)
  File "_mssql.pyx", line 1779, in _mssql._substitute_params (_mssql.c:18463)
  File "_mssql.pyx", line 195, in _mssql.ensure_bytes (_mssql.c:2671)
TypeError: Expected bytes, got newbytes
2015-08-31 22:00:15,397 - tornado.access - ERROR - 500 GET /admin/airflow/graph?dag_id=example_python_operator (192.168.56.1) 22.74ms

[mistercrunch]

Yes, from looking at the stack trace this seems to be an issue with SqlAlchemy and the MSSQL dialect and Python3 compatibility (newbytes), @jlowin, any input on this?

[jlowin]

Interesting... Future bytes object is supposed to be bulletproof but I guess an explicit typecheck could fail. I don't use mssql so I won't be able to test this directly but I will try to diagnose it tonight.

[mistercrunch]

I guess that is part of the reason why type checks aren't pythonesque.

[jlowin]

@fphhotchips I'm having trouble replicating this. Under Python 2.7, when I try to view example_python_operator and capture the values of the task_ids in airflow/models.py:latest_execution_date, they are all str (the expected behavior), not newbytes.

Here's what I'm doing, maybe you could do the same -- I'm sorry if this is more spelunking than you signed up for! You'll need ipdb (pip install ipdb if you don't have it). Your traceback shows line 1619 of models.py creating the error (the one that says TI.task_id.in_(self.task_ids)). If you go three lines above that (just after the one that creates the session), you can add a new line:

session = ...
import ipdb; ipdb.set_trace() # new line
...

Save the file and start up the webserver again. Now when you try to view the DAG, your terminal (whichever one you started the webserver in) will open up a debugging window where you added the breakpoint. What is the output of typing type(self.task_ids[0])? I get str, but I think you must be getting newbytes... next step would be to figure out why.

[fphhotchips]

I also get str, but the exception still gets raised afterwards.

Further down the rabbit hole...

Edit:
It turns out that the list of task_ids has the following:

 'sleep_for_0',
 'sleep_for_1',
 'sleep_for_2',
 'sleep_for_3',
 'sleep_for_4',
 'sleep_for_5',
 'sleep_for_6',
 'sleep_for_7',
 'sleep_for_8',
 'sleep_for_9']

With types:

[<type 'str'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>, <class 'future.types.newstr.newstr'>]

[jlowin]

So strange, I don't have that. I have a theory though -- if you open the file that creates the example (example_python_operator.py) and look near the bottom, you'll see a loop that creates the sleeping tasks.

The line that creates the task id is:

task_id='sleep_for_'+str(i),

Could you try changing it to:

task_id='sleep_for_{}'.format(i),

And let me know if your system is still creating newstr types? I'm not sure if this is it but I'm not sure how else it could only be affecting the sleep tasks and not the first one -- but that str is suspicious. Please let me know.

[fphhotchips]

That fixes it. So essentially the builtins.str method that's imported at the top of the file gives a <class 'future.types.newstr.newstr'>, whereas just the regular str function gives type str. So, when you use the 'string' syntax for task_id='print_the_context' you get a different type than when you used str(i).

Mystery solved! Thanks for your help.

Is it worth someone submitting an issue on pymssql?
Further, does changing the line above break anything (particularly Python 3 compatibility) in any way? I can't see that it would, but as noted above I'm not exactly an expert on the airflow codebase.

[jlowin]

Ok, great!

I've been trying to make Airflow Py3 compatible without messing with the Py2 codebase too much. This depends heavily on the future library, which has lots of forwards/backwards-compatible objects (like str/newstr). Generally speaking, it "does the right thing" and doesn't interfere -- but sometimes it doesn't work (like here!). I'll submit a PR very soon to fix this behavior.

And changing that line is totally fine -- but we want to fix this at a deeper level to make sure no one does it by accident. Sorry you had to encounter this bug!

[jlowin]

@fphhotchips I think this particular problem has already been fixed in the latest version of Airflow -- @mistercrunch removed the problematic line in 3b35748. Could you try upgrading and see if it goes away? Meanwhile, I will look for any similar issues. Thanks!

[syvineckruyk]

👍

[mistercrunch]

Seems resolved, closing.