Airflow timeout error in chicago_taxi_pipeline example

[richardwu]

I followed the steps for installing and running the chicago taxi example and I seem to be getting this timeout error when initializing the DAG in airflow:

[2019-05-04 16:29:56,995] {__init__.py:416} ERROR - Failed to import: /Users/rwu1997/airflow/dags/taxi/taxi_pipeline_simple.py
Traceback (most recent call last):
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/airflow/models/__init__.py", line 413, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/Users/rwu1997/airflow/dags/taxi/taxi_pipeline_simple.py", line 23, in <module>
    from tfx.components.evaluator.component import Evaluator
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tfx/components/evaluator/component.py", line 24, in <module>
    from tfx.components.evaluator import executor
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tfx/components/evaluator/executor.py", line 22, in <module>
    import tensorflow_model_analysis as tfma
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_model_analysis/__init__.py", line 19, in <module>
    from tensorflow_model_analysis import view
[2019-05-04 16:31:19 -0400] [82036] [CRITICAL] WORKER TIMEOUT (pid:82765)
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_model_analysis/view/__init__.py", line 15, in <module>
    from tensorflow_model_analysis.view.widget_view import render_plot
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_model_analysis/view/widget_view.py", line 21, in <module>
    from tensorflow_model_analysis.api import model_eval_lib
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_model_analysis/api/model_eval_lib.py", line 31, in <module>
    from tensorflow_model_analysis import types
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_model_analysis/types.py", line 24, in <module>
    from tensorflow_transform.beam import shared
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_transform/__init__.py", line 19, in <module>
    from tensorflow_transform.analyzers import *
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_transform/analyzers.py", line 39, in <module>
    from tensorflow_transform import tf_utils
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow_transform/tf_utils.py", line 24, in <module>
    from tensorflow.contrib.proto.python.ops import encode_proto_op
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow/contrib/__init__.py", line 49, in <module>
    from tensorflow.contrib import distributions
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/tensorflow/contrib/distributions/__init__.py", line 16, in <module>
    """
  File "/Users/rwu1997/anaconda3/envs/taxi_pipeline/lib/python2.7/site-packages/airflow/utils/timeout.py", line 43, in handle_timeout
    raise AirflowTaskTimeout(self.error_message)
AirflowTaskTimeout: Timeout, PID: 82765

This is my anaconda environment:

WARNING: The conda.compat module is deprecated and will be removed in a future release.
# packages in environment at /Users/rwu1997/anaconda3/envs/taxi_pipeline:
#
# Name                    Version                   Build  Channel
absl-py                   0.7.1                    pypi_0    pypi
alembic                   0.9.10                   pypi_0    pypi
apache-airflow            1.10.3                   pypi_0    pypi
apache-beam               2.12.0                   pypi_0    pypi
appnope                   0.1.0                    pypi_0    pypi
astor                     0.7.1                    pypi_0    pypi
attrs                     19.1.0                   pypi_0    pypi
avro                      1.8.2                    pypi_0    pypi
babel                     2.6.0                    pypi_0    pypi
backports-abc             0.5                      pypi_0    pypi
backports-shutil-get-terminal-size 1.0.0                    pypi_0    pypi
backports-ssl-match-hostname 3.7.0.1                  pypi_0    pypi
backports-weakref         1.0.post1                pypi_0    pypi
bleach                    3.1.0                    pypi_0    pypi
cachetools                3.1.0                    pypi_0    pypi
certifi                   2019.3.9                 py27_0
chardet                   3.0.4                    pypi_0    pypi
click                     7.0                      pypi_0    pypi
colorama                  0.4.1                    pypi_0    pypi
configparser              3.5.3                    pypi_0    pypi
crcmod                    1.7                      pypi_0    pypi
croniter                  0.3.30                   pypi_0    pypi
decorator                 4.4.0                    pypi_0    pypi
defusedxml                0.6.0                    pypi_0    pypi
dill                      0.2.9                    pypi_0    pypi
docker                    3.7.2                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
docutils                  0.14                     pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
enum34                    1.1.6                    pypi_0    pypi
fastavro                  0.21.22                  pypi_0    pypi
fasteners                 0.14.1                   pypi_0    pypi
flask                     1.0.2                    pypi_0    pypi
flask-admin               1.5.3                    pypi_0    pypi
flask-appbuilder          1.12.3                   pypi_0    pypi
flask-babel               0.12.2                   pypi_0    pypi
flask-caching             1.3.3                    pypi_0    pypi
flask-login               0.4.1                    pypi_0    pypi
flask-openid              1.2.5                    pypi_0    pypi
flask-sqlalchemy          2.4.0                    pypi_0    pypi
flask-swagger             0.2.13                   pypi_0    pypi
flask-wtf                 0.14.2                   pypi_0    pypi
funcsigs                  1.0.0                    pypi_0    pypi
functools32               3.2.3.post2              pypi_0    pypi
future                    0.16.0                   pypi_0    pypi
futures                   3.2.0                    pypi_0    pypi
gast                      0.2.2                    pypi_0    pypi
gitdb2                    2.0.5                    pypi_0    pypi
gitpython                 2.1.11                   pypi_0    pypi
google-api-core           1.10.0                   pypi_0    pypi
google-api-python-client  1.7.8                    pypi_0    pypi
google-apitools           0.5.26                   pypi_0    pypi
google-auth               1.6.3                    pypi_0    pypi
google-auth-httplib2      0.0.3                    pypi_0    pypi
google-cloud-bigquery     1.6.1                    pypi_0    pypi
google-cloud-bigtable     0.31.1                   pypi_0    pypi
google-cloud-core         0.28.1                   pypi_0    pypi
google-cloud-pubsub       0.39.0                   pypi_0    pypi
google-resumable-media    0.3.2                    pypi_0    pypi
googleapis-common-protos  1.5.10                   pypi_0    pypi
googledatastore           7.0.2                    pypi_0    pypi
grpc-google-iam-v1        0.11.4                   pypi_0    pypi
grpcio                    1.20.1                   pypi_0    pypi
gunicorn                  19.9.0                   pypi_0    pypi
h5py                      2.9.0                    pypi_0    pypi
hdfs                      2.5.2                    pypi_0    pypi
httplib2                  0.12.3                   pypi_0    pypi
idna                      2.8                      pypi_0    pypi
ipaddress                 1.0.22                   pypi_0    pypi
ipykernel                 4.10.0                   pypi_0    pypi
ipython                   5.8.0                    pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                7.4.2                    pypi_0    pypi
iso8601                   0.1.12                   pypi_0    pypi
itsdangerous              1.1.0                    pypi_0    pypi
jinja2                    2.10                     pypi_0    pypi
json-merge-patch          0.2                      pypi_0    pypi
jsonschema                3.0.1                    pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            5.2.4                    pypi_0    pypi
jupyter-console           5.2.0                    pypi_0    pypi
jupyter-core              4.4.0                    pypi_0    pypi
keras-applications        1.0.7                    pypi_0    pypi
keras-preprocessing       1.0.9                    pypi_0    pypi
libcxx                    4.0.1                hcfea43d_1
libcxxabi                 4.0.1                hcfea43d_1
libedit                   3.1.20181209         hb402a30_0
libffi                    3.2.1                h475c297_4
lockfile                  0.12.2                   pypi_0    pypi
lxml                      4.3.3                    pypi_0    pypi
mako                      1.0.9                    pypi_0    pypi
markdown                  2.6.11                   pypi_0    pypi
markupsafe                1.1.1                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
ml-metadata               0.13.2                   pypi_0    pypi
mock                      2.0.0                    pypi_0    pypi
monotonic                 1.5                      pypi_0    pypi
nbconvert                 5.5.0                    pypi_0    pypi
nbformat                  4.4.0                    pypi_0    pypi
ncurses                   6.1                  h0a44026_1
notebook                  5.7.8                    pypi_0    pypi
numpy                     1.16.3                   pypi_0    pypi
oauth2client              3.0.0                    pypi_0    pypi
ordereddict               1.1                      pypi_0    pypi
pandas                    0.24.2                   pypi_0    pypi
pandocfilters             1.4.2                    pypi_0    pypi
pathlib2                  2.3.3                    pypi_0    pypi
pbr                       5.2.0                    pypi_0    pypi
pendulum                  1.4.4                    pypi_0    pypi
pexpect                   4.7.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pip                       19.1                     py27_0
prometheus-client         0.6.0                    pypi_0    pypi
prompt-toolkit            1.0.16                   pypi_0    pypi
proto-google-cloud-datastore-v1 0.90.4                   pypi_0    pypi
protobuf                  3.7.1                    pypi_0    pypi
psutil                    5.6.2                    pypi_0    pypi
ptyprocess                0.6.0                    pypi_0    pypi
pyarrow                   0.11.1                   pypi_0    pypi
pyasn1                    0.4.5                    pypi_0    pypi
pyasn1-modules            0.2.5                    pypi_0    pypi
pydot                     1.2.4                    pypi_0    pypi
pygments                  2.3.1                    pypi_0    pypi
pyparsing                 2.4.0                    pypi_0    pypi
pyrsistent                0.15.1                   pypi_0    pypi
python                    2.7.16               h97142e2_0
python-daemon             2.1.2                    pypi_0    pypi
python-dateutil           2.8.0                    pypi_0    pypi
python-editor             1.0.4                    pypi_0    pypi
python-openid             2.2.5                    pypi_0    pypi
pytz                      2019.1                   pypi_0    pypi
pytzdata                  2019.1                   pypi_0    pypi
pyvcf                     0.6.8                    pypi_0    pypi
pyyaml                    3.13                     pypi_0    pypi
pyzmq                     18.0.1                   pypi_0    pypi
qtconsole                 4.4.4                    pypi_0    pypi
readline                  7.0                  h1de35cc_5
requests                  2.21.0                   pypi_0    pypi
rsa                       4.0                      pypi_0    pypi
scandir                   1.10.0                   pypi_0    pypi
scikit-learn              0.20.3                   pypi_0    pypi
scipy                     0.19.1                   pypi_0    pypi
send2trash                1.5.0                    pypi_0    pypi
setproctitle              1.1.10                   pypi_0    pypi
setuptools                41.0.1                   py27_0
simplegeneric             0.8.1                    pypi_0    pypi
singledispatch            3.4.0.3                  pypi_0    pypi
six                       1.12.0                   pypi_0    pypi
smmap2                    2.0.5                    pypi_0    pypi
sqlalchemy                1.2.19                   pypi_0    pypi
sqlite                    3.28.0               ha441bb4_0
tabulate                  0.8.3                    pypi_0    pypi
tenacity                  4.12.0                   pypi_0    pypi
tensorboard               1.12.2                   pypi_0    pypi
tensorflow                1.12.0                   pypi_0    pypi
tensorflow-data-validation 0.12.0                   pypi_0    pypi
tensorflow-metadata       0.12.1                   pypi_0    pypi
tensorflow-model-analysis 0.12.1                   pypi_0    pypi
tensorflow-transform      0.12.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.8.2                    pypi_0    pypi
testpath                  0.4.2                    pypi_0    pypi
text-unidecode            1.2                      pypi_0    pypi
tfx                       0.12.0                   pypi_0    pypi
thrift                    0.11.0                   pypi_0    pypi
tk                        8.6.8                ha441bb4_0
tornado                   5.1.1                    pypi_0    pypi
traitlets                 4.3.2                    pypi_0    pypi
typing                    3.6.6                    pypi_0    pypi
tzlocal                   1.5.1                    pypi_0    pypi
unicodecsv                0.14.1                   pypi_0    pypi
uritemplate               3.0.0                    pypi_0    pypi
urllib3                   1.24.3                   pypi_0    pypi
wcwidth                   0.1.7                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          0.56.0                   pypi_0    pypi
werkzeug                  0.14.1                   pypi_0    pypi
wheel                     0.33.1                   py27_0
widgetsnbextension        3.4.2                    pypi_0    pypi
wtforms                   2.2.1                    pypi_0    pypi
zlib                      1.2.11               h1de35cc_3
zope-deprecation          4.4.0                    pypi_0    pypi

[richardwu]

Seems like the default airflow timeout threshold was set too low: might be a good idea to add to the README that dagbag_import_timeout should be bumped to something higher e.g. 100 in airflow.cfg.

[1025KB]

Just curious, does it happen everytime without dagbag_import_timeout change?

[richardwu]

It did for me: the default configuration spawns 4 webserver workers and it seems that all of them each individually attempt to load the dagbag. At least one of them would experience a timeout error upon startup.

I also don't have the beefiest computer (Macbook pro 2015, dual core w/ hyperthreading).

[1025KB]

Hi, Richard, I noticed that unit of dagbag_import_timeout is second, so 30s should be enough, I also find they have a discuss related to decrease it further more, they mentioned some case that it might take a long time importing, e.g., scheduler machine is overloaded, it might be the reason