failed to build puckel/docker-airflow

[sdikby]

Hi,

when trying to build puckel/docker-airflow i got this error: Cython.Compiler.Errors.InternalError: Internal compiler error: 'algos_common_helper.pxi' not found

More logs:

Downloading/unpacking pandas<1.0.0,>=0.15.2 (from airflow[celery,hdfs,hive,postgres]==1.7.1.3)
Running setup.py (path:/tmp/pip-build-fReTFb/pandas/setup.py) egg_info for package pandas

package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
pandas/index.pyx: cannot find cimported module 'datetime'
pandas/index.pyx: cannot find cimported module 'util'
./pandas/hashtable.pxd: cannot find cimported module 'khash'
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/tmp/pip-build-fReTFb/pandas/setup.py", line 680, in <module>
    **setuptools_kwargs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "<string>", line 15, in replacement_run
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 306, in find_sources
    mm.run()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 533, in run
    self.add_defaults()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 562, in add_defaults
    sdist.add_defaults(self)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 36, in add_defaults
    self._add_defaults_ext()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 19, in finalize_options
    self.distribution.ext_modules)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 809, in cythonize
    aliases=aliases)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 714, in create_extension_list
    kwds = deps.distutils_info(file, aliases, base).values
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 590, in distutils_info
    return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 600, in transitive_merge
    node, extract, merge, seen, {}, self.cimported_files)[0]
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 605, in transitive_merge_helper
    deps = extract(node)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 581, in distutils_info0
    externs = self.cimports_and_externs(filename)[1]
  File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 491, in cimports_and_externs
    for include in self.included_files(filename):
  File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 474, in included_files
    include_path = self.context.find_include_file(include, None)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 274, in find_include_file
    error(pos, "'%s' not found" % filename)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Errors.py", line 177, in error
    raise InternalError(message)
Cython.Compiler.Errors.InternalError: Internal compiler error: 'algos_common_helper.pxi' not found
Complete output from command python setup.py egg_info:
running egg_info

creating pip-egg-info/pandas.egg-info

writing requirements to pip-egg-info/pandas.egg-info/requires.txt

writing pip-egg-info/pandas.egg-info/PKG-INFO

writing top-level names to pip-egg-info/pandas.egg-info/top_level.txt

writing dependency_links to pip-egg-info/pandas.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/pandas.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found

package init file 'pandas/io/tests/sas/init.py' not found (or not a regular file)

pandas/index.pyx: cannot find cimported module 'datetime'

pandas/index.pyx: cannot find cimported module 'util'

./pandas/hashtable.pxd: cannot find cimported module 'khash'

Traceback (most recent call last):

File "", line 17, in

File "/tmp/pip-build-fReTFb/pandas/setup.py", line 680, in

**setuptools_kwargs)

File "/usr/lib/python2.7/distutils/core.py", line 151, in setup

dist.run_commands()

File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands

self.run_command(cmd)

File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command

cmd_obj.run()

File "", line 15, in replacement_run

File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 306, in find_sources

mm.run()

File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 533, in run

self.add_defaults()

File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 562, in add_defaults

sdist.add_defaults(self)

File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 36, in add_defaults

self._add_defaults_ext()

File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext

build_ext = self.get_finalized_command('build_ext')

File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command

cmd_obj.ensure_finalized()

File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

self.finalize_options()

File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 19, in finalize_options

self.distribution.ext_modules)

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 809, in cythonize

aliases=aliases)

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 714, in create_extension_list

kwds = deps.distutils_info(file, aliases, base).values

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 590, in distutils_info

return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 600, in transitive_merge

node, extract, merge, seen, {}, self.cimported_files)[0]

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 605, in transitive_merge_helper

deps = extract(node)

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 581, in distutils_info0

externs = self.cimports_and_externs(filename)[1]

File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper

res = cache[args] = f(self, *args)

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 491, in cimports_and_externs

for include in self.included_files(filename):

File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper

res = cache[args] = f(self, *args)

File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 474, in included_files

include_path = self.context.find_include_file(include, None)

File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 274, in find_include_file

error(pos, "'%s' not found" % filename)

File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Errors.py", line 177, in error

raise InternalError(message)

Cython.Compiler.Errors.InternalError: Internal compiler error: 'algos_common_helper.pxi' not found

---

Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-build-fReTFb/pandas
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c set -ex && buildDeps=' python-dev python-pip libkrb5-dev libsasl2-dev libssl-dev libffi-dev build-essential libblas-dev liblapack-dev ' && echo "deb http://http.debian.net/debian jessie-backports main" >/etc/apt/sources.list.d/backports.list && apt-get update -yqq && apt-get install -yqq --no-install-recommends $buildDeps apt-utils curl netcat locales &amp;&amp; apt-get install -yqq -t jessie-backports python-requests libpq-dev &amp;&amp; apt-get install -y sshpass &amp;&amp; apt-get install -y openssh-server &amp;&amp; sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen && locale-gen && update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 && useradd -ms /bin/bash -d ${AIRFLOW_HOME} airflow && pip install --upgrade setuptools && pip install Cython && pip install pytz==2015.7 && pip install python-dateutil && pip install numpy && pip install cryptography && pip install pyOpenSSL && pip install ndg-httpsclient && pip install pyasn1 && pip install psycopg2 && pip install airflow[celery,postgres,hive,hdfs]==$AIRFLOW_VERSION && apt-get remove --purge -yqq $buildDeps libpq-dev && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/man /usr/share/doc /usr/share/doc-base' returned a non-zero code: 1

---

any ideas ?

[anurajp]

I am facing the same issue

[anurajp]

I have temporarily fixed it by installing the older version of pandas. pandas dependency is coming from airflow. Add pip install pandas==0.18.1 to the docker file(before pip install airflow) to force older version of pandas used in airflow. Seems to be an issue with pandas 0.19

[sdikby]

@anurajp yo it works ! thanks.
But i can't acces the webUI after building. with docker ps it shows that the scheduler keeps restarting every ~1 min. Any idea?
PS: i'm using a modifed version of the celereExecutor docker compose file:

version: '2'
services:
rabbitmq:
image: rabbitmq:3-management
restart: always
ports:
- "15672:15672"
- "5672:5672"
environment:
- RABBITMQ_DEFAULT_USER=airflow
- RABBITMQ_DEFAULT_PASS=airflow
- RABBITMQ_DEFAULT_VHOST=airflow

postgres:
    image: postgres
    environment:
        - POSTGRES_USER=airflow
        - POSTGRES_PASSWORD=airflow
        - POSTGRES_DB=airflow
    volumes:
        - /stor/OSF_Tracking/tmp/postgresql:/var/lib/postgresql/data

webserver:
    image: puckel/docker-airflow:1.7.1.3-3
    restart: always
    depends_on:
        - postgres
        - rabbitmq
    environment:
        # - LOAD_EX=n
        - EXECUTOR=Celery
    volumes:
        - /stor/OSF_Tracking/airflow_docker/dags_folder:/usr/local/airflow/dags
        - /stor/OSF_Tracking/airflow_docker/logs:/usr/local/airflow/logs
    ports:
        - "9595:9595"
    command: webserver

flower:
    image: puckel/docker-airflow:1.7.1.3-3
    restart: always
    depends_on:
        - rabbitmq
    environment:
        - EXECUTOR=Celery
    ports:
        - "5555:5555"
    command: flower

scheduler:
    image: puckel/docker-airflow:1.7.1.3-3
    restart: always
    depends_on:
        - webserver
    volumes:
        - /stor/OSF_Tracking/airflow_docker/dags_folder:/usr/local/airflow/dags
        - /stor/OSF_Tracking/airflow_docker/logs:/usr/local/airflow/logs
    environment:
        # - LOAD_EX=n
        - EXECUTOR=Celery
    command: scheduler -n 5

worker:
    image: puckel/docker-airflow:1.7.1.3-3
    restart: always
    depends_on:
        - scheduler
    volumes:
        - /stor/OSF_Tracking/airflow_docker/dags_folder:/usr/local/airflow/dags
        - /stor/OSF_Tracking/airflow_docker/logs:/usr/local/airflow/logs
    environment:
        - EXECUTOR=Celery
    command: worker

[anurajp]

@sdikby could you follow the webserver and scheduler log and paste the error?

[sdikby]

@anurajp i solved it. it was some network problem with an old container version of docker airflow.
I have one last question if i may. To connect to my host via airflow docker i use a tool named sshpass. Icall it each time within any airflow operator for example to execute some extern bash script or to do something in HDFS. I think it is some brute solution because my username/password for the host are hard coded in each DAG. Can you maybe point me to another alternative? Which method do you use to interact with extern component from airflow docker ?

[anurajp]

@sdikby apologies for late reply. We use SSHOperator to connect airflow docker with our gateway machine. Our gateway machine has all hadoop clients installed. So far it has worked pretty nicely for us.

[sdikby]

@anurajp would you please tell me exactly how you do it with SSHOperator ? maybe with a SSHOperator task layout example? and if i should do something else beside instanciating this operator?