Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky docker example test around mpi4py builds #38988

Closed
1 task done
Taragolis opened this issue Apr 13, 2024 · 0 comments · Fixed by #39495
Closed
1 task done

Flaky docker example test around mpi4py builds #38988

Taragolis opened this issue Apr 13, 2024 · 0 comments · Fixed by #39495
Labels
kind:meta High-level information important to the community Quarantine Issues that are occasionally failing and are quarantined

Comments

@Taragolis
Copy link
Contributor

Body

Our test with build example docker image (https://airflow.apache.org/docs/docker-stack/build.html#example-when-you-add-packages-requiring-compilation) started failed recently with build error

#6 12.80 Building wheels for collected packages: mpi4py
#6 12.80   Building wheel for mpi4py (pyproject.toml): started
#6 13.71   Building wheel for mpi4py (pyproject.toml): finished with status 'error'
#6 13.74   error: subprocess-exited-with-error
#6 13.74   
#6 13.74   × Building wheel for mpi4py (pyproject.toml) did not run successfully.
#6 13.74   │ exit code: 1
#6 13.74   ╰─> [212 lines of output]
#6 13.74       running bdist_wheel
#6 13.74       running build
#6 13.74       running build_src
#6 13.74       running build_py
#6 13.74       creating build
#6 13.74       creating build/lib.linux-x86_64-cpython-38
#6 13.74       creating build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/run.py -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/bench.py -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       creating build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/_lib.py -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       creating build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/dl.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/py.typed -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-cpython-38/mpi4py
#6 13.74       creating build/lib.linux-x86_64-cpython-38/mpi4py/include
#6 13.74       creating build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py
#6 13.74       copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py
#6 13.74       copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py
#6 13.74       copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py
#6 13.74       copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py
#6 13.74       copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-cpython-38/mpi4py/include/mpi4py
#6 13.74       copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/_lib.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/futures
#6 13.74       copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-cpython-38/mpi4py/util
#6 13.74       running build_clib
#6 13.74       MPI configuration: [mpi] from 'mpi.cfg'
#6 13.74       MPI C compiler:    /usr/bin/mpicc
#6 13.74       MPI C++ compiler:  /usr/bin/mpicxx
#6 13.74       MPI F compiler:    /usr/bin/mpifort
#6 13.74       MPI F90 compiler:  /usr/bin/mpif90
#6 13.74       MPI F77 compiler:  /usr/bin/mpif77
#6 13.74       checking for library 'lmpe' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -llmpe -o _configtest
#6 13.74       /usr/bin/ld: cannot find -llmpe: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       building 'mpe' dylib library
#6 13.74       creating build/temp.linux-x86_64-cpython-38
#6 13.74       creating build/temp.linux-x86_64-cpython-38/src
#6 13.74       creating build/temp.linux-x86_64-cpython-38/src/lib-pmpi
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/mpe.o
#6 13.74       creating build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi
#6 13.74       /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/mpe.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libmpe.so
#6 13.74       checking for library 'vt-mpi' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -lvt-mpi -o _configtest
#6 13.74       /usr/bin/ld: cannot find -lvt-mpi: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       checking for library 'vt.mpi' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -lvt.mpi -o _configtest
#6 13.74       /usr/bin/ld: cannot find -lvt.mpi: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       building 'vt' dylib library
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/vt.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt.o
#6 13.74       /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libvt.so
#6 13.74       checking for library 'vt-mpi' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -lvt-mpi -o _configtest
#6 13.74       /usr/bin/ld: cannot find -lvt-mpi: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       checking for library 'vt.mpi' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -lvt.mpi -o _configtest
#6 13.74       /usr/bin/ld: cannot find -lvt.mpi: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       building 'vt-mpi' dylib library
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-mpi.o
#6 13.74       /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-mpi.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libvt-mpi.so
#6 13.74       checking for library 'vt-hyb' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -lvt-hyb -o _configtest
#6 13.74       /usr/bin/ld: cannot find -lvt-hyb: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       checking for library 'vt.ompi' ...
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c _configtest.c -o _configtest.o
#6 13.74       /usr/bin/mpicc _configtest.o -lvt.ompi -o _configtest
#6 13.74       /usr/bin/ld: cannot find -lvt.ompi: No such file or directory
#6 13.74       collect2: error: ld returned 1 exit status
#6 13.74       failure.
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       building 'vt-hyb' dylib library
#6 13.74       /usr/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c src/lib-pmpi/vt-hyb.c -o build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-hyb.o
#6 13.74       /usr/bin/mpicc -shared -Wl,--no-as-needed build/temp.linux-x86_64-cpython-38/src/lib-pmpi/vt-hyb.o -o build/lib.linux-x86_64-cpython-38/mpi4py/lib-pmpi/libvt-hyb.so
#6 13.74       running build_ext
#6 13.74       MPI configuration: [mpi] from 'mpi.cfg'
#6 13.74       MPI C compiler:    /usr/bin/mpicc
#6 13.74       MPI C++ compiler:  /usr/bin/mpicxx
#6 13.74       MPI F compiler:    /usr/bin/mpifort
#6 13.74       MPI F90 compiler:  /usr/bin/mpif90
#6 13.74       MPI F77 compiler:  /usr/bin/mpif77
#6 13.74       checking for dlopen() availability ...
#6 13.74       checking for header 'dlfcn.h' ...
#6 13.74       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/airflow/.local/include -I/usr/local/include/python3.8 -c _configtest.c -o _configtest.o
#6 13.74       success!
#6 13.74       removing: _configtest.c _configtest.o
#6 13.74       success!
#6 13.74       checking for library 'dl' ...
#6 13.74       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/airflow/.local/include -I/usr/local/include/python3.8 -c _configtest.c -o _configtest.o
#6 13.74       Traceback (most recent call last):
#6 13.74         File "/home/airflow/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#6 13.74           main()
#6 13.74         File "/home/airflow/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#6 13.74           json_out['return_val'] = hook(**hook_input['kwargs'])
#6 13.74         File "/home/airflow/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
#6 13.74           return _build_backend().build_wheel(wheel_directory, config_settings,
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 410, in build_wheel
#6 13.74           return self._build_with_temp_dir(
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
#6 13.74           self.run_setup()
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
#6 13.74           exec(code, locals())
#6 13.74         File "<string>", line 644, in <module>
#6 13.74         File "<string>", line 641, in main
#6 13.74         File "<string>", line 492, in run_setup
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 541, in setup
#6 13.74           return fcn_setup(**attrs)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 104, in setup
#6 13.74           return distutils.core.setup(**attrs)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
#6 13.74           return run_commands(dist)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
#6 13.74           dist.run_commands()
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#6 13.74           self.run_command(cmd)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
#6 13.74           super().run_command(command)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#6 13.74           cmd_obj.run()
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 368, in run
#6 13.74           self.run_command("build")
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
#6 13.74           self.distribution.run_command(command)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
#6 13.74           super().run_command(command)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#6 13.74           cmd_obj.run()
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
#6 13.74           self.run_command(cmd_name)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
#6 13.74           self.distribution.run_command(command)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
#6 13.74           super().run_command(command)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#6 13.74           cmd_obj.run()
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line [1213](https://github.com/apache/airflow/actions/runs/8674524298/job/23787271307#step:9:1214), in run
#6 13.74           cmd_build_ext.build_ext.run(self)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 91, in run
#6 13.74           _build_ext.run(self)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
#6 13.74           self.build_extensions()
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 1241, in build_extensions
#6 13.74           self.build_extension(ext)
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 1272, in build_extension
#6 13.74           self.config_extension(ext)
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 1255, in config_extension
#6 13.74           configure(ext, config_cmd)
#6 13.74         File "<string>", line 188, in configure_dl
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 607, in check_library
#6 13.74           ok = self.try_link(body,  headers, include_dirs,
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/config.py", line 246, in try_link
#6 13.74           self._link(body, headers, include_dirs, libraries, library_dirs, lang)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/config.py", line 138, in _link
#6 13.74           self.compiler.link_executable(
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line 781, in link_executable
#6 13.74           self.link(
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 233, in link
#6 13.74           lib_opts = gen_lib_options(self, library_dirs, runtime_library_dirs, libraries)
#6 13.74         File "/tmp/pip-build-env-q8hjjo_2/overlay/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line [1232](https://github.com/apache/airflow/actions/runs/8674524298/job/23787271307#step:9:1233), in gen_lib_options
#6 13.74           lib_opts.extend(always_iterable(compiler.runtime_library_dir_option(dir)))
#6 13.74         File "/tmp/pip-install-k1zjj31l/mpi4py_42b09b3638ba45588d66a70d447ce5d7/conf/mpidistutils.py", line 68, in rpath_option
#6 13.74           if option.startswith('-R'):
#6 13.74       AttributeError: 'list' object has no attribute 'startswith'
#6 13.74       [end of output]
#6 13.74

We should resolve this issue, potentially the solution might be appear into the upstream issue task:

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:meta High-level information important to the community Quarantine Issues that are occasionally failing and are quarantined
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Unquarantine docker example test around mpi4py builds Taragolis/airflow
1 participant
@Taragolis