Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyArrow dependency broken for TFX #2811

Closed
numerology opened this issue Jan 7, 2020 · 2 comments · Fixed by #2813
Closed

PyArrow dependency broken for TFX #2811

numerology opened this issue Jan 7, 2020 · 2 comments · Fixed by #2813

Comments

@numerology
Copy link

Temporarily disable TFX test

@numerology
Copy link
Author

Investigation shows that TFX has the requirement PyArrow < 0.15, but we somehow get PyArrow 0.15.1

@numerology
Copy link
Author

Currently observed two failure mode:

  1. backend server image failure, for example:
�[0m�[91mTraceback (most recent call last):
  File "core/parameterized_tfx_oss/parameterized_tfx_oss.py", line 22, in <module>
    from tfx.components.evaluator.component import Evaluator
  File "/usr/local/lib/python3.5/site-packages/tfx/components/__init__.py", line 27, in <module>
    from tfx.components.example_validator.component import ExampleValidator
  File "/usr/local/lib/python3.5/site-packages/tfx/components/example_validator/component.py", line 24, in <module>
    from tfx.components.example_validator import executor
  File "/usr/local/lib/python3.5/site-packages/tfx/components/example_validator/executor.py", line 22, in <module>
    import tensorflow_data_validation as tfdv
  File "/usr/local/lib/python3.5/site-packages/tensorflow_data_validation/__init__.py", line 24, in <module>
    from tensorflow_data_validation.api.stats_api import GenerateStatistics
  File "/usr/local/lib/python3.5/site-packages/tensorflow_data_validation/api/stats_api.py", line 52, in <module>
    from tensorflow_data_validation.statistics import stats_impl
  File "/usr/local/lib/python3.5/site-packages/tensorflow_data_validation/statistics/stats_impl.py", line 32, in <module>
    from tensorflow_data_validation.arrow import arrow_util
  File "/usr/local/lib/python3.5/site-packages/tensorflow_data_validation/arrow/arrow_util.py", line 24, in <module>
    from tfx_bsl.arrow import array_util
  File "/usr/local/lib/python3.5/site-packages/tfx_bsl/arrow/array_util.py", line 17, in <module>
    from tfx_bsl.cc.tfx_bsl_extension.arrow.array_util import *
ImportError: libarrow.so.14: cannot open shared object file: No such file or directory
The command '/bin/sh -c set -e; find core -maxdepth 2 -name '*.py' -type f | while read pipeline; do python3 $pipeline; done' returned a non-zero code: 1

This is because in RUN python3 -m pip install tfx==0.15.0, pyarrow 0.15.1 was installed. However, tfx-bsl requires pyarrow<0.15.0 as stated here

  1. Sample test failure, for example.

My guess is that somehow there is another dependency of TFX shadowing the tfx-bsl requirements. Thus one solution can be we pin down the PyArrow version temporarily before TFX 0.21

magdalenakuhn17 pushed a commit to magdalenakuhn17/pipelines that referenced this issue Oct 22, 2023
…low#2811)

* Update AIX example to use custom explainer spec

Signed-off-by: Tommy Li <Tommy.chaoping.li@ibm.com>

* Update aix-explainer.yaml

* Remove predictor host and http port args

---------

Signed-off-by: Tommy Li <Tommy.chaoping.li@ibm.com>
Co-authored-by: Dan Sun <dsun20@bloomberg.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant