Failure to install GPflow alongside Tensorflow 2.16.1

[RiccardoBuscicchio]

Build issue

Failure to install GPflow dependencies alongside Tensorflow 2.16.1

To reproduce

Minimal, reproducible example
I created a minimal conda environment with python=3.10 only

conda create --name gpflowenv python=3.10 --no-default-packages

Then activated it and installed gpflow, assuming all the dependencies are dealt with.

conda activate gpflowenv
pip install gpflow

Afterwards, I check the relevant packages version

import tensorflow as tf 
import keras 
print(tf.__version__) 
print(keras.__version__) 

which gives

 '2.16.1' '3.1.1' 

However when I import gpflow I can a missing package (sympy) error.
After installing it via pip, I get another missing package error tf_keras (see the stack trace below).
After pip install tf-keras (version 2.16.0), importing gpflow doesn't return any error, only a warning about missing TensorRT (see second slack trace below).

I'd like to know whether GPflow is correctly installed as it is.

Stack trace, or error message

Missing tf_keras

>> import gpflow
2024-03-24 16:58:02.638116: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-24 16:58:02.638473: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-03-24 16:58:02.640577: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-03-24 16:58:02.669390: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-24 16:58:03.120969: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Failed to import TF-Keras. Please note that TF-Keras is not installed by default when you install TensorFlow Probability. This is so that JAX-only users do not have to install TensorFlow or TF-Keras. To use TensorFlow Probability with TensorFlow, please install the tf-keras or tf-keras-nightly package.
This can be be done through installing the tensorflow-probability[tf] extra.

Failed to import TF-Keras. Please note that TF-Keras is not installed by default when you install TensorFlow Probability. This is so that JAX-only users do not have to install TensorFlow or TF-Keras. To use TensorFlow Probability with TensorFlow, please install the tf-keras or tf-keras-nightly package.
This can be be done through installing the tensorflow-probability[tf] extra.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/gpflow/__init__.py", line 17, in <module>
    from . import (
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/gpflow/conditionals/__init__.py", line 1, in <module>
    from . import conditionals, multioutput, sample_conditionals
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/gpflow/conditionals/conditionals.py", line 20, in <module>
    from ..base import MeanAndVariance
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/gpflow/base.py", line 20, in <module>
    import tensorflow_probability as tfp
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/tensorflow_probability/__init__.py", line 22, in <module>
    from tensorflow_probability.python import *  # pylint: disable=wildcard-import
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/tensorflow_probability/python/__init__.py", line 152, in <module>
    dir(globals()[pkg_name])  # Forces loading the package from its lazy loader.
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 57, in __dir__
    module = self._load()
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 37, in _load
    self._on_first_access()
  File "/home/riccardo/Software/anaconda3/envs/gpflowenv/lib/python3.10/site-packages/tensorflow_probability/python/__init__.py", line 79, in _validate_tf_environment
    import tf_keras  # pylint: disable=unused-import
ModuleNotFoundError: No module named 'tf_keras'
>>> 

*** After tf_keras 2.16.0 is installed ***

>>> import gpflow
2024-03-24 17:03:07.398803: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-24 17:03:07.399160: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-03-24 17:03:07.401240: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-03-24 17:03:07.431572: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-24 17:03:07.869311: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>>

Expected behavior

I would expect sympy and tf_keras to be set as requirements.

System information

• GPflow version: 2.9.1
• GPflow installed from: pip install gpflow
• TensorFlow version: 2.16.1
• Python version: 3.10
• Operating system: Ubuntu 22.04

[uri-granta]

Thanks for raising this! The latest tested supported version is TF 2.15. We will try to get to supporting TF 2.16 soon.

[uri-granta]

FYI I've started looking at this. While we resolve this, it may be possible to get the most recent release of gpflow working with tensorflow 2.16.1 by doing two things:

• run pip install tensorflow-probability[tf]==0.24.0 to install the correct version of tf_keras
• run export TF_USE_LEGACY_KERAS=1 before starting python to switch tf to using Keras 2

[RiccardoBuscicchio]

Thanks for the feedback uri-granta. The temporary workaround works on my system.