[Import error] Possible conflicts with tensorflow==2.16.1, tensorflow_probaobility==0.23.0, causalimpact==0.2.0 on python==3.11.8 and python==3.12.2

[sayantanghosh]

Dear tfp-causalimpact team,

Thank you for a fantastic package. This is just a note for fresh installs in a conda environment.

Premise

I noticed something this morning while trying to install the package on a fresh conda environment (conda 23.11.0) with python==3.11.8. Running pip install tfp-causalimpact installs the latest version of tensorflow==2.16.1, and tensorflow_probability==0.23.0, which while installing without any errors, produces errors while importing (arising from tfp).

I have also included a possible workaround below.

Steps to reproduce

For python==3.11.8

conda --version
conda 23.11.0
conda create -n causalimpact python==3.11.8
pip install tfp-causalimpact ipython jupyter 
ipython

import causalimpact
import pandas as pd
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

For python==3.12.2

The same problem is also reproduced with the following configuration:

%watermark -v -m -p numpy,scipy,tensorflow,keras
Python implementation: CPython
Python version       : 3.12.2
IPython version      : 8.22.2

numpy     : 1.26.4
scipy     : not installed
tensorflow: 2.16.1
keras     : 3.0.5

Compiler    : Clang 14.0.6 
OS          : Darwin
Release     : 22.6.0
Machine     : x86_64
Processor   : i386
CPU cores   : 4
Architecture: 64bit

Issue

There seems to be some clash between these tensorflow_probability==0.23.0 and tensorflow==2.16.1, which causes import error for causalimpact and here is the error log:

In [1]: import causalimpact
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 1
----> 1 import causalimpact

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/causalimpact/__init__.py:29
     26 os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
     28 # pylint: disable=g-import-not-at-top
---> 29 from causalimpact.causalimpact_lib import CausalImpactAnalysis
     30 from causalimpact.causalimpact_lib import DataOptions
     31 from causalimpact.causalimpact_lib import fit_causalimpact

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/causalimpact/causalimpact_lib.py:23
     20 import math
     21 from typing import Dict, List, Optional, Tuple, Union
---> 23 from causalimpact import posterior_processing
     24 import causalimpact.data as cid
     25 from causalimpact.indices import InputDateType

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/causalimpact/posterior_processing.py:20
     16 """Library for working with (results from) the posterior."""
     18 from typing import List, Text, Tuple
---> 20 from causalimpact import data as cid
     21 import numpy as np
     22 import pandas as pd

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/causalimpact/data.py:23
     21 import pandas as pd
     22 import tensorflow as tf
---> 23 import tensorflow_probability as tfp
     26 class CausalImpactData:
     27   """Class for storing and preparing data for modeling.
     28 
     29   This class handles all of the data-related functions of CausalImpact. It
   (...)
     74       `pre_period`.
     75   """

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/__init__.py:20
     15 """Tools for probabilistic reasoning in TensorFlow."""
     17 # Contributors to the `python/` dir should not alter this file; instead update
     18 # `python/__init__.py` as necessary.
---> 20 from tensorflow_probability import substrates
     21 # from tensorflow_probability.google import staging  # DisableOnExport
     22 # from tensorflow_probability.google import tfp_google  # DisableOnExport
     23 from tensorflow_probability.python import *  # pylint: disable=wildcard-import

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/substrates/__init__.py:17
      1 # Copyright 2019 The TensorFlow Probability Authors.
      2 #
      3 # Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     13 # limitations under the License.
     14 # ============================================================================
     15 """TensorFlow Probability alternative substrates."""
---> 17 from tensorflow_probability.python.internal import all_util
     18 from tensorflow_probability.python.internal import lazy_loader  # pylint: disable=g-direct-tensorflow-import
     21 jax = lazy_loader.LazyLoader(
     22     'jax', globals(),
     23     'tensorflow_probability.substrates.jax')

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/__init__.py:138
    135 if _tf_loaded():
    136   # Non-lazy load of packages that register with tensorflow or keras.
    137   for pkg_name in _maybe_nonlazy_load:
--> 138     dir(globals()[pkg_name])  # Forces loading the package from its lazy loader.
    141 all_util.remove_undocumented(__name__, _lazy_load + _maybe_nonlazy_load)

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/internal/lazy_loader.py:57, in LazyLoader.__dir__(self)
     56 def __dir__(self):
---> 57   module = self._load()
     58   return dir(module)

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/internal/lazy_loader.py:40, in LazyLoader._load(self)
     38   self._on_first_access = None
     39 # Import the target module and insert it into the parent's namespace
---> 40 module = importlib.import_module(self.__name__)
     41 if self._parent_module_globals is not None:
     42   self._parent_module_globals[self._local_name] = module

File ~/anaconda3/envs/tf2161/lib/python3.11/importlib/__init__.py:126, in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/experimental/__init__.py:31
     15 """TensorFlow Probability API-unstable package.
     16 
     17 This package contains potentially useful code which is under active development
   (...)
     27 You are welcome to try any of this out (and tell us how well it works for you!).
     28 """
     30 from tensorflow_probability.python.experimental import auto_batching
---> 31 from tensorflow_probability.python.experimental import bayesopt
     32 from tensorflow_probability.python.experimental import bijectors
     33 from tensorflow_probability.python.experimental import distribute

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/experimental/bayesopt/__init__.py:17
      1 # Copyright 2023 The TensorFlow Probability Authors.
      2 #
      3 # Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     13 # limitations under the License.
     14 # ============================================================================
     15 """TensorFlow Probability experimental Bayesopt package."""
---> 17 from tensorflow_probability.python.experimental.bayesopt import acquisition
     18 from tensorflow_probability.python.internal import all_util
     20 _allowed_symbols = [
     21     'acquisition',
     22 ]

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/experimental/bayesopt/acquisition/__init__.py:19
     17 from tensorflow_probability.python.experimental.bayesopt.acquisition.acquisition_function import AcquisitionFunction
     18 from tensorflow_probability.python.experimental.bayesopt.acquisition.acquisition_function import MCMCReducer
---> 19 from tensorflow_probability.python.experimental.bayesopt.acquisition.expected_improvement import GaussianProcessExpectedImprovement
     20 from tensorflow_probability.python.experimental.bayesopt.acquisition.expected_improvement import ParallelExpectedImprovement
     21 from tensorflow_probability.python.experimental.bayesopt.acquisition.expected_improvement import StudentTProcessExpectedImprovement

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/experimental/bayesopt/acquisition/expected_improvement.py:19
     15 """Expected Improvement."""
     17 import tensorflow.compat.v2 as tf
---> 19 from tensorflow_probability.python.distributions import normal
     20 from tensorflow_probability.python.distributions import student_t
     21 from tensorflow_probability.python.experimental.bayesopt.acquisition import acquisition_function

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/distributions/__init__.py:110
    108 from tensorflow_probability.python.distributions.pareto import Pareto
    109 from tensorflow_probability.python.distributions.pert import PERT
--> 110 from tensorflow_probability.python.distributions.pixel_cnn import PixelCNN
    111 from tensorflow_probability.python.distributions.plackett_luce import PlackettLuce
    112 from tensorflow_probability.python.distributions.poisson import Poisson

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/distributions/pixel_cnn.py:33
     31 from tensorflow_probability.python.internal import reparameterization
     32 from tensorflow_probability.python.internal import tensorshape_util
---> 33 from tensorflow_probability.python.layers import weight_norm
     36 class PixelCNN(distribution.Distribution):
     37   """The Pixel CNN++ distribution.
     38 
     39   Pixel CNN++ [(Salimans et al., 2017)][1] models a distribution over image
   (...)
    228        Learning_, 2016. https://arxiv.org/pdf/1601.06759.pdf
    229   """

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/layers/__init__.py:27
     25 from tensorflow_probability.python.layers.dense_variational import DenseReparameterization
     26 from tensorflow_probability.python.layers.dense_variational_v2 import DenseVariational
---> 27 from tensorflow_probability.python.layers.distribution_layer import CategoricalMixtureOfOneHotCategorical
     28 from tensorflow_probability.python.layers.distribution_layer import DistributionLambda
     29 from tensorflow_probability.python.layers.distribution_layer import IndependentBernoulli

File ~/anaconda3/envs/tf2161/lib/python3.11/site-packages/tensorflow_probability/python/layers/distribution_layer.py:68
     47 from tensorflow_probability.python.layers.internal import tensor_tuple
     50 __all__ = [
     51     'CategoricalMixtureOfOneHotCategorical',
     52     'DistributionLambda',
   (...)
     64     'VariationalGaussianProcess',
     65 ]
---> 68 tf.keras.__internal__.utils.register_symbolic_tensor_type(dtc._TensorCoercible)  # pylint: disable=protected-access
     71 def _event_size(event_shape, name=None):
     72   """Computes the number of elements in a tensor with shape `event_shape`.
     73 
     74   Args:
   (...)
     82     a scalar tensor.
     83   """

AttributeError: module 'keras._tf_keras.keras' has no attribute '__internal__'

Note that conda's latest python==3.12.2

However everything works when I downgrade to tensorflow==2.15.0.

Possible workaround

So a possible solution is to use
pip install tensorflow==2.15.0 tfp-causalimpact.

Would it be possible for the team to publish a REQUIREMENTS.txt with a frozen tensorflow==2.15.0 version, until things get sorted with tensorflow==2.16.1 and tensorflow_probability==0.23.0?

My system's settings that work are:

%watermark -v -m -p numpy,scipy,tensorflow,tensorflow_probability,keras,causalimpact

Python implementation: CPython
Python version       : 3.11.8
IPython version      : 8.20.0

numpy                 : 1.26.4
scipy                 : 1.12.0
tensorflow            : 2.15.0
tensorflow_probability: 0.23.0
keras                 : 2.15.0
causalimpact          : 0.2.0

Compiler    : Clang 16.0.6 
OS          : Darwin
Release     : 22.6.0
Machine     : x86_64
Processor   : i386
CPU cores   : 4
Architecture: 64bit

Hope you guys can look into it, and maybe this will help someone else stumbling on this same issue.