GitHub Actions only error: AttributeError("'NoneType' object has no attribute '_sorted_indices'")

[matthewfeickert]

Summary

In PR #1274 there is a AttributeError("'NoneType' object has no attribute '_sorted_indices'") that is happening for Python 3.10 only for unclear reasons. This is blocking release v0.7.0. This is also not reproducible locally on my laptop.

OS / Environment

GitHub Actions on Python 3.10

Steps to Reproduce

$ pytest src/pyhf/tensor/jax_backend.py

File Upload (optional)

No response

Expected Results

For the CI to pass

Actual Results

=================================== FAILURES ===================================
____________ [doctest] pyhf.tensor.jax_backend.jax_backend.astensor ____________
207 
208         Convert to a JAX ndarray.
209 
210         Example:
211 
212             >>> import pyhf
213             >>> pyhf.set_backend("jax")
UNEXPECTED EXCEPTION: AttributeError("'NoneType' object has no attribute '_sorted_indices'")
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest pyhf.tensor.jax_backend.jax_backend.astensor[1]>", line 1, in <module>
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/events.py", line 161, in register_wrapper
    result = func(*args, **kwargs)
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/tensor/manager.py", line 192, in set_backend
    events.trigger("tensorlib_changed")()
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/events.py", line 70, in __call__
    func()(arg(), *args, **kwargs)
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/tensor/common.py", line 33, in _precompute
    self.sorted_indices = tensorlib.astensor(self._sorted_indices, dtype='int')
AttributeError: 'NoneType' object has no attribute '_sorted_indices'
/Users/runner/work/pyhf/pyhf/src/pyhf/tensor/jax_backend.py:213: UnexpectedException
------------------------------ Captured log call -------------------------------
INFO     absl:xla_bridge.py:174 Remote TPU is not linked into jax; skipping remote TPU.
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'

pyhf Version

PR #1274

Code of Conduct

• I agree to follow the Code of Conduct

[matthewfeickert]

@kratsg @lukasheinrich If I run our CI from scratch inside of a python:3.10 Docker container I am unable to reproduce the error, which makes me think we're seeing a transient issue with GitHub Actions and not something real on our side.

Should we just go ahead and merge PR #1274 and hope for the best while opening up some GitHub Issues with the virtual environment team? Though this isn't happening on master at f7e974f, so there is something in the PR #1274 code that is causing this on the GitHub Action runners. :?

[matthewfeickert]

Expanding on #1274 (comment), this is able to be reproduced locally if one comments out one of the docstrings of any of the 3 functions under the pyhf.infer.intervals.upper_limits API

> git diff
diff --git a/src/pyhf/infer/intervals/upper_limits.py b/src/pyhf/infer/intervals/upper_limits.py
index 5cc93c13..c609091b 100644
--- a/src/pyhf/infer/intervals/upper_limits.py
+++ b/src/pyhf/infer/intervals/upper_limits.py
@@ -33,22 +33,22 @@ def toms748_scan(
     Parameter of Interest (POI) using an automatic scan through
     POI-space, using the :func:`~scipy.optimize.toms748` algorithm.
 
-    Example:
-        >>> import numpy as np
-        >>> import pyhf
-        >>> pyhf.set_backend("numpy")
-        >>> model = pyhf.simplemodels.uncorrelated_background(
-        ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
-        ... )
-        >>> observations = [51, 48]
-        >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
-        >>> obs_limit, exp_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
-        ...     data, model, 0., 5., rtol=0.01
-        ... )
-        >>> obs_limit
-        array(1.01156939)
-        >>> exp_limits
-        [array(0.5600747), array(0.75702605), array(1.06234693), array(1.50116923), array(2.05078912)]
+    # Example:
+    #     >>> import numpy as np
+    #     >>> import pyhf
+    #     >>> pyhf.set_backend("numpy")
+    #     >>> model = pyhf.simplemodels.uncorrelated_background(
+    #     ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
+    #     ... )
+    #     >>> observations = [51, 48]
+    #     >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
+    #     >>> obs_limit, exp_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
+    #     ...     data, model, 0., 5., rtol=0.01
+    #     ... )
+    #     >>> obs_limit
+    #     array(1.01156939)
+    #     >>> exp_limits
+    #     [array(0.5600747), array(0.75702605), array(1.06234693), array(1.50116923), array(2.05078912)]
 
     Args:
         data (:obj:`tensor`): The observed data.
@@ -231,7 +231,6 @@ def upper_limit(
         array(1.01764089)
         >>> exp_limits
         [array(0.59576921), array(0.76169166), array(1.08504773), array(1.50170482), array(2.06654952)]
-        >>> pyhf.set_backend("jax")
 
     Args:
         data (:obj:`tensor`): The observed data.

and then run

$ pytest \
          src/pyhf/infer/intervals/ \
          src/pyhf/tensor/jax_backend.py
================================================================================= test session starts =================================================================================
platform linux -- Python 3.10.6, pytest-7.1.3, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Matplotlib: 3.6.0
Freetype: 2.6.1
rootdir: /home/feickert/Code/GitHub/pyhf, configfile: pyproject.toml
plugins: socket-0.5.1, console-scripts-1.3.1, benchmark-3.4.1, cov-3.0.0, mock-3.8.2, mpl-0.16.1, requests-mock-1.10.0, anyio-3.6.1
collected 18 items                                                                                                                                                                    

src/pyhf/infer/intervals/upper_limits.py ..                                                                                                                                     [ 11%]
src/pyhf/tensor/jax_backend.py F...............                                                                                                                                 [100%]

====================================================================================== FAILURES =======================================================================================
_______________________________________________________________ [doctest] pyhf.tensor.jax_backend.jax_backend.astensor ________________________________________________________________
207 
208         Convert to a JAX ndarray.
209 
210         Example:
211 
212             >>> import pyhf
213             >>> pyhf.set_backend("jax")
UNEXPECTED EXCEPTION: AttributeError("'NoneType' object has no attribute '_sorted_indices'")
Traceback (most recent call last):
  File "/home/feickert/.pyenv/versions/3.10.6/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest pyhf.tensor.jax_backend.jax_backend.astensor[1]>", line 1, in <module>
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/events.py", line 161, in register_wrapper
    result = func(*args, **kwargs)
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/manager.py", line 192, in set_backend
    events.trigger("tensorlib_changed")()
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/events.py", line 70, in __call__
    func()(arg(), *args, **kwargs)
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/common.py", line 33, in _precompute
    self.sorted_indices = tensorlib.astensor(self._sorted_indices, dtype='int')
AttributeError: 'NoneType' object has no attribute '_sorted_indices'
/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/jax_backend.py:213: UnexpectedException
---------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker: 
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.
WARNING  absl:xla_bridge.py:357 No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

I think this might mean we have a bug in pyhf.tensor.manager or in how

pyhf/tests/conftest.py

Lines 49 to 56 in f7e974f

	@pytest.fixture(scope='function', autouse=True)
	def reset_backend():
	"""
	This fixture is automatically run to reset the backend before and after a test function runs.
	"""
	pyhf.set_backend('numpy', default=True)
	yield reset_backend
	pyhf.set_backend('numpy', default=True)

is used by doctest. Not sure, so input welcome.

[matthewfeickert]

Here's the issue described in the pytest How to run doctests docs:

Note that like the normal conftest.py, the fixtures are discovered in the directory tree conftest is in. Meaning that if you put your doctest with your source code, the relevant conftest.py needs to be in the same directory tree. Fixtures will not be discovered in a sibling directory tree!

So if doctest is invoked with

$ pytest src/

then tests/conftest.py isn't picked up and none of the autouse pytest fixtures in tests/conftest.py are then available for use in the doctests.