Skip to content

[Bug]: CI regression job fails at pytest startup — unpinned arize-phoenix pulls a broken release #304

Description

@ango10

What happened?

The regression.yml CI job fails on every PR as of 2026-08-05. pytest crashes during
plugin loading — before collecting a single test — so the failure is unrelated to whatever
the PR changed.

pyproject.toml declares the otel extra as:

otel = [
  "arize-phoenix>=15.0.0",
  ...
]

There is no upper bound. regression.yml installs -e ".[dev,otel]", so CI resolves to
the newest arize-phoenix on every run. 19.18.0 was published on 2026-08-05 — the same day
the failures started.

A pytest11 setuptools entrypoint imports phoenix, whose phoenix/trace/dsl/filter.py
declares a frozen dataclass with a mappingproxy default for boolean_names. Python's
dataclasses rejects unhashable defaults, so the import raises and takes pytest down with it.

This is a supply-chain exposure, not a one-off: any unpinned upper bound lets an upstream
release break CI for every open PR with no change on our side.

What did you expect to happen?

pytest tests/ -x -q should collect and run the suite. CI should not be able to break because
a third-party dependency published a new release, with no change to this repo.

Reproduction steps

  1. Check out main (no local changes needed — pyproject.toml on main already has the
    unpinned dependency).
  2. On Python 3.11: pip install -e ".[dev,otel]"
  3. pytest tests/ -x -q
  4. pytest exits 1 during load_setuptools_entrypoints("pytest11"); zero tests run.

No eval config is involved — this fails before collection.

Version info

assert_ai 0.1.0, Python 3.11.15 (ubuntu-latest, actions/setup-python "3.11"), arize-phoenix resolved to latest (19.18.0)

Logs

File ".../_pytest/config/init.py", line 1583, in parse
self.pluginmanager.load_setuptools_entrypoints("pytest11")
...
File ".../phoenix/trace/dsl/filter.py", line 190, in
DataClass(frozen=True)
...
File ".../dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'mappingproxy'> for field boolean_names is not allowed: use default_factory

Proposed fix

Add an upper bound in pyproject.toml:

"arize-phoenix>=15.0.0,<19.18",

Prefer this over PYTEST_DISABLE_PLUGIN_AUTOLOAD=1, which would also drop pytest-timeout
(installed by the dev extra) and would mask the next upstream break rather than surface it.

Worth auditing the other unbounded pins in the same file while we're here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions