Unit tests fail due to protobuf error: TypeError: Descriptors cannot not be created directly

[matthayes]

Expected Behavior

Unit tests should pass

Current Behavior

I have two consecutive builds where the only difference between them is a change in the Python comments. The first succeeds and the second fails.

succeeds: https://github.com/databrickslabs/dbx/runs/6516812865?check_suite_focus=true
fails: https://github.com/databrickslabs/dbx/runs/6519187157?check_suite_focus=true

The failing build has several test failures that look like this:

_________________ ERROR collecting tests/unit/test_execute.py __________________
tests/unit/test_execute.py:6: in <module>
    from mlflow import ActiveRun
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/__init__.py:32: in <module>
    import mlflow.tracking._model_registry.fluent
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/tracking/__init__.py:8: in <module>
    from mlflow.tracking.client import MlflowClient
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/tracking/client.py:16: in <module>
    from mlflow.entities import Experiment, Run, RunInfo, Param, Metric, RunTag, FileInfo, ViewType
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/entities/__init__.py:6: in <module>
    from mlflow.entities.experiment import Experiment
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/entities/experiment.py:2: in <module>
    from mlflow.entities.experiment_tag import ExperimentTag
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/entities/experiment_tag.py:2: in <module>
    from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/protos/service_pb2.py:18: in <module>
    from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/mlflow-1.26.0-py3.8.egg/mlflow/protos/scalapb/scalapb_pb2.py:29: in <module>
    options = _descriptor.FieldDescriptor(
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/protobuf-4.21.0rc2-py3.8-linux-x86_64.egg/google/protobuf/descriptor.py:560: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot not be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

I compare the packages that are installed between these two versions.

Before:

Installed /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/alembic-1.7.7-py3.8.egg
Searching for protobuf>=3.7.0
Reading https://pypi.org/simple/protobuf/
Downloading https://files.pythonhosted.org/packages/f9/b1/7345c0a1264808cbea2a99d2e7fce0e94cbfa76a2737929f2a3f017d3e85/protobuf-4.21.0_rc_1-py3-none-any.whl#sha256=97933682fefe98b3bf7651ec7a5d6b0096df12335b903863d4523b39612e09a1
Best match: protobuf 4.21.0rc1
Processing protobuf-4.21.0_rc_1-py3-none-any.whl
Installing protobuf-4.21.0_rc_1-py3-none-any.whl to /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages
Adding protobuf 4.21.0rc1 to easy-install.pth file

After:

Installed /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/alembic-1.7.7-py3.8.egg
Searching for protobuf>=3.7.0
Reading https://pypi.org/simple/protobuf/
Downloading https://files.pythonhosted.org/packages/ba/d7/8fad2bb0536b78cefc13f9f83a97cfc4c7fcb29ddfc8bc209162f26d61f7/protobuf-4.21.0_rc_2-cp37-abi3-manylinux2014_x86_64.whl#sha256=098a01ee1ddf94aac69e3a607cb5cd74101a8db1a0ab065d46cb5a37a1526fee
Best match: protobuf 4.21.0rc2
Processing protobuf-4.21.0_rc_2-cp37-abi3-manylinux2014_x86_64.whl
Installing protobuf-4.21.0_rc_2-cp37-abi3-manylinux2014_x86_64.whl to /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages
Adding protobuf 4.21.0rc2 to easy-install.pth file

It seems the issue is that a new protobuf version has been released, which the second build is now using. The protobuf version went from 4.21.0rc1 to 4.21.0rc2. Presumably there were some files generated with the older version that no longer work with the newer version.

Steps to Reproduce (for bugs)

Run unit tests in CI

Context

N/A

Your Environment

N/A

[matthayes]

Note that 4.21.0rc2 was just released 6 hours ago as of this moment:

https://pypi.org/project/protobuf/#history

[matthayes]

mlflow requires protobuf>=3.7.0, which is how the newer version is getting picked up:

https://github.com/mlflow/mlflow/blob/master/setup.py#L51

[matthayes]

Addressed in #256

[renardeinside]

Covered in the relevant PR.