Skip to content

Downgrading mlflow to avoid protobuf issues on nightly builds.#3784

Merged
vfdev-5 merged 5 commits into
pytorch:masterfrom
aaishwarymishra:failing-ci
Jun 9, 2026
Merged

Downgrading mlflow to avoid protobuf issues on nightly builds.#3784
vfdev-5 merged 5 commits into
pytorch:masterfrom
aaishwarymishra:failing-ci

Conversation

@aaishwarymishra

@aaishwarymishra aaishwarymishra commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Description:

Nightly CI/CD builds for pytorch/ignite were failing due to a protobuf compatibility issue introduced by a newer version of mlflow. This PR fixes the broken nightly test pipeline by downgrading mlflow to the last known working version, which resolves the protobuf conflict without requiring changes to the core library code.

Changes made:

  • Downgraded mlflow in requirements-dev.txt to a stable version compatible with the current protobuf setup
  • Investigated and ruled out alternative fixes (e.g., switching to Python backend for protobuf via env variable)

Root cause:
A newer mlflow release pulled in a protobuf version that conflicted with the nightly build environment, causing CI failures unrelated to any code changes in ignite itself.

Error message:

tests/ignite/contrib/handlers/test_logger_utils.py::test_setup_mlflow_logging FAILED [ 10%]
  tests/ignite/handlers/test_mlflow_logger.py::test_output_handler_output_transform FAILED [ 20%]
  tests/ignite/handlers/test_mlflow_logger.py::test_output_handler_metric_names FAILED [ 30%]
  tests/ignite/handlers/test_mlflow_logger.py::test_output_handler_both FAILED [ 40%]
  tests/ignite/handlers/test_mlflow_logger.py::test_output_handler_with_global_step_transform FAILED [ 50%]
  tests/ignite/handlers/test_mlflow_logger.py::test_output_handler_with_global_step_from_engine FAILED [ 60%]
  tests/ignite/handlers/test_mlflow_logger.py::test_output_handler_state_attrs FAILED [ 70%]
  tests/ignite/handlers/test_mlflow_logger.py::test_integration FAILED     [ 80%]
  tests/ignite/handlers/test_mlflow_logger.py::test_integration_as_context_manager FAILED [ 90%]
  tests/ignite/handlers/test_mlflow_logger.py::test_mlflow_bad_metric_name_handling FAILED [100%]
  
  =================================== FAILURES ===================================
  __________________________ test_setup_mlflow_logging ___________________________
  
  dirname = PosixPath('/tmp/tmpay6xvoa5')
  
      @pytest.mark.skipif(sys.platform.startswith("win"), reason="Skip on Windows")
      def test_setup_mlflow_logging(dirname):
  >       mlf_logger = _test_setup_logging(
              setup_logging_fn=setup_mlflow_logging,
              kwargs_dict={"tracking_uri": str(dirname / "p1")},
              output_handler_cls=handlers.mlflow_logger.OutputHandler,
              opt_params_handler_cls=handlers.mlflow_logger.OptimizerParamsHandler,
              with_eval=False,
              with_optim=False,
          )
  
  tests/ignite/contrib/handlers/test_logger_utils.py:243: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  tests/ignite/contrib/handlers/test_logger_utils.py:66: in _test_setup_logging
      x_logger = setup_logging_fn(**kwargs_dict)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ignite/handlers/logger_utils.py:271: in setup_mlflow_logging
      logger = MLflowLogger(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^
  ignite/handlers/mlflow_logger.py:148: in __init__
      import mlflow
  .venv/lib/python3.13/site-packages/mlflow/__init__.py:44: in <module>
      from mlflow import (
  .venv/lib/python3.13/site-packages/mlflow/artifacts/__init__.py:11: in <module>
      from mlflow.entities.file_info import FileInfo
  .venv/lib/python3.13/site-packages/mlflow/entities/__init__.py:15: in <module>
      from mlflow.entities.dataset import Dataset
  .venv/lib/python3.13/site-packages/mlflow/entities/dataset.py:2: in <module>
      from mlflow.protos.service_pb2 import Dataset as ProtoDataset
  .venv/lib/python3.13/site-packages/mlflow/protos/service_pb2.py:26: in <module>
      from opentelemetry.proto.trace.v1 import trace_pb2 as opentelemetry_dot_proto_dot_trace_dot_v1_dot_trace__pb2
  .venv/lib/python3.13/site-packages/opentelemetry/proto/trace/v1/trace_pb2.py:14: in <module>
      from opentelemetry.proto.common.v1 import common_pb2 as opentelemetry_dot_proto_dot_common_dot_v1_dot_common__pb2
  .venv/lib/python3.13/site-packages/opentelemetry/proto/common/v1/common_pb2.py:36: in <module>
      _descriptor.FieldDescriptor(
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  
  cls = <class 'google.protobuf.descriptor.FieldDescriptor'>
  name = 'string_value'
  full_name = 'opentelemetry.proto.common.v1.AnyValue.string_value', index = 0
  number = 1, type = 9, cpp_type = 9, label = 1, default_value = ''
  message_type = None, enum_type = None, containing_type = None
  is_extension = False, extension_scope = None, options = None
  serialized_options = None, has_default_value = False, containing_oneof = None
  json_name = None
  file = <google._upb._message.FileDescriptor object at 0x7f0f7db704f0>
  create_key = <object object at 0x7f0f7eb276d0>
  
      def __new__(
          cls,
          name,
          full_name,
          index,
          number,
          type,
          cpp_type,
          label,
          default_value,
          message_type,
          enum_type,
          containing_type,
          is_extension,
          extension_scope,
          options=None,
          serialized_options=None,
          has_default_value=True,
          containing_oneof=None,
          json_name=None,
          file=None,
          create_key=None,
      ):  # pylint: disable=redefined-builtin
  >     _message.Message._CheckCalledFromGeneratedFile()
  E     TypeError: Descriptors cannot be created directly.
  E     If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
  E     If you cannot immediately regenerate your protos, some other possible workarounds are:
  E      1. Downgrade the protobuf package to 3.20.x or lower.
  E      2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
  E     
  E     More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
  
  .venv/lib/python3.13/site-packages/google/protobuf/descriptor.py:675: TypeError

@github-actions github-actions Bot added the ci CI label Jun 9, 2026
Comment thread .github/workflows/unit-tests.yml Outdated
@aaishwarymishra aaishwarymishra changed the title downgrade protobuf use python backend for protobuf for nightly tests Jun 9, 2026
@aaishwarymishra aaishwarymishra changed the title use python backend for protobuf for nightly tests Downgrading mlflow to avoid protobuf issues on nightly builds. Jun 9, 2026
@aaishwarymishra
aaishwarymishra requested a review from vfdev-5 June 9, 2026 16:32
Comment thread requirements-dev.txt
Co-authored-by: vfdev <vfdev.5@gmail.com>
@aaishwarymishra

Copy link
Copy Markdown
Collaborator Author

@vfdev-5 done

@vfdev-5
vfdev-5 added this pull request to the merge queue Jun 9, 2026
Merged via the queue into pytorch:master with commit 8de8822 Jun 9, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants