Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaking change in protobuf v4.21.0-rc1 breaks import tensorflow #56077

Closed
matthewfeickert opened this issue May 12, 2022 · 12 comments
Closed

Breaking change in protobuf v4.21.0-rc1 breaks import tensorflow #56077

matthewfeickert opened this issue May 12, 2022 · 12 comments
Assignees
Labels
stat:awaiting response Status - Awaiting response from author subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.9 Issues found in the TF 2.9 release (or RCs) type:build/install Build and install issues

Comments

@matthewfeickert
Copy link

matthewfeickert commented May 12, 2022

Issue Type

Bug

Source

binary

Tensorflow Version

tf 2.9.0-rc2

Custom Code

No

OS Platform and Distribution

Linux Ubuntu 20.04

Mobile device

No response

Python version

3.10

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

This issue is to give a warning (that might already be known in internal CI) that protobuf v4.21.0-rc1 which was released today (2022-05-11) causes breaking changes in messages between Python and C++ that affect TensorFlow and will crash with

AttributeError: module 'google._upb._message' has no attribute 'Message'. Did you mean: 'CMessage'?

on import tensorflow.

This is somewhat noted in the protobuf release notes for Python

The C extension module for Python has been rewritten to use the upb library. This is expected to deliver significant performance benefits, especially when parsing large payloads. There are some minor breaking changes, but these should not impact most users. For more information see: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

where the Python Updates notes mentions

Applications that rely on sharing messages between Python and C++ break in the new version. Most developers won't be affected by this, but users of Nucleus{.external} and possibly other libraries may be. As a workaround, you can set an environment variable that forces the library to preserve compatibility.

which finally in Sharing Messages Between Python and C++ tells us that

Prior to Python 4.21.0, Python apps could share messages with C++ using a native extension. Starting in Python 4.21.0, sharing messages between Python and C++ is not supported by the default install. To enable this capability when working with 4.x and later versions of the Python API, define the environment variable, PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp, and ensure that the Python/C++ extension is installed.

However,

$ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp

just now leads to an

ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/venv/lib/python3.10/site-packages/google/protobuf/pyext/__init__.py)

Additionally tagging @deannagarcia who released protobuf v4.21.0-rc1.

This was noticed in pyhf's nightly pre-release and HEAD of dependencies CI jobs.

Standalone code to reproduce the issue

Here's a python:3.10 Docker image example

Click to exapnd:
$ docker run --rm -ti python:3.10 /bin/bash
root@a0c6928654f7:/# python -m venv venv && . venv/bin/activate
(venv) root@a0c6928654f7:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@a0c6928654f7:/# python -m pip --quiet install 'tensorflow==2.8.0'  # First show this with stable release v2.8.0
(venv) root@a0c6928654f7:/# python -m pip list | grep 'tensorflow\|protobuf'
protobuf                     3.20.1
tensorflow                   2.8.0
tensorflow-io-gcs-filesystem 0.25.0
(venv) root@a0c6928654f7:/# python -c 'import tensorflow; print(tensorflow.__version__)'
2022-05-12 03:21:47.631420: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-12 03:21:47.631464: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2.8.0
(venv) root@a0c6928654f7:/# python -m pip --quiet install --pre --upgrade protobuf
(venv) root@a0c6928654f7:/# python -m pip show protobuf
Name: protobuf
Version: 4.21.0rc1
Summary: 
Home-page: 
Author: 
Author-email: 
License: 
Location: /venv/lib/python3.10/site-packages
Requires: 
Required-by: tensorboard, tensorflow
(venv) root@a0c6928654f7:/# python -c 'import tensorflow'
2022-05-12 03:23:19.770470: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-12 03:23:19.770491: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
AttributeError: module 'google._upb._message' has no attribute 'Message'. Did you mean: 'CMessage'?
(venv) root@a0c6928654f7:/# export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp  # try protobuf environment variable suggestion
(venv) root@a0c6928654f7:/# python -c 'import tensorflow'
2022-05-12 03:23:50.884921: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-12 03:23:50.884945: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/venv/lib/python3.10/site-packages/google/protobuf/internal/api_implementation.py:109: UserWarning: Selected implementation cpp is not available.
  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 7, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 51, in <module>
    from google.protobuf.pyext import _message
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/venv/lib/python3.10/site-packages/google/protobuf/pyext/__init__.py)
(venv) root@a0c6928654f7:/# unset PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION
(venv) root@a0c6928654f7:/# python -m pip --quiet install --pre --upgrade tensorflow  # Try release candidate v2.9.0-rc2
(venv) root@a0c6928654f7:/# python -m pip show tensorflow
Name: tensorflow
Version: 2.9.0rc2
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /venv/lib/python3.10/site-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, packaging, protobuf, setuptools, six, tensorboard, tensorflow-estimator, tensorflow-io-gcs-filesystem, termcolor, typing-extensions, wrapt
Required-by
(venv) root@a0c6928654f7:/# python -c 'import tensorflow'
2022-05-12 03:26:20.957106: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-12 03:26:20.957131: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
AttributeError: module 'google._upb._message' has no attribute 'Message'. Did you mean: 'CMessage'?
(venv) root@a0c6928654f7:/# export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
(venv) root@a0c6928654f7:/# python -c 'import tensorflow'
2022-05-12 03:27:01.166115: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-12 03:27:01.166141: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/venv/lib/python3.10/site-packages/google/protobuf/internal/api_implementation.py:109: UserWarning: Selected implementation cpp is not available.
  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 7, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 51, in <module>
    from google.protobuf.pyext import _message
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/venv/lib/python3.10/site-packages/google/protobuf/pyext/__init__.py)

Relevant log output

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
AttributeError: module 'google._upb._message' has no attribute 'Message'. Did you mean: 'CMessage'?
/venv/lib/python3.10/site-packages/google/protobuf/internal/api_implementation.py:109: UserWarning: Selected implementation cpp is not available.
  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 7, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 51, in <module>
    from google.protobuf.pyext import _message
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/venv/lib/python3.10/site-packages/google/protobuf/pyext/__init__.py)
@google-ml-butler google-ml-butler bot added the type:bug Bug label May 12, 2022
@mohantym mohantym added type:build/install Build and install issues subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.9 Issues found in the TF 2.9 release (or RCs) and removed type:bug Bug labels May 12, 2022
@mohantym mohantym assigned gadagashwini and unassigned mohantym May 12, 2022
@kwadra
Copy link

kwadra commented May 12, 2022

Same problem in TF 2.7 / Python 3.8 started yesterday.

Pipfile:
tensorflow-cpu = "==2.7.1"

We didn't have an explicit protobuf requirement until this started but adding one did help us work around the issue:
protobuf = "~=3.20.1"

import tensorflow as tf

../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/init.py:41: in
from tensorflow.python.tools import module_util as _module_util
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/python/init.py:41: in
from tensorflow.python.eager import context
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/python/eager/context.py:33: in
from tensorflow.core.framework import function_pb2
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/core/framework/function_pb2.py:16: in
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:16: in
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:16: in
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:16: in
from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:36: in
_descriptor.FieldDescriptor(
../../../.local/share/virtualenvs/herc-xx-tKdjcwX4/lib/python3.8/site-packages/google/protobuf/descriptor.py:560: in new
_message.Message._CheckCalledFromGeneratedFile()
E AttributeError: module 'google._upb._message' has no attribute 'Message'

@parthea
Copy link

parthea commented May 12, 2022

The issue may be resolved if you install grpcio-tools>=1.44.0. See protocolbuffers/protobuf#9954

@haberman
Copy link

_message.Message._CheckCalledFromGeneratedFile()

This indicates that you are using generated code from an old protoc.

Per the announcement, Protobuf 4.21.0 requires that the _pb2.py files were generated by protoc >= 3.19.0.

Regenerating the _pb2.py files from a new protoc should fix this problem.

@matthewfeickert
Copy link
Author

matthewfeickert commented May 12, 2022

The issue may be resolved if you install grpcio-tools>=1.44.0. See protocolbuffers/protobuf#9954

This is because grpcio-tools downgrades protobuf to v3.x as @haberman points out in protocolbuffers/protobuf#9954 (comment).

$ docker run --rm -ti python:3.10 /bin/bash
root@018b400a104b:/# python -m venv venv && . venv/bin/activate
(venv) root@018b400a104b:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@018b400a104b:/# python -m pip --quiet install --pre tensorflow
(venv) root@018b400a104b:/# python -m pip list | grep 'tensorflow\|protobuf\|grpcio'
grpcio                       1.46.1
protobuf                     4.21.0rc1
tensorflow                   2.9.0rc2
tensorflow-estimator         2.9.0rc0
tensorflow-io-gcs-filesystem 0.25.0
(venv) root@018b400a104b:/# python -m pip --quiet install 'grpcio-tools>=1.44.0'
(venv) root@018b400a104b:/# python -m pip list | grep 'tensorflow\|protobuf\|grpcio'
grpcio                       1.46.1
grpcio-tools                 1.46.1
protobuf                     3.20.1
tensorflow                   2.9.0rc2
tensorflow-estimator         2.9.0rc0
tensorflow-io-gcs-filesystem 0.25.0
(venv) root@018b400a104b:/# python -c 'import tensorflow; print(tensorflow.__version__)'
2022-05-12 17:27:55.187531: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-12 17:27:55.187556: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2.9.0-rc2
(venv) root@018b400a104b:/# 

Regenerating the _pb2.py files from a new protoc should fix this problem.

Thanks @haberman. So seems like this is something that the TensorFlow team will need to do for the next release, as well as probably change their lower bounds on protobuf

to

    'protobuf >= 4.21.0',

@gadagashwini
Copy link
Contributor

@matthewfeickert,
tensorflow==2.9.0-rc2 support grpcio==1.44.0 and protobuf==3.17.3.

Protobuf and grpcio version compatible ranges

protobuf>=3.9.2
grpcio<2.0,>=1.24.3

@gadagashwini gadagashwini added the stat:awaiting response Status - Awaiting response from author label May 13, 2022
@matthewfeickert
Copy link
Author

matthewfeickert commented May 13, 2022

tensorflow==2.9.0-rc2 support grpcio==1.44.0 and protobuf==3.17.3.

Protobuf and grpcio version compatible ranges

protobuf>=3.9.2
grpcio<2.0,>=1.24.3

@gadagashwini Yes, I showed the (relevant snippets) of v2.9.0-rc2 install_requires in my previous response #56077 (comment). Here is a slightly expanded view again

'protobuf >= 3.9.2',
'setuptools',
'six >= 1.12.0',
'termcolor >= 1.1.0',
'typing_extensions >= 3.6.6',
'wrapt >= 1.11.0',
'tensorflow-io-gcs-filesystem >= 0.23.1',
# grpcio does not build correctly on big-endian machines due to lack of
# BoringSSL support.
# See https://github.com/tensorflow/tensorflow/issues/17882.
'grpcio >= 1.24.3, < 2.0' if sys.byteorder == 'little' else None,

As soon as protobuf v4.21.0 is released then it will get picked up by TensorFlow and break installs. This is easy to fix post install (users just downgrade protobuf) but if the TensorFlow dev team can coordinate with the protobuf team on when release candidates will get bumped to releases, TensorFlow v2.9.0 could be released with everything working smoothly.

@haberman I'm not sure if you're able to say yet, but do you know roughly when the protobuf team is targeting release of v4.21.0? (Also, is there a reason the release is tagged as v21.0-rc1 in GitHub when it is v4.21.0-rc1)?

edit: Actually, I guess this isn't really a problem at all. The TensorFlow team just needs to make sure that they regenerate the _pb2.py files for the next RC or release with the appropriate version of protoc installed, given @haberman's comment #56077 (comment)

Per the announcement, Protobuf 4.21.0 requires that the _pb2.py files were generated by protoc >= 3.19.0.

Regenerating the _pb2.py files from a new protoc should fix this problem.

right?

Though I guess on some OS that requires getting it not from your package manager given:

$ docker run --rm -ti python:3.10 /bin/bash
root@ac4c778a09c5:/# apt-get update && apt-get install -y protobuf-compiler
root@ac4c778a09c5:/# protoc --version
libprotoc 3.12.4
root@ac4c778a09c5:/# cat /etc/os-release | grep PRETTY_NAME=
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"

@matthewfeickert matthewfeickert changed the title Breaking change in protobuf v21.0-rc1 breaks import tensorflow Breaking change in protobuf v4.21.0-rc1 breaks import tensorflow May 13, 2022
@haberman
Copy link

The TensorFlow team just needs to make sure that they regenerate the _pb2.py files for the next RC or release with the appropriate version of protoc installed, given @haberman's comment #56077 (comment)

Yes exactly, I think regenerating the _pb2.py files as soon as you can is the best resolution here.

Though I guess on some OS that requires getting it not from your package manager given:

Yes, you generally want stronger guarantees that protoc will match your runtime library. C++ requires an exact match. Other languages have been somewhat more lax, but it's best to avoid relying on that.

do you know roughly when the protobuf team is targeting release of v4.21.0?

We are hoping to release v4.21.0 next week. Would you be able to regenerate the _pb2.py files in the next few days?

Also, is there a reason the release is tagged as v21.0-rc1 in GitHub when it is v4.21.0-rc1

See the note here: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#versioning

@matthewfeickert
Copy link
Author

matthewfeickert commented May 16, 2022

This never got addressed by the TensorFlow team and now TensorFlow v2.9.0 is will be broken.

(Sorry on phone while on vacation so I ran the pre-release workflow before posting that. TensorFlow V2.9.0 will be broken once protobuf makes a release. But until then the current release works, but it will break.)

copybara-service bot pushed a commit that referenced this issue May 20, 2022
See also: #53234
See also: protocolbuffers/protobuf#9954
See also: #56077

PiperOrigin-RevId: 450054200
tensorflow-jenkins pushed a commit that referenced this issue May 20, 2022
See also: #53234
See also: protocolbuffers/protobuf#9954
See also: #56077

PiperOrigin-RevId: 450054200
mihaimaruseac added a commit that referenced this issue May 20, 2022
See also: #53234
See also: protocolbuffers/protobuf#9954
See also: #56077

PiperOrigin-RevId: 450054200
pranve pushed a commit to pranve/tensorflow that referenced this issue May 21, 2022
pranve pushed a commit to pranve/tensorflow that referenced this issue May 21, 2022
pranve pushed a commit to pranve/tensorflow that referenced this issue May 21, 2022
@google-ml-butler
Copy link

Are you satisfied with the resolution of your issue?
Yes
No

matthewfeickert added a commit to scikit-hep/pyhf that referenced this issue May 26, 2022
* protobuf v3.20.1 is the last release to support ABI compatibility
with tensorflow pre tensorflow 2.9.1. The next release of protobuf,
v4.21.0, breaks ABI compatibility and so requires an upper bound of
protobuf v3.20.1 for older tensorflow releases to not break. tensorflow
released a series of patch releases in response to protobuf v4.21.0:
v2.6.5, v2.7.3, v2.8.2, v2.9.1.
   - c.f. tensorflow/tensorflow#56077
matthewfeickert added a commit to scikit-hep/pyhf that referenced this issue May 27, 2022
* Update lower bound on tensorflow to v2.6.5 as it is the "oldest" SemVer release
that is not broken on install by protobuf v4.21.0 ABI incompatibility.
   - c.f. tensorflow/tensorflow#56077
   - Amend PR #1869
* Update tests/constraints.txt to use tensorflow==2.6.5.
mirkolenz added a commit to recap-utr/nlp-service that referenced this issue Jun 13, 2022
Tensorflow is temporarily removed due to incompatabilities with protobuf
See tensorflow/tensorflow#56077
georgepaw pushed a commit to graphcore/tensorflow that referenced this issue Jul 25, 2022
Summary:
See also: tensorflow/tensorflow#53234
See also: protocolbuffers/protobuf#9954
See also: tensorflow/tensorflow#56077

TF1.15 Only

Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm

Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm

Maniphest Tasks: T63128

Differential Revision: https://phabricator.sourcevertex.net/D67981
chxin66 pushed a commit to chxin66/tensorflow that referenced this issue Sep 5, 2022
copybara-service bot pushed a commit to tensorflow/gnn that referenced this issue Sep 15, 2022
copybara-service bot pushed a commit to tensorflow/gnn that referenced this issue Sep 15, 2022
copybara-service bot pushed a commit to tensorflow/gnn that referenced this issue Sep 15, 2022
copybara-service bot pushed a commit to tensorflow/gnn that referenced this issue Sep 15, 2022
wes-turner added a commit to wes-turner/determined that referenced this issue Mar 2, 2023
Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce.
wes-turner added a commit to wes-turner/determined that referenced this issue Mar 2, 2023
Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce.
wes-turner added a commit to determined-ai/determined that referenced this issue Mar 2, 2023
build: [MLG-336] Limit the version of protobuf

Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce.
tayritenour added a commit to determined-ai/determined that referenced this issue Mar 9, 2023
* docs: Improvements to HPC launcher docs (#6042)

* Provide inline info about agent-specific scheduling options that do not apply to HPC Launcher
  configurations.
* Identify enroot-specific differences from docker (like for Singularity)
* Provide reference to custom resource pools as an option to deal with non-homogenous
  Slurm/PBS partitions.

* chore: Allow newer Node versions 17-19 (#6038)

* fix: k8s rm gives wrong slot count in rendezvous (#6044)

* chore: bump version: 0.19.12-dev0 -> 0.20.0-dev0 (#6048)

* chore: remove `applicableRoutespace` (#6040)

* chore: warning fixes in web (#6041)

* chore: fix warnings

* chore: change eslint rules

* chore: fix gpu nightly errors (#6046)

* chore: missing nodev18 (#6050)

* chore: add a dedicated exception for cli errors (#5649)

switch sys.exit calls in cli with a new user-facing exception.

* fix: SSO layout (#6053)

* chore: clean up UI kit (#6039)

* fix: lopsided training with 2,1 gpus (#6054)

There was a guard to skip local zmq setup when local_size < 2, but that
became no longer valid when local_size varied from worker to worker.

The result is one extra global allgather in some cases, no big deal.

* docs: add rbac ntsc & mr release notes (#6049)

* chore: manual bump version (#6058)

* ci: retry downloading GKE auth plugin [DET-8956] (#6056)

We got a failure due to a timeout on this, so let it retry a few times.

* docs: update Singularity known issues. (#6047)

* fix: Add #rank to worker segment instead of timestamp segment of Pytorch Profiler files [MLG-326] (#6037)

* Add pytorch profiler specific handling logic for appending rank to file name

* Change to use f-string

* fix only file name being passed in

* remove print statement

* fix: handle agent shutdown msg (#6065)

* chore: manually bump vite version (#6066)

* feat: Display better x-axis ticks on charts with time axis [WEB-849] (#6051)

* remove xTickValues from props now that it can be calculated internally

* test: add logging to a flaky test (#6068)

Test is flaky but hard to pin down, so add some prints for next time.

* fix: Unrelated models are shown in a workspace model registry tab (#6067)

* feat: Added task-based historical allocation endpoint [DET-8537] (#6015)

* fix: show `not found` and `spinner` properly (#6070)

* fix: show `not found` and `spinner` properly

* chore: change home redirect path

* fix: projectDetail page

* fix: project.workspaceId

* build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto (#6061)

* build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto

Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.5 to 0.3.8.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](golang/text@v0.3.5...v0.3.8)

---
updated-dependencies:
- dependency-name: golang.org/x/text
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: gpt-neox docker image and startup hook to work for non-provileged user (#6060)

* test: add locking around migration [DET-8957] (#6071)

In integration tests, multiple processes can attempt to run migrations
against the same database at once, which can lead to errors because
PostgreSQL's `CREATE TABLE IF NOT EXISTS` is not great with concurrency
(it allows for a time-of-check/time-of-use failure).

The specific errors we were seeing were conflicts in the pg_type table,
so the code now locks that table for the duration of the migration
transaction.

More information:
https://www.postgresql.org/message-id/CA+TgmoZAdYVtwBfp1FL2sMZbiHCWT4UPrzRLNnX1Nb30Ku3-gg@mail.gmail.com
https://stackoverflow.com/questions/29900845

* fix: logging inconsistent newlines in slurm (#6074)

* fix: checkpoint helper for points > 1000 and points > maxDatapoints (#6069)

* fix: replace migration table lock with advisory lock (#6077)

Taking a table lock sometimes runs into permissions issues; advisory
locking should avoid that.

Also, I realized the locking should probably be after the deferred
transaction close instead of before.

* build: check npm version on install (#6079)

This removes the `check-requirements` make target in the react folder
and replaces it with npm's native version check against the engines
property. This should make managing the node version slightly easier
because there's one less place to check.

* build: Apply webui lint fixes in precommit (#6078)

* allow linters to fix in precommit

This updates the web linters to automatically apply fixes when doing a
pre-commit check. This should ideally streamline the commit process to
reduce the amount of times the user needs to run prettier and eslint
before committing.

* tweak stylelint and eslint commands

* stage changed files

* type file_paths argument

* ci: adjust target accuracy for a test (#6085)

We got one failure [1] where the accuracy ended up just a hair below
0.83, so drop the target.

[1] https://app.circleci.com/pipelines/github/determined-ai/determined/33883/workflows/4a5d3257-6061-4f4d-bd66-096a580a5959/jobs/1194282/steps

* chore: UserBadge moves into design kit (#6086)

* chore: Remove chart feature flag, remove unused code [WEB-930] (#6064)

* tooltipsPlugin and TrialDetailsOverview alternates go into place
* checkpoint helper for points > 1000 and points > maxDatapoints
* move former LearningCurveChart into TrialsComparison
* sync up with #6069 changes

* fix: replace defaultvalue with initialValue (#6076)

* fix: Dont suggest moving model into its current workspace (#6088)

* ci: delete database at beginning of det deploy tests [DET-8937] (#6089)

Previously, the database was being retained between tests, sometimes
causing tests to fail when extra agents appeared due to agent
reattach. The tests should generally be independent anyway, so reset
the database (by default, with an option to disable) each time the
cluster or master comes up.

* feat: add Facepile component (#6081)

* fix: pre-commit web bug fix (#6090)

* ci: make GKE test jobs run serially (#6096)

We keep hitting GKE GPU quotas; this will probably help with that.

* fix: GPU counting for k8s cluster info page (#6094)

* fix: test-e2e-gke-parallel use t4 (#6093)

* ci: retry protoc download (#6095)

We got an incorrect file downloaded one time [1], so retry this
download, like in 2906257 (#5996).

[1] https://app.circleci.com/pipelines/github/determined-ai/determined/34074/workflows/e48681f8-8b75-4349-82eb-06e922d8bfcb/jobs/1202610

* refactor: add Card to UI Kit [WEB-818] (#5893)

* docs: Launcher doc improvements (#6099)

- Generalize journalctl command example --since option to work on Unbutu.
- Clarify user_name/group_name account requirements.

* feat: Attend to TODOS accross the code base (#6087)

* perf: tweak metrics series query. (#6105)

* chore: race could cause run container to return a different error than expected [DET-8870] (#6092)

* chore: add more metadata to slurm logs (#6030)

* chore: remove `ExpCompareMetricNames`, `ExpCompareTrialsSample` endpoints. (#6106)

* docs: fix reported DataPoint label doc (#6107)

* fix: tolerate additional non-CPU, non-GPU quotas in k8s (#6109)

* fix: stop filtering of valid options to reflect build issues (#6116)

* fix: modal theme color (#6117)

* fix: add bgColor in trial comparison table (#6119)

* fix: browser console warnings (#6122)

* fix: browser console warnings

* fix: remove spread operator

* chore: UIKit Pivot renaming (#6120)

* fix: correct minor JSX syntax (#6126)

* docs: add myst_parser extension (#6127)

We would like to support markdown-format documentation.  There are still
some kinks to be worked out with converting rst to myst files, but this
is a start.

* docs: fix some broken redirects (#6129)

* feat: add 3rd batch of TODO removals (#6115)

* feat: generic proxy configs [DET-8761] (#5978)

* build: [MLG-336] Limit the version of protobuf (#6134)

build: [MLG-336] Limit the version of protobuf

Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce.

* chore: update detectron2 example to use v0.6 and reenable nightly test [MLG-301] (#6103)

* Run model in EventStorage context

* Use new Docker images

* Remove pytest.skip from test_detectron2_coco_pytorch_const

* Update README.md

* Minor code reduction

* Dockerfile (listed in .detignore)

* Use determinedai repo instead of a personal repo

* Makefile for building and publishing the Docker image

* docs: Bring content changes from docusaurus-ls beta (#6121)

* docs: Bring content changes from docusaurus-ls beta

Bring over content changes from the beta including reorganization changes.

* additional organizational edits

updating index pages, adding a top nav to welcome page

* added redirects

* revisions based on feedback

* rstfmt run

* feat: display workspace icon in ProjectCard (#6125)

* fix: checkpoint GC should set resource pool (#6136) [DET-9018]

* docs: bump rstfmt version (#6138)

* chore: add dev cli option to get auth token (#6008)

add a `curl` option to help with curling various endpoints

* build(deps): bump golang.org/x/net from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0 in /proto (#6130)

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)

* perf: Improved performance of historical allocation task endpoint, removed training/validation times (#6135)

* fix: FOUNDENG-438 Podman tests from the gate are breaking znodes again (#6146)

* chore: add Toggle component to UI Kit [WEB-841] (#6144)

* chore: Add tags to UI kit [WEB-816] (#6100)

* chore: Move SelectFilter into kit folder and update it [WEB-843] (#6102)

* fix: replace `InlineEditor` with UIKit input (#6082)

* fix: replace `InlineEditor`

* fix: add modal for experiment name

* fix: layout of settings page

* fix: setting page

* fix: minor changes

* feat: move experiment `description` and `tags` into edit modal

* chore: add `N/A` when description is empty in experiment detail

* fix: value bug

* fix: revert tag; remove tag from edit modal due to design inconsistancy

* chore:  add/test pt-only images and bumpenvs (#6097)

* add pt images to some unit tests

* add pt-images to circleci config

* run bumpenvs procedure

* fix test function signatures and linting

* fix warnings linting

* fix docs

* expand unit tests coverage

* build(deps): bump github.com/prometheus/client_golang from 1.10.0 to 1.11.1 in /master (#6004)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.10.0 to 1.11.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.10.0...v1.11.1)

* feat: add det.import_from_path (#5737)

import_from_path allows users to import arbitrary code from a
checkpoint, even if the modules in the checkpoint have the same name as
modules they already have imported, but contain different code.

This is common when importing, for example, an old model_def.py that has
been updated since the original checkpoint was saved.

* fix: post rank_id correctly for fluent-less logging (#6151) [DET-8999]

* ci: send Slack notification on GKE node pool creation failure (#6152)

In order to prevent quota failures from showing up as CI failures, this
makes node pool creation failure send a Slack notification and mark the
job as successful.

I couldn't figure out how to use the Slack orb while distinguishing this
particular situation from the general failure case, so I just slapped in
a direct request to the already-configured Slack webhook for sending
messages to #ci-bots.

`circleci-agent step halt` marks the job as successful, which is why we
want a notification at all. For some reason, CircleCI fails to provide
an equivalent for marking the job as canceled or some other state
besides success/failure; we could make a call to the CircleCI API to
cancel the current job, but that would rely on having a CircleCI token
available, which we're trying to get away from.

* chore: drop unused columns from `raw_steps`, `raw_validations`, and `raw_checkpoints`. (#6110)

* fix: render spinner while auth check pending (#6098)

* chore: update hpc-launching-architecture doc - add default slurm option --no-requeue (#6141)

* docs: Content updates (#6154)

formatted the setup cluster table to match the approved version in the docusaurus ls beta

* feat: display user id in `det user list`. (#6156)

* fix: Additional tables get experiment- / workspace-specific storagePath [WEB-962] (#6128)

* fix: Additional tables get experiment- and workspace-specific storagePath

* useMemo

* fix: selection width in `move experiment` modal (#6149)

* fix: selection width in `move experiment` modal

* fix: add form wrapping

* chore: remove change

* feat: show trial hyperparameters for custom searchers [MLG-343] (#6162)

* feat: show trial hyperparameters for custom searchers [MLG-343]

* fix: corrected timestamp handling to do an interval overlap instead of contains (#6164)

* chore:  add release notes (#6167)

* chore: add release notes

* format with rstfmt

* chore: suppress help output for det dev (#6145)

avoid showing the `dev` option in `det -h` output

* chore: lock api state for backward compatibility check

* chore: bump version: 0.20.0-dev0 -> 0.20.1-dev0

* fix: separate Router and authCheck (#6170)

* fix: useMemo does not depend on trial having been loaded (#6173)

* chore: pass Labels/project/workspace to TaskSpec (#6172)

* refactor: replace user store with observables [WEB-799] (#6140)

* fix: hide Foldable menu options when button is visible (#6178)

This fixes an issue where, when using a `PageHeaderFoldable` component,
options that appear in the header always appear in the overflow menu.

* feat: add labels to GCP instances created with det deploy gcp [MLG-170] (#6147)

* feat: add labels to GCP instances created with det deploy gcp

* Changes to mimic det deploy aws --add-tags

* Add labels to other resources as well

* revert: reflag new chart experience (#6181)

* build: eliminate java dependency for typescript swagger bindings (#6139)

* fix: Close expiriment fork/ continue trial modal properly (#6174)

* fix: Continue Trial flow does not take the new `max_length` (#6168)

* fix: pass workspace ID when creating tensor board from WebUI [WEB-1019] (#6186)

* fix: don't print ':' when err msg is empty (#6190)

* fix: exp move modal (#6183)

* fix: exp move modal

* fix: minor fixes

* fix: add `archived` param and simplify query (#6175)

* fix: add `archived` param and simplify query

* chore: indent

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jerry J. Harrow <84593277+jerryharrow@users.noreply.github.com>
Co-authored-by: Nick Doiron <nick.doiron@hpe.com>
Co-authored-by: NicholasBlaskey <nick.blaskey@hpe.com>
Co-authored-by: Eric <31023784+eecsliu@users.noreply.github.com>
Co-authored-by: Keita Nonaka <keita.nonaka@hpe.com>
Co-authored-by: Erik Wilson <erik.wilson@hpe.com>
Co-authored-by: Hamid Zare <12127420+hamidzr@users.noreply.github.com>
Co-authored-by: johnkim-det <97752292+johnkim-det@users.noreply.github.com>
Co-authored-by: Ryan <rb@hpe.com>
Co-authored-by: Danny Zhu <dzhu@hpe.com>
Co-authored-by: CanmingCobble <107056780+CanmingCobble@users.noreply.github.com>
Co-authored-by: szewaiyuen6 <sze-wai.yuen@hpe.com>
Co-authored-by: Bradley Laney <bradley.laney@hpe.com>
Co-authored-by: julian-determined-ai <103522725+julian-determined-ai@users.noreply.github.com>
Co-authored-by: Corban Beaird <corban.beaird@hpe.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: liamcli <liam@determined.ai>
Co-authored-by: Ashton G <ashton.galloway@hpe.com>
Co-authored-by: thiagodallacqua-hpe <104855841+thiagodallacqua-hpe@users.noreply.github.com>
Co-authored-by: Max <max.russell@hpe.com>
Co-authored-by: Maksim <maksim.kouznetsov@hpe.com>
Co-authored-by: Emily <15078396+EmilyBonar@users.noreply.github.com>
Co-authored-by: Ilia Glazkov <ilia.glazkov@hpe.com>
Co-authored-by: Caleb Hoyoul Kang <caleb.kang@hpe.com>
Co-authored-by: Wes Turner <wesley.turner@hpe.com>
Co-authored-by: Daniel R. Hunter <103537968+drh-determined-ai@users.noreply.github.com>
Co-authored-by: Tara Charter <tara.charter@hpe.com>
Co-authored-by: rcorujo <90728398+rcorujo@users.noreply.github.com>
Co-authored-by: Guangqing Tang <40620519+gt2345@users.noreply.github.com>
Co-authored-by: MikhailKardash <mikhail.kardash@hpe.com>
Co-authored-by: Jagadeesh Madagundi <jagadeesh545@gmail.com>
Co-authored-by: gt2345 <gt2345@columbia.edu>
Co-authored-by: Trent Watson <trent.watson@hpe.com>
tayritenour added a commit to determined-ai/determined that referenced this issue May 30, 2023
* Save model info; add Core API DeepSpeed example.

* extract ds profiler results

* Place activation mem into search metric.

* Obtain metrics from the model info file

* remove unused code

* adapted to passing dicts into report_completed

* cleanup and small changes

* refactoring and trial helper classes

* initial random search logic started

* minor changes

* minor cleanup

* use context manager, expanded base searcher

* remove ds_autotuning dir from examples

* minor edits

* readme updates and other cleanup

* bug fixes and a hack to avoid needing nested model dirs

* README updates

* Feat deepspeed autotune (#5875)

Added current Core API prototype.

* switched to triggering their autotuning in our trials

* remove checkpoint wrapper

* implementing checkpointing

* implementing checkpointing

* feat: allow includes in custom searcher experiment [MLG-338] (#6091)

* cleanups, bug fixing, and more examples

* adding native dsat tests

* cleanup

* minor edits

* fix missing is_chief

* Feat deepspeed autotune (#6159)

Trigger the native DS AT exit behavior for all trials.

* minor edits

* Feat deepspeed autotune git fixes (#6180)

deleted extraneous files

* readme fix and make cifar10 work off grenoble (#6187)

* Deepspeed Feature Branch - merge master (#6193)

* docs: Improvements to HPC launcher docs (#6042)

* Provide inline info about agent-specific scheduling options that do not apply to HPC Launcher
  configurations.
* Identify enroot-specific differences from docker (like for Singularity)
* Provide reference to custom resource pools as an option to deal with non-homogenous
  Slurm/PBS partitions.

* chore: Allow newer Node versions 17-19 (#6038)

* fix: k8s rm gives wrong slot count in rendezvous (#6044)

* chore: bump version: 0.19.12-dev0 -> 0.20.0-dev0 (#6048)

* chore: remove `applicableRoutespace` (#6040)

* chore: warning fixes in web (#6041)

* chore: fix warnings

* chore: change eslint rules

* chore: fix gpu nightly errors (#6046)

* chore: missing nodev18 (#6050)

* chore: add a dedicated exception for cli errors (#5649)

switch sys.exit calls in cli with a new user-facing exception.

* fix: SSO layout (#6053)

* chore: clean up UI kit (#6039)

* fix: lopsided training with 2,1 gpus (#6054)

There was a guard to skip local zmq setup when local_size < 2, but that
became no longer valid when local_size varied from worker to worker.

The result is one extra global allgather in some cases, no big deal.

* docs: add rbac ntsc & mr release notes (#6049)

* chore: manual bump version (#6058)

* ci: retry downloading GKE auth plugin [DET-8956] (#6056)

We got a failure due to a timeout on this, so let it retry a few times.

* docs: update Singularity known issues. (#6047)

* fix: Add #rank to worker segment instead of timestamp segment of Pytorch Profiler files [MLG-326] (#6037)

* Add pytorch profiler specific handling logic for appending rank to file name

* Change to use f-string

* fix only file name being passed in

* remove print statement

* fix: handle agent shutdown msg (#6065)

* chore: manually bump vite version (#6066)

* feat: Display better x-axis ticks on charts with time axis [WEB-849] (#6051)

* remove xTickValues from props now that it can be calculated internally

* test: add logging to a flaky test (#6068)

Test is flaky but hard to pin down, so add some prints for next time.

* fix: Unrelated models are shown in a workspace model registry tab (#6067)

* feat: Added task-based historical allocation endpoint [DET-8537] (#6015)

* fix: show `not found` and `spinner` properly (#6070)

* fix: show `not found` and `spinner` properly

* chore: change home redirect path

* fix: projectDetail page

* fix: project.workspaceId

* build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto (#6061)

* build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto

Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.5 to 0.3.8.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](golang/text@v0.3.5...v0.3.8)

---
updated-dependencies:
- dependency-name: golang.org/x/text
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: gpt-neox docker image and startup hook to work for non-provileged user (#6060)

* test: add locking around migration [DET-8957] (#6071)

In integration tests, multiple processes can attempt to run migrations
against the same database at once, which can lead to errors because
PostgreSQL's `CREATE TABLE IF NOT EXISTS` is not great with concurrency
(it allows for a time-of-check/time-of-use failure).

The specific errors we were seeing were conflicts in the pg_type table,
so the code now locks that table for the duration of the migration
transaction.

More information:
https://www.postgresql.org/message-id/CA+TgmoZAdYVtwBfp1FL2sMZbiHCWT4UPrzRLNnX1Nb30Ku3-gg@mail.gmail.com
https://stackoverflow.com/questions/29900845

* fix: logging inconsistent newlines in slurm (#6074)

* fix: checkpoint helper for points > 1000 and points > maxDatapoints (#6069)

* fix: replace migration table lock with advisory lock (#6077)

Taking a table lock sometimes runs into permissions issues; advisory
locking should avoid that.

Also, I realized the locking should probably be after the deferred
transaction close instead of before.

* build: check npm version on install (#6079)

This removes the `check-requirements` make target in the react folder
and replaces it with npm's native version check against the engines
property. This should make managing the node version slightly easier
because there's one less place to check.

* build: Apply webui lint fixes in precommit (#6078)

* allow linters to fix in precommit

This updates the web linters to automatically apply fixes when doing a
pre-commit check. This should ideally streamline the commit process to
reduce the amount of times the user needs to run prettier and eslint
before committing.

* tweak stylelint and eslint commands

* stage changed files

* type file_paths argument

* ci: adjust target accuracy for a test (#6085)

We got one failure [1] where the accuracy ended up just a hair below
0.83, so drop the target.

[1] https://app.circleci.com/pipelines/github/determined-ai/determined/33883/workflows/4a5d3257-6061-4f4d-bd66-096a580a5959/jobs/1194282/steps

* chore: UserBadge moves into design kit (#6086)

* chore: Remove chart feature flag, remove unused code [WEB-930] (#6064)

* tooltipsPlugin and TrialDetailsOverview alternates go into place
* checkpoint helper for points > 1000 and points > maxDatapoints
* move former LearningCurveChart into TrialsComparison
* sync up with #6069 changes

* fix: replace defaultvalue with initialValue (#6076)

* fix: Dont suggest moving model into its current workspace (#6088)

* ci: delete database at beginning of det deploy tests [DET-8937] (#6089)

Previously, the database was being retained between tests, sometimes
causing tests to fail when extra agents appeared due to agent
reattach. The tests should generally be independent anyway, so reset
the database (by default, with an option to disable) each time the
cluster or master comes up.

* feat: add Facepile component (#6081)

* fix: pre-commit web bug fix (#6090)

* ci: make GKE test jobs run serially (#6096)

We keep hitting GKE GPU quotas; this will probably help with that.

* fix: GPU counting for k8s cluster info page (#6094)

* fix: test-e2e-gke-parallel use t4 (#6093)

* ci: retry protoc download (#6095)

We got an incorrect file downloaded one time [1], so retry this
download, like in 2906257 (#5996).

[1] https://app.circleci.com/pipelines/github/determined-ai/determined/34074/workflows/e48681f8-8b75-4349-82eb-06e922d8bfcb/jobs/1202610

* refactor: add Card to UI Kit [WEB-818] (#5893)

* docs: Launcher doc improvements (#6099)

- Generalize journalctl command example --since option to work on Unbutu.
- Clarify user_name/group_name account requirements.

* feat: Attend to TODOS accross the code base (#6087)

* perf: tweak metrics series query. (#6105)

* chore: race could cause run container to return a different error than expected [DET-8870] (#6092)

* chore: add more metadata to slurm logs (#6030)

* chore: remove `ExpCompareMetricNames`, `ExpCompareTrialsSample` endpoints. (#6106)

* docs: fix reported DataPoint label doc (#6107)

* fix: tolerate additional non-CPU, non-GPU quotas in k8s (#6109)

* fix: stop filtering of valid options to reflect build issues (#6116)

* fix: modal theme color (#6117)

* fix: add bgColor in trial comparison table (#6119)

* fix: browser console warnings (#6122)

* fix: browser console warnings

* fix: remove spread operator

* chore: UIKit Pivot renaming (#6120)

* fix: correct minor JSX syntax (#6126)

* docs: add myst_parser extension (#6127)

We would like to support markdown-format documentation.  There are still
some kinks to be worked out with converting rst to myst files, but this
is a start.

* docs: fix some broken redirects (#6129)

* feat: add 3rd batch of TODO removals (#6115)

* feat: generic proxy configs [DET-8761] (#5978)

* build: [MLG-336] Limit the version of protobuf (#6134)

build: [MLG-336] Limit the version of protobuf

Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce.

* chore: update detectron2 example to use v0.6 and reenable nightly test [MLG-301] (#6103)

* Run model in EventStorage context

* Use new Docker images

* Remove pytest.skip from test_detectron2_coco_pytorch_const

* Update README.md

* Minor code reduction

* Dockerfile (listed in .detignore)

* Use determinedai repo instead of a personal repo

* Makefile for building and publishing the Docker image

* docs: Bring content changes from docusaurus-ls beta (#6121)

* docs: Bring content changes from docusaurus-ls beta

Bring over content changes from the beta including reorganization changes.

* additional organizational edits

updating index pages, adding a top nav to welcome page

* added redirects

* revisions based on feedback

* rstfmt run

* feat: display workspace icon in ProjectCard (#6125)

* fix: checkpoint GC should set resource pool (#6136) [DET-9018]

* docs: bump rstfmt version (#6138)

* chore: add dev cli option to get auth token (#6008)

add a `curl` option to help with curling various endpoints

* build(deps): bump golang.org/x/net from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0 in /proto (#6130)

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)

* perf: Improved performance of historical allocation task endpoint, removed training/validation times (#6135)

* fix: FOUNDENG-438 Podman tests from the gate are breaking znodes again (#6146)

* chore: add Toggle component to UI Kit [WEB-841] (#6144)

* chore: Add tags to UI kit [WEB-816] (#6100)

* chore: Move SelectFilter into kit folder and update it [WEB-843] (#6102)

* fix: replace `InlineEditor` with UIKit input (#6082)

* fix: replace `InlineEditor`

* fix: add modal for experiment name

* fix: layout of settings page

* fix: setting page

* fix: minor changes

* feat: move experiment `description` and `tags` into edit modal

* chore: add `N/A` when description is empty in experiment detail

* fix: value bug

* fix: revert tag; remove tag from edit modal due to design inconsistancy

* chore:  add/test pt-only images and bumpenvs (#6097)

* add pt images to some unit tests

* add pt-images to circleci config

* run bumpenvs procedure

* fix test function signatures and linting

* fix warnings linting

* fix docs

* expand unit tests coverage

* build(deps): bump github.com/prometheus/client_golang from 1.10.0 to 1.11.1 in /master (#6004)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.10.0 to 1.11.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.10.0...v1.11.1)

* feat: add det.import_from_path (#5737)

import_from_path allows users to import arbitrary code from a
checkpoint, even if the modules in the checkpoint have the same name as
modules they already have imported, but contain different code.

This is common when importing, for example, an old model_def.py that has
been updated since the original checkpoint was saved.

* fix: post rank_id correctly for fluent-less logging (#6151) [DET-8999]

* ci: send Slack notification on GKE node pool creation failure (#6152)

In order to prevent quota failures from showing up as CI failures, this
makes node pool creation failure send a Slack notification and mark the
job as successful.

I couldn't figure out how to use the Slack orb while distinguishing this
particular situation from the general failure case, so I just slapped in
a direct request to the already-configured Slack webhook for sending
messages to #ci-bots.

`circleci-agent step halt` marks the job as successful, which is why we
want a notification at all. For some reason, CircleCI fails to provide
an equivalent for marking the job as canceled or some other state
besides success/failure; we could make a call to the CircleCI API to
cancel the current job, but that would rely on having a CircleCI token
available, which we're trying to get away from.

* chore: drop unused columns from `raw_steps`, `raw_validations`, and `raw_checkpoints`. (#6110)

* fix: render spinner while auth check pending (#6098)

* chore: update hpc-launching-architecture doc - add default slurm option --no-requeue (#6141)

* docs: Content updates (#6154)

formatted the setup cluster table to match the approved version in the docusaurus ls beta

* feat: display user id in `det user list`. (#6156)

* fix: Additional tables get experiment- / workspace-specific storagePath [WEB-962] (#6128)

* fix: Additional tables get experiment- and workspace-specific storagePath

* useMemo

* fix: selection width in `move experiment` modal (#6149)

* fix: selection width in `move experiment` modal

* fix: add form wrapping

* chore: remove change

* feat: show trial hyperparameters for custom searchers [MLG-343] (#6162)

* feat: show trial hyperparameters for custom searchers [MLG-343]

* fix: corrected timestamp handling to do an interval overlap instead of contains (#6164)

* chore:  add release notes (#6167)

* chore: add release notes

* format with rstfmt

* chore: suppress help output for det dev (#6145)

avoid showing the `dev` option in `det -h` output

* chore: lock api state for backward compatibility check

* chore: bump version: 0.20.0-dev0 -> 0.20.1-dev0

* fix: separate Router and authCheck (#6170)

* fix: useMemo does not depend on trial having been loaded (#6173)

* chore: pass Labels/project/workspace to TaskSpec (#6172)

* refactor: replace user store with observables [WEB-799] (#6140)

* fix: hide Foldable menu options when button is visible (#6178)

This fixes an issue where, when using a `PageHeaderFoldable` component,
options that appear in the header always appear in the overflow menu.

* feat: add labels to GCP instances created with det deploy gcp [MLG-170] (#6147)

* feat: add labels to GCP instances created with det deploy gcp

* Changes to mimic det deploy aws --add-tags

* Add labels to other resources as well

* revert: reflag new chart experience (#6181)

* build: eliminate java dependency for typescript swagger bindings (#6139)

* fix: Close expiriment fork/ continue trial modal properly (#6174)

* fix: Continue Trial flow does not take the new `max_length` (#6168)

* fix: pass workspace ID when creating tensor board from WebUI [WEB-1019] (#6186)

* fix: don't print ':' when err msg is empty (#6190)

* fix: exp move modal (#6183)

* fix: exp move modal

* fix: minor fixes

* fix: add `archived` param and simplify query (#6175)

* fix: add `archived` param and simplify query

* chore: indent

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jerry J. Harrow <84593277+jerryharrow@users.noreply.github.com>
Co-authored-by: Nick Doiron <nick.doiron@hpe.com>
Co-authored-by: NicholasBlaskey <nick.blaskey@hpe.com>
Co-authored-by: Eric <31023784+eecsliu@users.noreply.github.com>
Co-authored-by: Keita Nonaka <keita.nonaka@hpe.com>
Co-authored-by: Erik Wilson <erik.wilson@hpe.com>
Co-authored-by: Hamid Zare <12127420+hamidzr@users.noreply.github.com>
Co-authored-by: johnkim-det <97752292+johnkim-det@users.noreply.github.com>
Co-authored-by: Ryan <rb@hpe.com>
Co-authored-by: Danny Zhu <dzhu@hpe.com>
Co-authored-by: CanmingCobble <107056780+CanmingCobble@users.noreply.github.com>
Co-authored-by: szewaiyuen6 <sze-wai.yuen@hpe.com>
Co-authored-by: Bradley Laney <bradley.laney@hpe.com>
Co-authored-by: julian-determined-ai <103522725+julian-determined-ai@users.noreply.github.com>
Co-authored-by: Corban Beaird <corban.beaird@hpe.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: liamcli <liam@determined.ai>
Co-authored-by: Ashton G <ashton.galloway@hpe.com>
Co-authored-by: thiagodallacqua-hpe <104855841+thiagodallacqua-hpe@users.noreply.github.com>
Co-authored-by: Max <max.russell@hpe.com>
Co-authored-by: Maksim <maksim.kouznetsov@hpe.com>
Co-authored-by: Emily <15078396+EmilyBonar@users.noreply.github.com>
Co-authored-by: Ilia Glazkov <ilia.glazkov@hpe.com>
Co-authored-by: Caleb Hoyoul Kang <caleb.kang@hpe.com>
Co-authored-by: Wes Turner <wesley.turner@hpe.com>
Co-authored-by: Daniel R. Hunter <103537968+drh-determined-ai@users.noreply.github.com>
Co-authored-by: Tara Charter <tara.charter@hpe.com>
Co-authored-by: rcorujo <90728398+rcorujo@users.noreply.github.com>
Co-authored-by: Guangqing Tang <40620519+gt2345@users.noreply.github.com>
Co-authored-by: MikhailKardash <mikhail.kardash@hpe.com>
Co-authored-by: Jagadeesh Madagundi <jagadeesh545@gmail.com>
Co-authored-by: gt2345 <gt2345@columbia.edu>
Co-authored-by: Trent Watson <trent.watson@hpe.com>

* handle add user

* Revert "Deepspeed Feature Branch - merge master (#6193)"

This reverts commit 05ee6a2.

* Move dsat into harness (#6254)

move dsat into harness

* fixed too-large dataset bug (#6262)

* MLG-337 (#6282)

 ds_config.json centered workflow and cleanup

* Feat deepspeed autotune fix merge conflict (#6289)

Merging with master, fixing a merge conflict, and minor cleanup.

* reset webui to latest master (#6291)

* linting fixes (#6296)

* Refactor DS AT for Trial Compatibility (#6307)

Refactored the searcher to use a json based config workflow with overwrite_deepspeed_args, as in (most of) the official examples.

* fix: remove all Close operations (#6383)

Refactored to remove all Close operations to avoid race condition errors. Also added quick no op example and fixed other bugs.

* restore util.py

* restoring more files to lastest master version

* merge in util.py changes

* Remove OOM catcher.

* Add Close operations back in

* Standardize autotuning config names.

* General clean up

* Fix dsat reporting bug.

* Minor changes.

* Clean up.

* Try/except hack around dead agent due to exit

* Minor changes.

* feat: allow users to specify zero optim search space and runner config (#6452)

* fix: do not merge user zero search config with defaults (#6464)

* populate the custom searcher logs with the correct event (#6470)

* fix: search state accounting (#6473)

* feat: add simple linear batch test searcher (#6482)

* fix: searcher refactor (#6513)

* deprecate the zero_search_config functionality (#6517)

* fix: timing fix and cleanup with tests (#6555)

* remove single should_stop and add more granular state

* adding the beginning of some autotuning tests

* real unit tests for DS AT

* Clean up the trial tracker properties, base searcher, and names

* Do not close twice

---------

Co-authored-by: Taylor Ritenour <taylor.ritenour@hpe.com>

* fix: minor cleanup (#6558)

* feat: deepspeed autotune trial based methods (#6575)

* fix: minor bug broke Trial class DS AT (#6613)

* feat: optional steps completed in context manager (#6619)

Make steps_completed in reporting context optional

Update examples to use updated context manager

minor example changes

Minor model changes

Delete old trial class examples

Add torchvision model trial example

* MLG-369: Some initial tests for DS AT (#6551)

* real unit tests for DS AT

* clean up and standardize the tests more

* also send a Close operation

* touch up the tests

* clarifications and clean up

* feat: deepspeed autotune cli args (#6643)

* Add __main__ and reuse original args on-cluster.

* Cleanup

* Adding include

* Finish adding include

* more args

* add search runner exp_id to follow on exp description

* minor comments

* Different registering system and starting the queue

* more args and cleanup

* About to use a queue (kind of)

* Attempting to enforce trial constraints

* Add early stopping as arg

* Remove old autotuning args and get them instead from cli

* Corrected args bugs

* Fixed many refactoring bugs

* Cleanup and bug fixes

* Move exp_conf edits to _run_dsat

* Add zero stages arg

* Move exp conf changes out of searcher and use zero_stages arg

* Fix simple searcher bug

* minor cleanup

* cleanup

* Starting refactor away from searcher state

* Changed searcher state computation to trial tracker

* Revert to single configs everywhere

* Update readme

* Edit added description text

* Add closed attr to Trials and bug fixes

* clarifying comment

* Rebase onto latest feature branch

* Add actual deque, fix bugs

* Remove print test

* small trial example fix

* Clean up try/except hack

* config edits

* feat: visualize cli args (#6651)

* Add CLI args to config hparams for visualization

* remove pickle path arg

* fix up the dsat tests (#6664)

* fix up the dsat tests

* make sure to pass through the args parsing function

* touch up the tests so that they can issue a failure to the experiment (#6676)

* chore: update deepspeed to 0.8.3 [MLG-399] (#6685)

* feat: hf trainer examples (#6687)

* chore: refactor dsat module to be independent of deepspeed imports [MLG-499] (#6694)

* chore: refactor dsat module to be independent of deepspeed imports [MLG-499]

* also update the test and det_callback.py

* chore: move over the examples for DSAT [MLG-500] (#6717)

* feat: add deepspeed autotuning examples [MLG-500]

* some clean up and UX improvements

* remove double parens, make sure that orchestrator id is on the far left

* feat: add follow on exp option (#6720)

* feat: migrate the huggingface det_callback [MLG-487] (#6724)

* migrating the det_callback

* feat: migrate the huggingface det_callback [MLG-487]

* don't export DetCallback through the top level integrations

* feat: minor test updates (#6746)

* feat: use lock with json overwrite for hf (#6742)

* feat: use lock with json overwrite for hf

* handle merge with our previous DetCallback refactors

---------

Co-authored-by: Garrett Goon <garrett.goon@hpe.com>

* feat: basic trial tracker tests (#6754)

* fix: hf overwrite bug

* fix: remove old import (#6761)

* feat: adding e2e tests for DSAT, enabling searcher to shutdown client experiment [MLG-369] (#6781)

* migrating the det_callback

* feat: migrate the huggingface det_callback [MLG-487]

* wip working on the e2e tests

* getting the basics of the tests running. Still appear to be issues though

* fixing up the tests

* fixing up tests

* handle cases where explicit exceptions are raised in the dsat search runner

* clean up for merging

* revert restarts change

* feat: add search method tests (#6785)

* add search method tests

* update hf ex readme

* quick fix for the unit tests (#6796)

* feat: write best ds config json to checkpoint (#6787)

* feat: refactor argparse into subparsers (#6801)

* feat: add binary search (#6806)

* fix: move lock to fix hf overwrites (#6828)

* fix: small fixes (#6825)

* expand message for model profile info failure

* correct the progress calculation

* Remove autotuning section from checkpointed best configs

* also write the best metrics to the checkpoint

* fix: proper placement of start/end profile step (#6834)

* feat: more random search test (#6837)

* chore: stabilize static typed python [MLG-498] (#6846)

* flake8 fixes

* mypy issues

* updates according to comments for mypy changes (#6864)

* feat: asha (#6852)

* merged in prev asha code

* starting asha tests

* more tests and cleanup

* test cleanup

* asha params closer to current nomenclature

* refactor asha args

* import fix

* mypy

* add asha to __all__

* add search data to stage 3 test

* chore: move hf examples (#6871)

* replace old hf integrations examples with new ones

* fix no-op bug in hf helper function

* fix helper function imports

* update readme

* use the python module for `searcher` directly (#6883)

* use the python module for `searcher` directly rather than importing the individual elements

* additional fixes

* feat: clean up dsat examples (#6891)

* moving files

* moving more files

* more file movement

* stage 1 in config

* core api script cleanup

* deepspeed.yaml core api cleanup

* align deepspeed.yaml files

* align ds_config.json files

* remove lr scheduler

* shorten length to 100

* model_def.py cleanup

* Added checkpointing and better metrics reporting

* cleaned up readme

* change example dir name

* add torchvision examples to test_official.py

* starting e2e tests

* add all e2e tests

* remove accidental double test

* chore: to do cleanup (#6895)

* cli docs

* cli doc strings

* remove todo comment

* update doc strings in dsat _utils.py

* searcher class doc strings

* More search method doc strings for non-public classes

* remove searcher state tests

* remove many todos in _dsat_search_method.py

* remove todos elsewhere

* fix: do not schedule the same trial twice (#6896)

* attempting to fix tests (#6899)

* fix: move examples dir one level up and finish docs/Makefile changes (#6898)

* fix: remove improper test_official.py tests (#6900)

* fix docs formatting (#6902)

* fix docs formatting

* add deepspeed autotune directory to example builds

* support hf examples (#6910)

* feat: ds config from include (#6905)

* move overwrite_deepspeed_config back to det.pytorch.deepspeed

* allow for the ds json to be --include'd

* self.hparams -> hparams bug

* doc string edits

* move ds_config.json back inside of no_op

* chore: update the custom searcher docs [MLG-447] (#6934)

* updating the docs for custom searcher wip

* wip, fixing up the docs, making sure things and clean and link properly

* chore: update the custom searcher docs [MLG-447]

* updates according to comments

* fix docs build

* changes according to comments in dsat branch (#6943)

* changes according to comments.

- removing no_op
- removing cache_dir in hf examples
- removing erroneous release-notes

* revert the changes to the environments so we are in sync with bumpenvs

* adjust the huggingface versions to the current minor version

* update version

* bug: fully wrap hf JSON loading around a FileLock (#6950)

* feat: deepspeed autotune user guide (#6929)

* starting dsat user guide

* import cleanup in hf examples

* more editing

* starting to list cli options

* formatting

* git mv hf examples to make more descriptive dirs

* remove TODO

* incorporating feedback

* links to examples

* cleanup

* Update cli help

* general cli options cleanup

* incorporate taylor's second round of feedback

* incorporate tara's comments

* fix: no auto in hf cli (#6963)

* add int check to hf cli arg overwrite

* fix accidential trivial test

* new test ensuring auto not used in hf cli args

* add checks against copying "auto" to HF CLI entrypoint

* add link to the dsat user guide (#6964)

* add link to the dsat user guide

* updated wording

* add defaults to dsat cli help menu (#6970)

* feat: more tests and asha cleanup (#6966)

* account for asha early stopping in test

* clean up lineage_completed_rung

* more full mock experiment tests

* Only skip completed trials added to queue

* base rungs off latest, rather than root

* max trial computation cleanup

* only include curr_rung <= rung_idx trial in best computation

* promotion respects rung idx test

* test cleanup

* more test cleanup

* add test_get_best_trial_in_lineage

* doc string cleanup

* fix up test_full_experiment_reverse_ordered_results

* minor wording edit

* always pop off highest curr_rung asha trial next

* fix the doc builds, add a release-note (#6973)

* fix the doc builds, add a release-note

* update docs names

* make flake8 behave

* update by the pre-commit complaints

* fix: readme cleanup (#6974)

* touch up hf trainer readme

* shorten up and simplify torchvision readme

* feat: update defaults and small tweaks (#6975)

* trials_per_random_config 5

* max trials 64

* min binary search trials 3

* fix text by avoiding trivial search range

* remove should_discard function to avoid possible locking

* increase timeouts for mock tests

* mypy fix

* update ceiling computation for new random trials

* base the ceiling off the max mbs computation, not the midpoint

* schedule longer lineages first in asha

* update docs to reflect new defaults

* fmt examples

* make isort behave

* address some comments about logging levels and comments

* remote erroneous TODO

* small spelling mistake

* move the `determined.integrations.huggingface.DetCallback` to `determined.transformers.DetCallback`

* fix: comment and environment cleanup (#6988)

* explain try/except in search runner

* remove while true and comments in _deepspeed_trial.py

* fix all deepspeed yaml files

* remove step id comment

* more todo cleanup

* make sure the docs build again

* fix the names of the e2e tests and in README

* don't run so many e2e_tests for deepspeed

* reduce hf image class ds slots per trial (#6998)

* fix e2e tests

* fix the convergence tests

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Maksim Kouznetsov <maksim.kouznetsov@hpe.com>
Co-authored-by: Garrett Goon <garrett.goon@hpe.com>
Co-authored-by: Garrett Goon <44747910+garrett361@users.noreply.github.com>
Co-authored-by: Jerry J. Harrow <84593277+jerryharrow@users.noreply.github.com>
Co-authored-by: Nick Doiron <nick.doiron@hpe.com>
Co-authored-by: NicholasBlaskey <nick.blaskey@hpe.com>
Co-authored-by: Eric <31023784+eecsliu@users.noreply.github.com>
Co-authored-by: Keita Nonaka <keita.nonaka@hpe.com>
Co-authored-by: Erik Wilson <erik.wilson@hpe.com>
Co-authored-by: Hamid Zare <12127420+hamidzr@users.noreply.github.com>
Co-authored-by: johnkim-det <97752292+johnkim-det@users.noreply.github.com>
Co-authored-by: Ryan <rb@hpe.com>
Co-authored-by: Danny Zhu <dzhu@hpe.com>
Co-authored-by: CanmingCobble <107056780+CanmingCobble@users.noreply.github.com>
Co-authored-by: szewaiyuen6 <sze-wai.yuen@hpe.com>
Co-authored-by: Bradley Laney <bradley.laney@hpe.com>
Co-authored-by: julian-determined-ai <103522725+julian-determined-ai@users.noreply.github.com>
Co-authored-by: Corban Beaird <corban.beaird@hpe.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: liamcli <liam@determined.ai>
Co-authored-by: Ashton G <ashton.galloway@hpe.com>
Co-authored-by: thiagodallacqua-hpe <104855841+thiagodallacqua-hpe@users.noreply.github.com>
Co-authored-by: Max <max.russell@hpe.com>
Co-authored-by: Emily <15078396+EmilyBonar@users.noreply.github.com>
Co-authored-by: Ilia Glazkov <ilia.glazkov@hpe.com>
Co-authored-by: Caleb Hoyoul Kang <caleb.kang@hpe.com>
Co-authored-by: Wes Turner <wesley.turner@hpe.com>
Co-authored-by: Daniel R. Hunter <103537968+drh-determined-ai@users.noreply.github.com>
Co-authored-by: Tara Charter <tara.charter@hpe.com>
Co-authored-by: rcorujo <90728398+rcorujo@users.noreply.github.com>
Co-authored-by: Guangqing Tang <40620519+gt2345@users.noreply.github.com>
Co-authored-by: MikhailKardash <mikhail.kardash@hpe.com>
Co-authored-by: Jagadeesh Madagundi <jagadeesh545@gmail.com>
Co-authored-by: gt2345 <gt2345@columbia.edu>
Co-authored-by: Trent Watson <trent.watson@hpe.com>
Co-authored-by: Emily Bonar <emily.bonar@hpe.com>
erikwilson added a commit to determined-ai/determined that referenced this issue Jun 1, 2023
* Save model info; add Core API DeepSpeed example.

* extract ds profiler results

* Place activation mem into search metric.

* Obtain metrics from the model info file

* remove unused code

* adapted to passing dicts into report_completed

* cleanup and small changes

* refactoring and trial helper classes

* initial random search logic started

* minor changes

* minor cleanup

* use context manager, expanded base searcher

* remove ds_autotuning dir from examples

* minor edits

* readme updates and other cleanup

* bug fixes and a hack to avoid needing nested model dirs

* README updates

* Feat deepspeed autotune (#5875)

Added current Core API prototype.

* switched to triggering their autotuning in our trials

* remove checkpoint wrapper

* implementing checkpointing

* implementing checkpointing

* feat: allow includes in custom searcher experiment [MLG-338] (#6091)

* cleanups, bug fixing, and more examples

* adding native dsat tests

* cleanup

* minor edits

* fix missing is_chief

* Feat deepspeed autotune (#6159)

Trigger the native DS AT exit behavior for all trials.

* minor edits

* Feat deepspeed autotune git fixes (#6180)

deleted extraneous files

* readme fix and make cifar10 work off grenoble (#6187)

* Deepspeed Feature Branch - merge master (#6193)

* docs: Improvements to HPC launcher docs (#6042)

* Provide inline info about agent-specific scheduling options that do not apply to HPC Launcher
  configurations.
* Identify enroot-specific differences from docker (like for Singularity)
* Provide reference to custom resource pools as an option to deal with non-homogenous
  Slurm/PBS partitions.

* chore: Allow newer Node versions 17-19 (#6038)

* fix: k8s rm gives wrong slot count in rendezvous (#6044)

* chore: bump version: 0.19.12-dev0 -> 0.20.0-dev0 (#6048)

* chore: remove `applicableRoutespace` (#6040)

* chore: warning fixes in web (#6041)

* chore: fix warnings

* chore: change eslint rules

* chore: fix gpu nightly errors (#6046)

* chore: missing nodev18 (#6050)

* chore: add a dedicated exception for cli errors (#5649)

switch sys.exit calls in cli with a new user-facing exception.

* fix: SSO layout (#6053)

* chore: clean up UI kit (#6039)

* fix: lopsided training with 2,1 gpus (#6054)

There was a guard to skip local zmq setup when local_size < 2, but that
became no longer valid when local_size varied from worker to worker.

The result is one extra global allgather in some cases, no big deal.

* docs: add rbac ntsc & mr release notes (#6049)

* chore: manual bump version (#6058)

* ci: retry downloading GKE auth plugin [DET-8956] (#6056)

We got a failure due to a timeout on this, so let it retry a few times.

* docs: update Singularity known issues. (#6047)

* fix: Add #rank to worker segment instead of timestamp segment of Pytorch Profiler files [MLG-326] (#6037)

* Add pytorch profiler specific handling logic for appending rank to file name

* Change to use f-string

* fix only file name being passed in

* remove print statement

* fix: handle agent shutdown msg (#6065)

* chore: manually bump vite version (#6066)

* feat: Display better x-axis ticks on charts with time axis [WEB-849] (#6051)

* remove xTickValues from props now that it can be calculated internally

* test: add logging to a flaky test (#6068)

Test is flaky but hard to pin down, so add some prints for next time.

* fix: Unrelated models are shown in a workspace model registry tab (#6067)

* feat: Added task-based historical allocation endpoint [DET-8537] (#6015)

* fix: show `not found` and `spinner` properly (#6070)

* fix: show `not found` and `spinner` properly

* chore: change home redirect path

* fix: projectDetail page

* fix: project.workspaceId

* build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto (#6061)

* build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto

Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.5 to 0.3.8.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](golang/text@v0.3.5...v0.3.8)

---
updated-dependencies:
- dependency-name: golang.org/x/text
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: gpt-neox docker image and startup hook to work for non-provileged user (#6060)

* test: add locking around migration [DET-8957] (#6071)

In integration tests, multiple processes can attempt to run migrations
against the same database at once, which can lead to errors because
PostgreSQL's `CREATE TABLE IF NOT EXISTS` is not great with concurrency
(it allows for a time-of-check/time-of-use failure).

The specific errors we were seeing were conflicts in the pg_type table,
so the code now locks that table for the duration of the migration
transaction.

More information:
https://www.postgresql.org/message-id/CA+TgmoZAdYVtwBfp1FL2sMZbiHCWT4UPrzRLNnX1Nb30Ku3-gg@mail.gmail.com
https://stackoverflow.com/questions/29900845

* fix: logging inconsistent newlines in slurm (#6074)

* fix: checkpoint helper for points > 1000 and points > maxDatapoints (#6069)

* fix: replace migration table lock with advisory lock (#6077)

Taking a table lock sometimes runs into permissions issues; advisory
locking should avoid that.

Also, I realized the locking should probably be after the deferred
transaction close instead of before.

* build: check npm version on install (#6079)

This removes the `check-requirements` make target in the react folder
and replaces it with npm's native version check against the engines
property. This should make managing the node version slightly easier
because there's one less place to check.

* build: Apply webui lint fixes in precommit (#6078)

* allow linters to fix in precommit

This updates the web linters to automatically apply fixes when doing a
pre-commit check. This should ideally streamline the commit process to
reduce the amount of times the user needs to run prettier and eslint
before committing.

* tweak stylelint and eslint commands

* stage changed files

* type file_paths argument

* ci: adjust target accuracy for a test (#6085)

We got one failure [1] where the accuracy ended up just a hair below
0.83, so drop the target.

[1] https://app.circleci.com/pipelines/github/determined-ai/determined/33883/workflows/4a5d3257-6061-4f4d-bd66-096a580a5959/jobs/1194282/steps

* chore: UserBadge moves into design kit (#6086)

* chore: Remove chart feature flag, remove unused code [WEB-930] (#6064)

* tooltipsPlugin and TrialDetailsOverview alternates go into place
* checkpoint helper for points > 1000 and points > maxDatapoints
* move former LearningCurveChart into TrialsComparison
* sync up with #6069 changes

* fix: replace defaultvalue with initialValue (#6076)

* fix: Dont suggest moving model into its current workspace (#6088)

* ci: delete database at beginning of det deploy tests [DET-8937] (#6089)

Previously, the database was being retained between tests, sometimes
causing tests to fail when extra agents appeared due to agent
reattach. The tests should generally be independent anyway, so reset
the database (by default, with an option to disable) each time the
cluster or master comes up.

* feat: add Facepile component (#6081)

* fix: pre-commit web bug fix (#6090)

* ci: make GKE test jobs run serially (#6096)

We keep hitting GKE GPU quotas; this will probably help with that.

* fix: GPU counting for k8s cluster info page (#6094)

* fix: test-e2e-gke-parallel use t4 (#6093)

* ci: retry protoc download (#6095)

We got an incorrect file downloaded one time [1], so retry this
download, like in 2906257 (#5996).

[1] https://app.circleci.com/pipelines/github/determined-ai/determined/34074/workflows/e48681f8-8b75-4349-82eb-06e922d8bfcb/jobs/1202610

* refactor: add Card to UI Kit [WEB-818] (#5893)

* docs: Launcher doc improvements (#6099)

- Generalize journalctl command example --since option to work on Unbutu.
- Clarify user_name/group_name account requirements.

* feat: Attend to TODOS accross the code base (#6087)

* perf: tweak metrics series query. (#6105)

* chore: race could cause run container to return a different error than expected [DET-8870] (#6092)

* chore: add more metadata to slurm logs (#6030)

* chore: remove `ExpCompareMetricNames`, `ExpCompareTrialsSample` endpoints. (#6106)

* docs: fix reported DataPoint label doc (#6107)

* fix: tolerate additional non-CPU, non-GPU quotas in k8s (#6109)

* fix: stop filtering of valid options to reflect build issues (#6116)

* fix: modal theme color (#6117)

* fix: add bgColor in trial comparison table (#6119)

* fix: browser console warnings (#6122)

* fix: browser console warnings

* fix: remove spread operator

* chore: UIKit Pivot renaming (#6120)

* fix: correct minor JSX syntax (#6126)

* docs: add myst_parser extension (#6127)

We would like to support markdown-format documentation.  There are still
some kinks to be worked out with converting rst to myst files, but this
is a start.

* docs: fix some broken redirects (#6129)

* feat: add 3rd batch of TODO removals (#6115)

* feat: generic proxy configs [DET-8761] (#5978)

* build: [MLG-336] Limit the version of protobuf (#6134)

build: [MLG-336] Limit the version of protobuf

Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce.

* chore: update detectron2 example to use v0.6 and reenable nightly test [MLG-301] (#6103)

* Run model in EventStorage context

* Use new Docker images

* Remove pytest.skip from test_detectron2_coco_pytorch_const

* Update README.md

* Minor code reduction

* Dockerfile (listed in .detignore)

* Use determinedai repo instead of a personal repo

* Makefile for building and publishing the Docker image

* docs: Bring content changes from docusaurus-ls beta (#6121)

* docs: Bring content changes from docusaurus-ls beta

Bring over content changes from the beta including reorganization changes.

* additional organizational edits

updating index pages, adding a top nav to welcome page

* added redirects

* revisions based on feedback

* rstfmt run

* feat: display workspace icon in ProjectCard (#6125)

* fix: checkpoint GC should set resource pool (#6136) [DET-9018]

* docs: bump rstfmt version (#6138)

* chore: add dev cli option to get auth token (#6008)

add a `curl` option to help with curling various endpoints

* build(deps): bump golang.org/x/net from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0 in /proto (#6130)

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)

* perf: Improved performance of historical allocation task endpoint, removed training/validation times (#6135)

* fix: FOUNDENG-438 Podman tests from the gate are breaking znodes again (#6146)

* chore: add Toggle component to UI Kit [WEB-841] (#6144)

* chore: Add tags to UI kit [WEB-816] (#6100)

* chore: Move SelectFilter into kit folder and update it [WEB-843] (#6102)

* fix: replace `InlineEditor` with UIKit input (#6082)

* fix: replace `InlineEditor`

* fix: add modal for experiment name

* fix: layout of settings page

* fix: setting page

* fix: minor changes

* feat: move experiment `description` and `tags` into edit modal

* chore: add `N/A` when description is empty in experiment detail

* fix: value bug

* fix: revert tag; remove tag from edit modal due to design inconsistancy

* chore:  add/test pt-only images and bumpenvs (#6097)

* add pt images to some unit tests

* add pt-images to circleci config

* run bumpenvs procedure

* fix test function signatures and linting

* fix warnings linting

* fix docs

* expand unit tests coverage

* build(deps): bump github.com/prometheus/client_golang from 1.10.0 to 1.11.1 in /master (#6004)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.10.0 to 1.11.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.10.0...v1.11.1)

* feat: add det.import_from_path (#5737)

import_from_path allows users to import arbitrary code from a
checkpoint, even if the modules in the checkpoint have the same name as
modules they already have imported, but contain different code.

This is common when importing, for example, an old model_def.py that has
been updated since the original checkpoint was saved.

* fix: post rank_id correctly for fluent-less logging (#6151) [DET-8999]

* ci: send Slack notification on GKE node pool creation failure (#6152)

In order to prevent quota failures from showing up as CI failures, this
makes node pool creation failure send a Slack notification and mark the
job as successful.

I couldn't figure out how to use the Slack orb while distinguishing this
particular situation from the general failure case, so I just slapped in
a direct request to the already-configured Slack webhook for sending
messages to #ci-bots.

`circleci-agent step halt` marks the job as successful, which is why we
want a notification at all. For some reason, CircleCI fails to provide
an equivalent for marking the job as canceled or some other state
besides success/failure; we could make a call to the CircleCI API to
cancel the current job, but that would rely on having a CircleCI token
available, which we're trying to get away from.

* chore: drop unused columns from `raw_steps`, `raw_validations`, and `raw_checkpoints`. (#6110)

* fix: render spinner while auth check pending (#6098)

* chore: update hpc-launching-architecture doc - add default slurm option --no-requeue (#6141)

* docs: Content updates (#6154)

formatted the setup cluster table to match the approved version in the docusaurus ls beta

* feat: display user id in `det user list`. (#6156)

* fix: Additional tables get experiment- / workspace-specific storagePath [WEB-962] (#6128)

* fix: Additional tables get experiment- and workspace-specific storagePath

* useMemo

* fix: selection width in `move experiment` modal (#6149)

* fix: selection width in `move experiment` modal

* fix: add form wrapping

* chore: remove change

* feat: show trial hyperparameters for custom searchers [MLG-343] (#6162)

* feat: show trial hyperparameters for custom searchers [MLG-343]

* fix: corrected timestamp handling to do an interval overlap instead of contains (#6164)

* chore:  add release notes (#6167)

* chore: add release notes

* format with rstfmt

* chore: suppress help output for det dev (#6145)

avoid showing the `dev` option in `det -h` output

* chore: lock api state for backward compatibility check

* chore: bump version: 0.20.0-dev0 -> 0.20.1-dev0

* fix: separate Router and authCheck (#6170)

* fix: useMemo does not depend on trial having been loaded (#6173)

* chore: pass Labels/project/workspace to TaskSpec (#6172)

* refactor: replace user store with observables [WEB-799] (#6140)

* fix: hide Foldable menu options when button is visible (#6178)

This fixes an issue where, when using a `PageHeaderFoldable` component,
options that appear in the header always appear in the overflow menu.

* feat: add labels to GCP instances created with det deploy gcp [MLG-170] (#6147)

* feat: add labels to GCP instances created with det deploy gcp

* Changes to mimic det deploy aws --add-tags

* Add labels to other resources as well

* revert: reflag new chart experience (#6181)

* build: eliminate java dependency for typescript swagger bindings (#6139)

* fix: Close expiriment fork/ continue trial modal properly (#6174)

* fix: Continue Trial flow does not take the new `max_length` (#6168)

* fix: pass workspace ID when creating tensor board from WebUI [WEB-1019] (#6186)

* fix: don't print ':' when err msg is empty (#6190)

* fix: exp move modal (#6183)

* fix: exp move modal

* fix: minor fixes

* fix: add `archived` param and simplify query (#6175)

* fix: add `archived` param and simplify query

* chore: indent

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jerry J. Harrow <84593277+jerryharrow@users.noreply.github.com>
Co-authored-by: Nick Doiron <nick.doiron@hpe.com>
Co-authored-by: NicholasBlaskey <nick.blaskey@hpe.com>
Co-authored-by: Eric <31023784+eecsliu@users.noreply.github.com>
Co-authored-by: Keita Nonaka <keita.nonaka@hpe.com>
Co-authored-by: Erik Wilson <erik.wilson@hpe.com>
Co-authored-by: Hamid Zare <12127420+hamidzr@users.noreply.github.com>
Co-authored-by: johnkim-det <97752292+johnkim-det@users.noreply.github.com>
Co-authored-by: Ryan <rb@hpe.com>
Co-authored-by: Danny Zhu <dzhu@hpe.com>
Co-authored-by: CanmingCobble <107056780+CanmingCobble@users.noreply.github.com>
Co-authored-by: szewaiyuen6 <sze-wai.yuen@hpe.com>
Co-authored-by: Bradley Laney <bradley.laney@hpe.com>
Co-authored-by: julian-determined-ai <103522725+julian-determined-ai@users.noreply.github.com>
Co-authored-by: Corban Beaird <corban.beaird@hpe.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: liamcli <liam@determined.ai>
Co-authored-by: Ashton G <ashton.galloway@hpe.com>
Co-authored-by: thiagodallacqua-hpe <104855841+thiagodallacqua-hpe@users.noreply.github.com>
Co-authored-by: Max <max.russell@hpe.com>
Co-authored-by: Maksim <maksim.kouznetsov@hpe.com>
Co-authored-by: Emily <15078396+EmilyBonar@users.noreply.github.com>
Co-authored-by: Ilia Glazkov <ilia.glazkov@hpe.com>
Co-authored-by: Caleb Hoyoul Kang <caleb.kang@hpe.com>
Co-authored-by: Wes Turner <wesley.turner@hpe.com>
Co-authored-by: Daniel R. Hunter <103537968+drh-determined-ai@users.noreply.github.com>
Co-authored-by: Tara Charter <tara.charter@hpe.com>
Co-authored-by: rcorujo <90728398+rcorujo@users.noreply.github.com>
Co-authored-by: Guangqing Tang <40620519+gt2345@users.noreply.github.com>
Co-authored-by: MikhailKardash <mikhail.kardash@hpe.com>
Co-authored-by: Jagadeesh Madagundi <jagadeesh545@gmail.com>
Co-authored-by: gt2345 <gt2345@columbia.edu>
Co-authored-by: Trent Watson <trent.watson@hpe.com>

* handle add user

* Revert "Deepspeed Feature Branch - merge master (#6193)"

This reverts commit 05ee6a2.

* Move dsat into harness (#6254)

move dsat into harness

* fixed too-large dataset bug (#6262)

* MLG-337 (#6282)

 ds_config.json centered workflow and cleanup

* Feat deepspeed autotune fix merge conflict (#6289)

Merging with master, fixing a merge conflict, and minor cleanup.

* reset webui to latest master (#6291)

* linting fixes (#6296)

* Refactor DS AT for Trial Compatibility (#6307)

Refactored the searcher to use a json based config workflow with overwrite_deepspeed_args, as in (most of) the official examples.

* fix: remove all Close operations (#6383)

Refactored to remove all Close operations to avoid race condition errors. Also added quick no op example and fixed other bugs.

* restore util.py

* restoring more files to lastest master version

* merge in util.py changes

* Remove OOM catcher.

* Add Close operations back in

* Standardize autotuning config names.

* General clean up

* Fix dsat reporting bug.

* Minor changes.

* Clean up.

* Try/except hack around dead agent due to exit

* Minor changes.

* feat: allow users to specify zero optim search space and runner config (#6452)

* fix: do not merge user zero search config with defaults (#6464)

* populate the custom searcher logs with the correct event (#6470)

* fix: search state accounting (#6473)

* feat: add simple linear batch test searcher (#6482)

* fix: searcher refactor (#6513)

* deprecate the zero_search_config functionality (#6517)

* fix: timing fix and cleanup with tests (#6555)

* remove single should_stop and add more granular state

* adding the beginning of some autotuning tests

* real unit tests for DS AT

* Clean up the trial tracker properties, base searcher, and names

* Do not close twice

---------

Co-authored-by: Taylor Ritenour <taylor.ritenour@hpe.com>

* fix: minor cleanup (#6558)

* feat: deepspeed autotune trial based methods (#6575)

* fix: minor bug broke Trial class DS AT (#6613)

* feat: optional steps completed in context manager (#6619)

Make steps_completed in reporting context optional

Update examples to use updated context manager

minor example changes

Minor model changes

Delete old trial class examples

Add torchvision model trial example

* MLG-369: Some initial tests for DS AT (#6551)

* real unit tests for DS AT

* clean up and standardize the tests more

* also send a Close operation

* touch up the tests

* clarifications and clean up

* feat: deepspeed autotune cli args (#6643)

* Add __main__ and reuse original args on-cluster.

* Cleanup

* Adding include

* Finish adding include

* more args

* add search runner exp_id to follow on exp description

* minor comments

* Different registering system and starting the queue

* more args and cleanup

* About to use a queue (kind of)

* Attempting to enforce trial constraints

* Add early stopping as arg

* Remove old autotuning args and get them instead from cli

* Corrected args bugs

* Fixed many refactoring bugs

* Cleanup and bug fixes

* Move exp_conf edits to _run_dsat

* Add zero stages arg

* Move exp conf changes out of searcher and use zero_stages arg

* Fix simple searcher bug

* minor cleanup

* cleanup

* Starting refactor away from searcher state

* Changed searcher state computation to trial tracker

* Revert to single configs everywhere

* Update readme

* Edit added description text

* Add closed attr to Trials and bug fixes

* clarifying comment

* Rebase onto latest feature branch

* Add actual deque, fix bugs

* Remove print test

* small trial example fix

* Clean up try/except hack

* config edits

* feat: visualize cli args (#6651)

* Add CLI args to config hparams for visualization

* remove pickle path arg

* fix up the dsat tests (#6664)

* fix up the dsat tests

* make sure to pass through the args parsing function

* touch up the tests so that they can issue a failure to the experiment (#6676)

* chore: update deepspeed to 0.8.3 [MLG-399] (#6685)

* feat: hf trainer examples (#6687)

* chore: refactor dsat module to be independent of deepspeed imports [MLG-499] (#6694)

* chore: refactor dsat module to be independent of deepspeed imports [MLG-499]

* also update the test and det_callback.py

* chore: move over the examples for DSAT [MLG-500] (#6717)

* feat: add deepspeed autotuning examples [MLG-500]

* some clean up and UX improvements

* remove double parens, make sure that orchestrator id is on the far left

* feat: add follow on exp option (#6720)

* feat: migrate the huggingface det_callback [MLG-487] (#6724)

* migrating the det_callback

* feat: migrate the huggingface det_callback [MLG-487]

* don't export DetCallback through the top level integrations

* feat: minor test updates (