Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Observability] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB #27952

Open
joshua-cogliati-inl opened this issue Aug 17, 2022 · 3 comments
Open

[Observability] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB #27952

joshua-cogliati-inl opened this issue Aug 17, 2022 · 3 comments
Labels
bug Something that is supposed to be working; but isn't dashboard Issues specific to the Ray Dashboard observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling P2 Important issue, but not time-critical

Comments

@joshua-cogliati-inl
Copy link

What happened + What you expected to happen

  1. The bug: ray timeline does not seem to work.
  2. Expected behavior: ray timeline would output a json file.
  3. logs etc:
$ ray timeline --address='10.159.13.9:59086'
2022-08-17 08:38:21,165	INFO scripts.py:1727 -- Connecting to Ray instance at 10.159.13.9:59086.
2022-08-17 08:38:21,165	INFO worker.py:957 -- Connecting to existing Ray cluster at address: 10.159.13.9:59086
2022-08-17 08:38:47,311	INFO scripts.py:1733 -- Trace file being written to /tmp/ray-timeline-2022-08-17_08-38-47.json
(pid=gcs_server) [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:410] ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB: 16494724187
(pid=gcs_server) [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:410] ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB: 16509463013
(pid=gcs_server) [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:410] ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB: 16524627863
(pid=gcs_server) [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:410] ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB: 16539796049
...

Ray status seems fine:

 ray status --address='10.159.13.9:59086'
======== Autoscaler status: 2022-08-17 08:41:02.793078 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_769e5331d9364c80db37c1400aced64a0515045125e5274b263252ab
 1 node_1dc43a88c2e9f0d9862f7d9e73abcc8ed4612caa3bd642370730f8eb
 1 node_8d68e860fe245b90a2a7eecdac4c1605a65a9e1753094420b3647ffb
 1 node_7b21b4cce2e071c3a242f9968052ca33a39e75b9f595f8bdfaaacdfb
 1 node_049948c1360b2a340771483cb796626681a6a36b9aba91ad9abc38c2
 1 node_736d602ec8b9cad236273ee49af94b01ca54f22cfe8ad8eb672fc73d
 1 node_d8970ba57ac61fe6015ee223672a9985aaf21f18aadc72f5ae73216c
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 4.0/70.0 CPU
 0.00/863.949 GiB memory
 0.94/374.255 GiB object_store_memory

Demands:
 (no resource demands)

(Note: logger.info(f"Trace file being written to {filename}") added before calling ray.timeline in ray/scripts/scripts.py )

Versions / Dependencies

$ pip list
Package                  Version
------------------------ ---------
absl-py                  0.15.0
aiohttp                  3.8.1
aiohttp-cors             0.7.0
aiosignal                1.2.0
astor                    0.8.1
astroid                  2.11.6
async-timeout            4.0.2
asynctest                0.13.0
attrs                    22.1.0
blessed                  1.19.1
blinker                  1.4
brotlipy                 0.7.0
cached-property          1.5.2
cachetools               5.2.0
certifi                  2022.6.15
cffi                     1.15.1
cftime                   1.6.0
charset-normalizer       2.1.0
click                    8.0.4
cloudpickle              1.6.0
colorama                 0.4.5
colorful                 0.5.4
coverage                 6.4.4
cryptography             37.0.4
cycler                   0.11.0
DaemonLite               0.0.2
dill                     0.3.5.1
distlib                  0.3.5
elementpath              3.0.2
filelock                 3.8.0
frozenlist               1.3.1
gast                     0.2.2
google-api-core          2.8.2
google-auth              2.10.0
google-auth-oauthlib     0.4.6
google-pasta             0.2.0
googleapis-common-protos 1.56.4
gpustat                  1.0.0rc1
grpcio                   1.43.0
h5py                     3.6.0
h5py-wrapper             1.1.0
hdfdict                  0.3.1
idna                     3.3
imageio                  2.9.0
importlib-metadata       4.11.4
importlib-resources      5.9.0
isort                    5.10.1
joblib                   1.1.0
jsonschema               4.10.0
Keras-Applications       1.0.8
Keras-Preprocessing      1.1.2
kiwisolver               1.4.4
lazy-object-proxy        1.7.1
lxml                     4.8.0
Markdown                 3.4.1
matplotlib               3.2.2
mccabe                   0.7.0
msgpack                  1.0.4
multidict                6.0.2
netCDF4                  1.5.8
nose                     1.3.7
numexpr                  2.8.0
numpy                    1.18.5
numpy-financial          1.0.0
nvidia-ml-py             11.495.46
oauthlib                 3.2.0
opencensus               0.11.0
opencensus-context       0.1.3
opt-einsum               3.3.0
pandas                   1.1.5
patsy                    0.5.2
Pillow                   9.2.0
pip                      22.2.2
pkgutil_resolve_name     1.3.10
platformdirs             2.5.2
ply                      3.11
prometheus-client        0.13.1
protobuf                 3.20.1
psutil                   5.9.1
py-spy                   0.3.12
pyasn1                   0.4.8
pyasn1-modules           0.2.7
pycparser                2.21
PyJWT                    2.4.0
pylint                   2.14.5
Pyomo                    6.4.1
pyOpenSSL                22.0.0
pyparsing                3.0.9
pyrsistent               0.18.1
PySocks                  1.7.1
pytest-runner            5.3.1
python-dateutil          2.8.2
pytz                     2022.2.1
pyu2f                    0.1.5
PyUtilib                 6.0.0
PyYAML                   6.0
ray                      1.13.0
requests                 2.28.1
requests-oauthlib        1.3.1
rsa                      4.9
scikit-learn             0.24.2
scipy                    1.5.3
setuptools               59.8.0
six                      1.16.0
smart-open               6.0.0
statsmodels              0.12.2
tensorboard              2.8.0
tensorboard-data-server  0.6.0
tensorboard-plugin-wit   1.8.1
tensorflow               2.0.0
tensorflow-estimator     2.5.0
termcolor                1.1.0
threadpoolctl            3.1.0
tomli                    2.0.1
tomlkit                  0.11.4
tornado                  6.2
typed-ast                1.5.4
typing_extensions        4.3.0
urllib3                  1.26.11
virtualenv               20.16.3
wcwidth                  0.2.5
Werkzeug                 0.16.1
wheel                    0.37.1
wrapt                    1.14.1
xarray                   0.16.2
xmlschema                2.0.2
xmltodict                0.12.0
yarl                     1.7.2
zipp                     3.8.1
$ python
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
$ conda list
# packages in environment at /home/cogljj/miniconda3/envs/raven_libraries_heron_newer_ray:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_tflow_select             2.3.0                       mkl  
absl-py                   0.15.0             pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.1            py37h540881e_1    conda-forge
aiohttp-cors              0.7.0                    pypi_0    pypi
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
ampl-mp                   3.1.0             h2cc385e_1006    conda-forge
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
astroid                   2.11.6           py37h89c1867_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
asynctest                 0.13.0                     py_0    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
blessed                   1.19.1                   pypi_0    pypi
blinker                   1.4                        py_1    conda-forge
brotlipy                  0.7.0           py37h540881e_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.6.15            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.2.0              pyhd8ed1ab_0    conda-forge
certifi                   2022.6.15        py37h89c1867_0    conda-forge
cffi                      1.15.1           py37h43b0acd_0    conda-forge
cftime                    1.6.0            py37h6c7ee08_0    conda-forge
charset-normalizer        2.1.0              pyhd8ed1ab_0    conda-forge
click                     8.0.4                    pypi_0    pypi
cloudpickle               1.6.0                      py_0    conda-forge
coin-or-cbc               2.10.8               h3786ebc_0    conda-forge
coin-or-cgl               0.60.6               h6f57e76_1    conda-forge
coin-or-clp               1.17.7               hc56784d_1    conda-forge
coin-or-osi               0.108.7              h2720bb7_1    conda-forge
coin-or-utils             2.11.6               h202d8b1_1    conda-forge
coincbc                   2.10.8            0_metapackage    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
colorful                  0.5.4                    pypi_0    pypi
coverage                  6.4.4            py37h540881e_0    conda-forge
cryptography              37.0.4           py37h38fbfac_0    conda-forge
curl                      7.83.1               h7bff187_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
dill                      0.3.5.1            pyhd8ed1ab_0    conda-forge
distlib                   0.3.5                    pypi_0    pypi
elementpath               3.0.2                    pypi_0    pypi
expat                     2.4.8                h27087fc_0    conda-forge
filelock                  3.8.0                    pypi_0    pypi
fontconfig                2.14.0               h8e229c2_0    conda-forge
freetype                  2.12.1               hca18f0e_0    conda-forge
frozenlist                1.3.1            py37h540881e_0    conda-forge
gast                      0.2.2                      py_0    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
glib                      2.72.1               h6239696_0    conda-forge
glib-tools                2.72.1               h6239696_0    conda-forge
glpk                      5.0                  h445213a_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
google-api-core           2.8.2                    pypi_0    pypi
google-auth               2.10.0             pyh6c4a22f_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
googleapis-common-protos  1.56.4                   pypi_0    pypi
gpustat                   1.0.0rc1                 pypi_0    pypi
grpc-cpp                  1.48.0               hbad87ad_3    conda-forge
grpcio                    1.43.0                   pypi_0    pypi
gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
gstreamer                 1.18.5               h9f60fe5_3    conda-forge
h5py                      3.6.0           nompi_py37hd308b1e_100    conda-forge
hdf4                      4.2.15               h9772cbc_4    conda-forge
hdf5                      1.12.1          nompi_h2386368_104    conda-forge
icu                       67.1                 he1b5a44_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
imageio                   2.9.0                      py_0    conda-forge
importlib-metadata        4.11.4           py37h89c1867_0    conda-forge
importlib-resources       5.9.0                    pypi_0    pypi
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
ipopt                     3.14.9               hc8a599a_0    conda-forge
isort                     5.10.1             pyhd8ed1ab_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
jsonschema                4.10.0                   pypi_0    pypi
keras-applications        1.0.8                      py_1    conda-forge
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py37h7cecad7_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lazy-object-proxy         1.7.1            py37h540881e_1    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20220623.0      cxx17_h48a1fff_1    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libclang                  11.1.0          default_ha53f305_1    conda-forge
libcurl                   7.83.1               h7bff187_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libglib                   2.72.1               h2d90d5f_0    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
liblapacke                3.9.0           16_linux64_openblas    conda-forge
libllvm11                 11.1.0               hf817b99_3    conda-forge
libnetcdf                 4.8.1           nompi_h329d8a1_102    conda-forge
libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_1    conda-forge
libpng                    1.6.37               h753d276_4    conda-forge
libpq                     12.9                 h16c4e8d_3  
libprotobuf               3.20.1               h6239696_1    conda-forge
libsqlite                 3.39.2               h753d276_1    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libtiff                   4.4.0                h0e0dad5_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.10               h68273f3_2    conda-forge
libxslt                   1.1.33               hf705e74_1    conda-forge
libzip                    1.9.2                hc869a4a_1    conda-forge
libzlib                   1.2.12               h166bdaf_2    conda-forge
lxml                      4.8.0            py37h540881e_3    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
matplotlib                3.2.2                         1    conda-forge
matplotlib-base           3.2.2            py37h1d35a4c_1    conda-forge
mccabe                    0.7.0              pyhd8ed1ab_0    conda-forge
metis                     5.1.0             h58526e2_1006    conda-forge
msgpack                   1.0.4                    pypi_0    pypi
multidict                 6.0.2            py37h540881e_1    conda-forge
mumps-include             5.2.1               ha770c72_11    conda-forge
mumps-seq                 5.2.1               h2104b81_11    conda-forge
mysql-common              8.0.30               haf5c9bc_0    conda-forge
mysql-libs                8.0.30               h28c427c_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
netcdf4                   1.5.8           nompi_py37hf784469_101    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
nose                      1.3.7                   py_1006    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.78                 h2350873_0    conda-forge
numexpr                   2.8.0           py37hfe5f03c_101    conda-forge
numpy                     1.18.5           py37h8960a57_0    conda-forge
numpy-financial           1.0.0              pyhd8ed1ab_0    conda-forge
nvidia-ml-py              11.495.46                pypi_0    pypi
oauthlib                  3.2.0              pyhd8ed1ab_0    conda-forge
opencensus                0.11.0                   pypi_0    pypi
opencensus-context        0.1.3                    pypi_0    pypi
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1q               h166bdaf_0    conda-forge
opt_einsum                3.3.0              pyhd8ed1ab_1    conda-forge
pandas                    1.1.5            py37hdc94413_0    conda-forge
patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pillow                    9.2.0            py37h850a105_2    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              2.5.2              pyhd8ed1ab_1    conda-forge
ply                       3.11                       py_1    conda-forge
prometheus-client         0.13.1                   pypi_0    pypi
protobuf                  3.20.1           py37hd23a5d3_0    conda-forge
psutil                    5.9.1            py37h540881e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
py-spy                    0.3.12                   pypi_0    pypi
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyjwt                     2.4.0              pyhd8ed1ab_0    conda-forge
pylint                    2.14.5             pyhd8ed1ab_0    conda-forge
pyomo                     6.4.1            py37hd23a5d3_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.1                   pypi_0    pypi
pyside2                   5.13.2           py37hfa98aef_4    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
python                    3.7.12          hb7a2778_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.2.1           pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyutilib                  6.0.0              pyh9f0ad1d_0    conda-forge
pyyaml                    6.0                      pypi_0    pypi
qt                        5.12.9               h763d07f_1    conda-forge
ray                       1.13.0                   pypi_0    pypi
re2                       2022.06.01           h27087fc_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
scikit-learn              0.24.2           py37hf0f1638_1    conda-forge
scipy                     1.5.3            py37h14a347d_0    conda-forge
scotch                    6.0.9                hb2e6521_2    conda-forge
setuptools                59.8.0           py37h89c1867_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smart-open                6.0.0                    pypi_0    pypi
sqlite                    3.39.2               h4ff8645_1    conda-forge
statsmodels               0.12.2           py37hb1e94ed_0    conda-forge
swig                      4.0.2                hd3c618e_2    conda-forge
tensorboard               2.8.0              pyhd8ed1ab_1    conda-forge
tensorboard-data-server   0.6.0            py37h38fbfac_2    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
tensorflow                2.0.0           mkl_py37h66b46cc_0  
tensorflow-base           2.0.0           mkl_py37h9204916_0  
tensorflow-estimator      2.5.0              pyh8a188c0_0    conda-forge
termcolor                 1.1.0              pyhd8ed1ab_3    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.11.4             pyha770c72_0    conda-forge
tornado                   6.2              py37h540881e_0    conda-forge
typed-ast                 1.5.4            py37h540881e_0    conda-forge
typing                    3.10.0.0           pyhd8ed1ab_0    conda-forge
typing-extensions         4.3.0                hd8ed1ab_0    conda-forge
typing_extensions         4.3.0              pyha770c72_0    conda-forge
unixodbc                  2.3.10               h583eb01_0    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
virtualenv                20.16.3                  pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
werkzeug                  0.16.1                     py_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1           py37h540881e_0    conda-forge
xarray                    0.16.2             pyhd8ed1ab_0    conda-forge
xmlschema                 2.0.2                    pypi_0    pypi
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.7.2            py37h540881e_2    conda-forge
zipp                      3.8.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.12               h166bdaf_2    conda-forge
zstd                      1.5.2                h8a70e8d_4    conda-forge

Reproduction script

Partial reproduction script.
Ray is started on head:

ray start --head

and the code uses ssh to switch to the nodes and starts them:

ray start --verbose --address=10.159.13.9:59086 --num-cpus 11 --min-worker-port 10002 --max-worker-port 10090

ray start --verbose --address=10.159.13.9:59086 --num-cpus 11 --min-worker-port 10002 --max-worker-port 10090

ray start --verbose --address=10.159.13.9:59086 --num-cpus 11 --min-worker-port 10002 --max-worker-port 10090

ray start --verbose --address=10.159.13.9:59086 --num-cpus 11 --min-worker-port 10002 --max-worker-port 10090

ray start --verbose --address=10.159.13.9:59086 --num-cpus 11 --min-worker-port 10002 --max-worker-port 10090

ray start --verbose --address=10.159.13.9:59086 --num-cpus 6 --min-worker-port 10002 --max-worker-port 10050

and then after ray has been running, I try and run ray timeline:

ray timeline --address='10.159.13.9:59086'

Issue Severity

No response
@joshua-cogliati-inl joshua-cogliati-inl added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Aug 17, 2022
@joshua-cogliati-inl joshua-cogliati-inl changed the title [<Ray component: Core] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB [<Ray component: Core>] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB Aug 17, 2022
@jjyao jjyao changed the title [<Ray component: Core>] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB [Core] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB Aug 18, 2022
@jjyao
Copy link
Collaborator

jjyao commented Aug 18, 2022

cc @rkooo567

@jjyao jjyao added the core Issues that should be addressed in Ray Core label Aug 18, 2022
@rkooo567
Copy link
Contributor

rkooo567 commented Oct 4, 2022

I think it is a known issue. ray timeline doesn't work well at large scale right now. We are planning to fix this soon (optimistically by Ray 2.2, or 2.3)

@rkooo567 rkooo567 added P1 Issue that should be fixed within a few weeks and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Oct 4, 2022
@rkooo567 rkooo567 added observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling triage Needs triage (eg: priority, bug/not-bug, and owning component) and removed core Issues that should be addressed in Ray Core P1 Issue that should be fixed within a few weeks labels Oct 24, 2022
@rkooo567 rkooo567 removed their assignment Oct 24, 2022
@richardliaw richardliaw added the core Issues that should be addressed in Ray Core label Oct 29, 2022
@rkooo567 rkooo567 added the dashboard Issues specific to the Ray Dashboard label Oct 30, 2022
@rkooo567
Copy link
Contributor

Actually what's the ray version? It may have been fixed in the master. We have upper limit of the amount of profile data stored (so the return payload may be way lower than 2GB all the time. Can you check it? )

@hora-anyscale hora-anyscale removed the core Issues that should be addressed in Ray Core label Dec 14, 2022
@hora-anyscale hora-anyscale changed the title [Core] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB [Observability] ray timeline errors with ray.rpc.GetAllProfileInfoReply exceeded maximum protobuf size of 2GB Dec 14, 2022
@hora-anyscale hora-anyscale added core Issues that should be addressed in Ray Core and removed core Issues that should be addressed in Ray Core labels Dec 16, 2022
@alanwguo alanwguo added P2 Important issue, but not time-critical and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something that is supposed to be working; but isn't dashboard Issues specific to the Ray Dashboard observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling P2 Important issue, but not time-critical
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@alanwguo @jjyao @richardliaw @joshua-cogliati-inl @rkooo567 @hora-anyscale