Skip to content

[core] fix: normalize the cmdline field in StatsPayload schema#64286

Merged
rueian merged 3 commits into
ray-project:masterfrom
rueian:fix-dashboard-reporter-pydantic
Jun 24, 2026
Merged

[core] fix: normalize the cmdline field in StatsPayload schema#64286
rueian merged 3 commits into
ray-project:masterfrom
rueian:fix-dashboard-reporter-pydantic

Conversation

@rueian

@rueian rueian commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Description

Fix this:

2026-06-19 23:17:11,662	ERROR reporter_agent.py:1983 -- Error publishing node physical stats.
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1971, in _run_loop
    json_payload = await loop.run_in_executor(
  File "/home/ray/anaconda3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1992, in _run_in_executor
    return asyncio.run(
  File "/home/ray/anaconda3/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/ray/anaconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 2039, in _async_compose_stats_payload
    return self._generate_stats_payload(stats)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 2076, in _generate_stats_payload
    parsed_stats = StatsPayload.parse_obj(stats_dict)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/pydantic/main.py", line 1370, in parse_obj
    return cls.model_validate(obj)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/pydantic/main.py", line 716, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for StatsPayload
workers.70.cmdline
  Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.12/v/list_type

The root cause is that psutil.Process.as_dict() returns None for any attribute whose getter raises AccessDenied/ZombieProcess (psutil's default ad_value is None). So a worker's cmdline can come back as None, but the StatsPayload schema (cmdline: List[str]) rejects.

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Pydantic field validator to normalize None values in cmdline fields to empty lists across ProcessInfo and StatsPayload models, and adds a corresponding unit test. The reviewer pointed out that the new test will fail in minimal Ray installations where Pydantic is not installed, and suggested skipping the test in those environments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread python/ray/dashboard/modules/reporter/tests/test_reporter.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Rueian <rueiancsie@gmail.com>
@rueian rueian added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Jun 23, 2026
Signed-off-by: Rueian Huang <rueiancsie@gmail.com>
@rueian rueian marked this pull request as ready for review June 23, 2026 23:14
@rueian rueian requested a review from a team as a code owner June 23, 2026 23:14
@rueian rueian merged commit 2d994b6 into ray-project:master Jun 24, 2026
6 checks passed
limarkdcunha pushed a commit to limarkdcunha/ray that referenced this pull request Jun 30, 2026
…roject#64286)

## Description
Fix this:
```
2026-06-19 23:17:11,662	ERROR reporter_agent.py:1983 -- Error publishing node physical stats.
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1971, in _run_loop
    json_payload = await loop.run_in_executor(
  File "/home/ray/anaconda3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1992, in _run_in_executor
    return asyncio.run(
  File "/home/ray/anaconda3/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/ray/anaconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 2039, in _async_compose_stats_payload
    return self._generate_stats_payload(stats)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 2076, in _generate_stats_payload
    parsed_stats = StatsPayload.parse_obj(stats_dict)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/pydantic/main.py", line 1370, in parse_obj
    return cls.model_validate(obj)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/pydantic/main.py", line 716, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for StatsPayload
workers.70.cmdline
  Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.12/v/list_type

```

The root cause is that `psutil.Process.as_dict()` returns None for any
attribute whose getter raises AccessDenied/ZombieProcess (psutil's
default ad_value is None). So a worker's cmdline can come back as None,
but the StatsPayload schema (cmdline: List[str]) rejects.

---------

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>
Signed-off-by: Rueian <rueiancsie@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants