[Data] Extract yield_block_with_stats#64267
Merged
goutamvenkat-anyscale merged 4 commits intoJun 25, 2026
Merged
Conversation
Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
owenowenisme
added
data
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the streaming generator protocol implementation across map and shuffle operators by introducing a common helper function yield_block_with_stats in python/ray/data/_internal/execution/util.py. This helper centralizes the logic of yielding a block, retrieving its serialization timing from the generator stats, and yielding the pickled metadata. Both map_operator.py and shuffle_tasks.py have been updated to use this new helper, reducing code duplication. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…-with-stats Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
goutamvenkat-anyscale
approved these changes
Jun 25, 2026
limarkdcunha
pushed a commit
to limarkdcunha/ray
that referenced
this pull request
Jun 30, 2026
## Description Address ray-project#63598 (comment) Extract the shared logic of yield block + pickled metadata ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
ShockYoungCHN
added a commit
to ShockYoungCHN/ray
that referenced
this pull request
Jul 3, 2026
…cutor.py, context.py Second batch of upstream/master alignment for the V2 shuffle path. These 4 files are conflict-free with our V3 additions and unblock a latent import failure from round 1. Files: - python/ray/data/_internal/execution/util.py Adds ``yield_block_with_stats`` (extracted per PR ray-project#64267). Round 1 synced shuffle_tasks.py, which now imports this helper at module load; without this sync, ``from ray.data._internal.execution.util import yield_block_with_stats`` at shuffle_tasks.py:15 raises ImportError, breaking every V2 shuffle path. This is the round-1-follow-up fix. - python/ray/data/_internal/execution/resource_manager.py Pure perf: rolls per-op ExecutionResources into globals via ``ExecutionResources.combine_sum(...)`` in one pass instead of an n-way for-loop of ``add``. No behavioral change; V3 ops (ShuffleReduceOpV3 etc.) go through the same code path. - python/ray/data/_internal/execution/streaming_executor.py Adds proper ``_issue_detector_manager`` attribute declaration in __init__ and an ``issue_detector_manager`` property. Fixes a latent bug where callbacks/insert_issue_detectors.py:19 dynamically sets this attribute on the executor without the class declaring it; harmless at runtime but invisible to typecheckers and fragile. - python/ray/data/context.py (surgical add, not full overwrite) Adds ``hash_shuffle_reduce_get_timeout_s: float`` field and its ``DEFAULT_HASH_SHUFFLE_REDUCE_GET_TIMEOUT_S`` env-parsed default (1800s), consumed by shuffle_tasks._get_shard_batch. NOT overwriting the whole file preserves our local ``use_hash_shuffle_v3`` flag, which upstream doesn't have. Deliberately deferred (all have V3 conflicts): - operator_fusion.py: upstream drops the generic emitter/absorber passes V3's fusion depends on. - plan_all_to_all_op.py: upstream removes _plan_hash_shuffle_repartition_v3. - physical_operator.py / base_physical_operator.py: upstream drops the absorbs_*_map_transformer base methods V3 overrides. - all_to_all_operator.py: upstream drops AbstractAllToAll.num_outputs. Bytecode compiles clean across all 4 files. Signed-off-by: Yuanzhuo Yang <yuanzhuoyang@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Address #63598 (comment)
Extract the shared logic of yield block + pickled metadata
Related issues
Additional information