[Train] Use user-defined train function name in training thread name and error stack traces#64292
[Train] Use user-defined train function name in training thread name and error stack traces#64292avigyabb wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request imports wraps from functools and applies the @wraps(train_fn) decorator to the train_fn_with_final_checkpoint_flush helper function in worker.py to preserve the metadata of the original training function. There are no review comments, so I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
liulehui
left a comment
There was a problem hiding this comment.
can this change be tested with a unit test?
could you also share a repro script so that:
- repro the original issue with the script
- show that this fix actually fix the issue,
- link the repro script and corresponding logs in the PR description.
Add a unit test asserting that the training thread launched by `RayTrainWorker.run_train_fn` is named after the user's training function (`TrainingThread(<user_fn>)`) rather than the internal `train_fn_with_final_checkpoint_flush` wrapper. This guards the `functools.wraps` fix so that stack traces from user code reference the user's function name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Avigya Basnet <avigyabb@stanford.edu>
64eddda to
b574a94
Compare
Description
When a user's training function raises an exception, the training thread (and the resulting stack trace) is named after Ray's internal wrapper
train_fn_with_final_checkpoint_flushinstead of the user-defined function.The root cause is in
RayTrainWorker.run_train_fn: the user's training function is wrapped by an innertrain_fn_with_final_checkpoint_flush, and this wrapper is what gets handed toThreadRunner.ThreadRunnernames the training thread viaget_callable_name(target), so it reads the wrapper's__name__and producesTrainingThread(train_fn_with_final_checkpoint_flush). We applyfunctools.wraps(train_fn)to the inner wrapper so it inherits the user function's__name__. This is similar to howconstruct_train_funcpreserves the user's name ontrain_fnviafunctools.wraps.Related issues
Fixes TRAIN-744 on Jira
Testing
Added a unit test
test_training_thread_name_matches_user_fninpython/ray/train/v2/tests/test_worker.py. It runsRayTrainWorker.run_train_fnwith a user function namedmy_custom_train_fnand asserts the launched training thread is namedTrainingThread(my_custom_train_fn)(rather thanTrainingThread(train_fn_with_final_checkpoint_flush)). Without thefunctools.wrapsfix this assertion fails, so the test guards the fix.Reproduction
Repro script and before/after logs: https://gist.github.com/avigyabb/626151d1b05526f38b221198f6858c65
The script uses Ray's real
ThreadRunnerandget_callable_nameand rebuilds the exact wrapper fromworker.py, toggling@functools.wrapsto show both states in a single run (no cluster required). It also has an--e2emode that drives the realDataParallelTrainer.Results (
thread_name_before_after.log):TrainingThread(train_fn_with_final_checkpoint_flush)TrainingThread(train_fn_per_worker)Additional information