[GCS]Report job error to gcs instead of direct publishing #14617
Conversation
WangTaoTheTonic
requested review from
raulchen,
kfstorm,
wuisawesome and
rkooo567
WangTaoTheTonic
changed the title
WangTaoTheTonic
changed the title
| ERROR_INFO_CHANNEL, job_id.Hex(), data_ptr->SerializeAsString(), callback); | ||
| RAY_LOG(DEBUG) << "Finished publishing job error, job id = " << job_id; | ||
| return status; | ||
| rpc::ReportJobErrorRequest request; |
There was a problem hiding this comment.
I have one question. How does the job error look like? Who was the subscriber, and are subscribers still getting data?
There was a problem hiding this comment.
We barely publish job error data, and only does dashboard subscribe it for showing what happened to a failed job.
Before publishers publish job error msg to redis directly and subscribers receive that. After this change they send this message to gcs first and let gcs publish this message. It is transparent to subscribers' side.
There was a problem hiding this comment.
One usage is that if you start a Python driver in terminal, you can see the error messages in the driver output even though errors are generated in another process or even on another node.
Why are these changes needed?
In raylet, redis client's asynchronous context is just used for
ReportJobErrorin gcs client. If we change the way of reporting job error to rpc based, one redis connection is short for each raylet. (with overhead brought by one extra hop)Related issue number
part of #14463, first step of redis connection reducing in raylet side
Checks
scripts/format.shto lint the changes in this PR.