Set error messages in helix for state transition failues

[xiangfu0]

It can help access the errors from the controller UI without searching server logs for segment errors or broker/server instances go into error status.

[kishoreg]

can you please elaborate "set error messages in helix"

[mcvsubbu]

If I understand what @fx19880617 is saying, it is to provide an API in the controller to retrieve Helix state transition error messages and perhaps history as well. I concur that it is a useful feature to have. Once we have the servers and the error messages, we can look into the server for a specific cause.

[xiangfu0]

Yes, right now we only know that segment in error state from external view. The goal is to also set the stacktrace for segment loading and easy for users to retrieve it.

[kishoreg]

Got it. One thing to keep in mind is that state transition error messages get cleaned up automatically by Helix. we should probably do something like this

• each server stores the error message/segment status in segmentDataManager
• exposes an api to get the status along with message
• controller can invoke the server api on demand

Please don't save the error stacktrace in Helix

[mcvsubbu]

I think Helix preserves the error stack/messages for a few days? State stored in segmentDataManager can go away on restart. I think there is value in retrieving helix messages (if they exist) as well.
On a recent issue in Linkedin, it turned out that the state transitions did not succeed and caused a performance hit on pinot servers. The errors went away upon restart. I can totally imagine an admin restarting the servers to get the system up & running.

It may make sense to have an API gather all debug information available and output it as a a blob of text. We can enhance this as we feel like, without regarding to backward compat, etc.

curl controller:port/tables/{tableName}/debugState 1> saved 2>&1

can be done before any restart is attempted. This can also retrieve states from each of the servers that have segments for the table.

[dharakk]

Hi, from experience, this will be a really useful feature to me. I would love to give it a try if nobody is working on it.