CallbackHandler async re-subscribe watcher can potentially miss events #331
Comments
|
The re-subscribe does not rely on the async logic. It is by default done in the ZkClient. |
|
The root cause of the issue is a race condition in Helix cache refresh logic. Basically, Helix relies on the ZK notification path to set up 2 things:
The race condition causes these 2 checks to have a different result. The assignment was not invalidated. But the cache was updated. As a result, the rebalancer refused to calculate a new mapping even knows the latest IdealState. The impact is limited in the CUSTOM rebalance mode only. Because the fist check is done for custom rebalancer only. |
This change fix issue apache#331. The design is ensuring one read only to avoid locking during the change notification. However, a later update introduced addition read. The result is that two reads may have different results because notification is lock free. This leads the cache to be in an inconsistent state. The impact is that the expected rebalance might not happen.
|
@Jackie-Jiang Could you please help to take a look at the proposed fix? Thanks. |
This change fix issue apache#331. The design is ensuring one read only to avoid locking during the change notification. However, a later update introduced addition read. The result is that two reads may have different results because notification is lock free. This leads the cache to be in an inconsistent state. The impact is that the expected rebalance might not happen.
* Fix the race condition while Helix refresh cluster status cache. This change fix issue #331. The design is ensuring one read only to avoid locking during the change notification. However, a later update introduced addition read. The result is that two reads may have different results because notification is lock free. This leads the cache to be in an inconsistent state. The impact is that the expected rebalance might not happen.
* Fix the race condition while Helix refresh cluster status cache. This change fix issue #331. The design is ensuring one read only to avoid locking during the change notification. However, a later update introduced addition read. The result is that two reads may have different results because notification is lock free. This leads the cache to be in an inconsistent state. The impact is that the expected rebalance might not happen.
|
Close this ticket since the fix has been merged to master. We will have a release soon next week with one more critical fix. |
…che#363) * Fix the race condition while Helix refresh cluster status cache. This change fix issue apache#331. The design is ensuring one read only to avoid locking during the change notification. However, a later update introduced addition read. The result is that two reads may have different results because notification is lock free. This leads the cache to be in an inconsistent state. The impact is that the expected rebalance might not happen.
In CallbackHandler, the re-subscription of the watcher for CALLBACK happens asynchronously. If the re-subscription happens after the change of the path, then we will miss the change until the next change happens.
The text was updated successfully, but these errors were encountered: