Please sign in to comment.
[SPARK-25837][CORE] Fix potential slowdown in AppStatusListener when …
…cleaning up stages ## What changes were proposed in this pull request? * Update `AppStatusListener` `cleanupStages` method to remove tasks for those stages in a single pass instead of 1 for each stage. * This fixes an issue where the cleanupStages method would get backed up, causing a backup in the executor in ElementTrackingStore, resulting in stages and jobs not getting cleaned up properly. Tasks seem most susceptible to this as there are a lot of them, however a similar issue could arise in other locations the `KVStore` `view` method is used. A broader fix might involve updates to `KVStoreView` and `InMemoryView` as it appears this interface and implementation can lead to multiple and inefficient traversals of the stored data. ## How was this patch tested? Using existing tests in AppStatusListenerSuite This is my original work and I license the work to the project under the project’s open source license. Closes #22883 from patrickbrownsync/cleanup-stages-fix. Authored-by: Patrick Brown <firstname.lastname@example.org> Signed-off-by: Marcelo Vanzin <email@example.com> (cherry picked from commit e9d3ca0) Signed-off-by: Marcelo Vanzin <firstname.lastname@example.org>
- Loading branch information...
Showing with 9 additions and 10 deletions.