Fix leak in activeTasks map in TaskQueue

[jtuglu1]

Currently we check only whether the timestamp on the task comes strictly before the current clock value during a syncFromStorage(). This can cause races between startPendingTasksOnRunner() and syncFromStorage() where the latter spuriously updates the clock values for all tasks in activeTasks, preventing syncFromStorage() from making any deletion progress. This can cause waiting tasks to continuously increase without GC.

Description

• I elected to update removeTaskInternal instead of updateTaskEntry because of this being opaque to the caller. I think the invariant that if the updateOperation is called on an entry ===> its lastUpdated field should also be updated makes sense here.
• It's my understanding that the definition of "waiting" is a task which is yet to be placed in the taskRunner queue. Therefore, I've adjusted getWaitingTaskCount to filter out the completed tasks. This should ensure that the only unknown tasks to the taskRunner are those which have not been placed in its queue (in line with the definition).
• This change also enhances the log to print how many tasks were actually removed from activeTasks for better debugging.

Release note

Fix task leak in activeTasks map in TaskQueue and fix waiting task metric count

---

Key changed/added classes in this PR

• indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[kfaraz]

LGTM 🚀
Thanks for the fix and the tests, @jtuglu-netflix !

[kfaraz]

@jtuglu-netflix , please let me know if this is good to merge or if you intend to include some more change here.

[jtuglu1]

@jtuglu-netflix , please let me know if this is good to merge or if you intend to include some more change here.

Nothing more to this PR

[jtuglu1]

@kfaraz all good here?

[kfaraz]

@kfaraz all good here?

Yes, merged.