Task cancellation monitoring service

[sgup432]

Description

• Adding task cancellation monitoring service changes. This detects tasks which are running more than X seconds(defined by a setting) after being cancelled. For now we are only tracking SearchShardTasks.
• Runs a separate monitoring thread with interval of 5 seconds. We keep it off though until unless we see any tasks cancellations.
• Added stats as well.
  This has dependency on changes(Task API) raised as part of this PR - Adding task cancellation timestamp in task API #7445. Once that is pushed, I will rebase.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#6953

Testing

Spun up a opensearch cluster. Introduced a deliberate delay in search request, then cancelled it. Verified that node stats and task API are being populated correctly.

curl -XGET localhost:9200/_nodes/stats?pretty
<rest of stats>....
"task_cancellation" : {
        "search_shard_task" : {
          "current_count_post_cancel" : 1,
          "total_count_post_cancel" : 1
        }
      }

 curl -XGET localhost:9200/_nodes/stats/task_cancellation?pretty
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "runTask",
  "nodes" : {
    "DyBaU6hwTVOPfRfLAF6uGw" : {
      "timestamp" : 1684523706582,
      "name" : "runTask-0",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "testattr" : "test",
        "shard_indexing_pressure_enabled" : "true"
      },
      "task_cancellation" : {
        "search_shard_task" : {
          "current_count_post_cancel" : 1,
          "total_count_post_cancel" : 1
        }
      }
    }
  }
}

Task API related testing here - #7445

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15787/
• CommitID: 79be38e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16665/
• CommitID: 8bd8aa7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16876/
• CommitID: eaf1718
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16942/
• CommitID: dc65e48
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17010/
• CommitID: 51f4369
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/17012/
• CommitID: ee0d61a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7642 (8077a04) into main (c888fce) will increase coverage by 0.23%.
The diff coverage is 84.24%.

@@             Coverage Diff              @@
##               main    #7642      +/-   ##
============================================
+ Coverage     70.84%   71.07%   +0.23%     
- Complexity    56464    56651     +187     
============================================
  Files          4702     4706       +4     
  Lines        266973   267135     +162     
  Branches      39157    39167      +10     
============================================
+ Hits         189131   189862     +731     
+ Misses        61897    61283     -614     
- Partials      15945    15990      +45     

Impacted Files	Coverage Δ
...min/cluster/stats/TransportClusterStatsAction.java	69.56% <ø> (ø)
...rg/opensearch/common/settings/ClusterSettings.java	92.50% <ø> (ø)
...rch/action/admin/cluster/node/stats/NodeStats.java	51.44% <33.33%> (-1.00%)	⬇️
...src/main/java/org/opensearch/node/NodeService.java	74.39% <50.00%> (-0.30%)	⬇️
...search/tasks/SearchShardTaskCancellationStats.java	77.27% <77.27%> (ø)
...rc/main/java/org/opensearch/tasks/TaskManager.java	66.66% <80.95%> (+0.91%)	⬆️
...va/org/opensearch/tasks/TaskCancellationStats.java	82.35% <82.35%> (ø)
...earch/tasks/TaskCancellationMonitoringService.java	90.32% <90.32%> (ø)
...on/admin/cluster/node/stats/NodesStatsRequest.java	93.24% <100.00%> (+0.09%)	⬆️
.../cluster/node/stats/TransportNodesStatsAction.java	100.00% <100.00%> (ø)
... and 2 more

... and 458 files with indirect coverage changes

[Bukhtawar]

Overall LGTM

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

• URL: https://build.ci.opensearch.org/job/gradle-check/17106/
• CommitID: ba766f3
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/17149/
• CommitID: 8077a04
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[Bukhtawar]

Lets ensure we open an issue for all things needing a followup.

[sgup432]

Lets ensure we open an issue for all things needing a followup.

@Bukhtawar Sure. I have created desired issues for the things we need to track or follow up on.