Add alerting tools IT; fix missing system index bug of SearchMonitorsTool

[ohltyler]

Description

This PR adds improvements to the existing SearchAlertsTool and SearchMonitors tools:

• adds basic integ tests (more to be added tracked in followup issue Improve alerting tool IT #136)
• handles edge cases of missing system indices for SearchMonitorsTool
• cleans up processing logic of responses into helper fns for SearchMonitorsTool

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ohltyler]

Note these integ tests will fail until opensearch-project/alerting#1382 is merged and published to sonatype, so the CI can pick up the updated alerting dependency. The CI can be reran when that happens. In the meantime confirmed everything passes locally using a locally-published version with the alerting bug fix.

[ohltyler]

TODOs: add/improve integ tests. Tracked in separate issue and mentioned in source code comments: #136

[codecov]

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (e278d8d) 80.83% compared to head (f5d056c) 80.65%.

Files	Patch %	Lines
...org/opensearch/agent/tools/SearchMonitorsTool.java	78.94%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #135      +/-   ##
============================================
- Coverage     80.83%   80.65%   -0.19%     
- Complexity      195      198       +3     
============================================
  Files            13       13              
  Lines          1002     1013      +11     
  Branches        133      133              
============================================
+ Hits            810      817       +7     
- Misses          141      145       +4     
  Partials         51       51              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lezzago]

We are trying to create the system index for alerting. The best way to do this is to create a dummy monitor, so that the schema mapping of the index is setup correctly and would be better to test for an integration test.

[ohltyler]

Agreed - this was an interim test fn that is actually currently not used. I'll remove and leave the TODO, as this will be grouped with the added tests in the followup issue #136. Thanks for the suggestion, I will likely create dummy monitors and alerts for testing the populated results of the search monitors and search alerts tools respectively.