[2.x] Enable security for bwc tests

[cwperks]

Description

Opening up a PR to describe the issues faced with BWC tests with the security plugin installed and solicit feedback.

I plan to forward port this change to main, but first wanted to show this working for 2.9 -> 2.10 tests (as of the time of this writing).

Thanks to the work that @scrawfor99 did in core to supply security settings to testClusters to be able to run the initial wait for cluster yellow checks with a URL that includes the right protocol (https when security is enabled) along with a username and password to authenticate the request.

I ran into 4 hurdles to get this to run:

1. Initially the cluster didn't form. After a lot of frustration, I ended up finding that by supplying network.bind_host and network.publish_host to both 127.0.0.1 it resolved the issue. These could probably be combined into a single network.host, but I chose to keep them separated.
2. I had issue testing changes to the gradle build-tools after making changes locally. This was the most frustrating hurdle, but ultimately the solution was to change the opensearch.version setting in bwc-test/build.gradle to 2.10.0-SNAPSHOT. This value is specifically used as the version of the gradle build-tools that the BWC tests use. The changes I made locally didn't reflect because I was publishing to maven local from the 2.x branch (currently 2.10) and it was looking for 2.9.0-SNAPSHOT artifacts. After updating the value it found my maven local snapshots. For this artifact you can produce maven local snapshots using ./gradlew :build-tools:publishToMavenLocal from the respective branch in the core repo.
3. After the waitForYellow checks were able to run successfully, the REST Client in the SecurityBackwardsCompatibilityIT was also having problems connecting to the cluster because it didn't recognize the certificates of the server. I ended up using the overly trustworthy route where there is no SSL verification for the REST Client used in this test. I borrowed this implementation from k-NN's ODFERestTestCase which is widely used in the plugin ecosystem. There is an open issue to abstract this class into common-utils. More work can be done here to ensure the rest-high-level-client runs with a truststore with the root certificate.
4. The last hurdle I faced was a WarningFailureException where the REST Client could not deserialize the cluster health response because of a warning that was returned with the response about the request including system indices. According to this comment, this may only be enabled in snapshots. To fix this, I set preserve cluster to true which bypasses the method where the error was thrown.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Enhancement

Issues Resolved

#3056

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #3257 (7fb2ef2) into 2.x (abeedb1) will increase coverage by 0.07%.
Report is 5 commits behind head on 2.x.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                2.x    #3257      +/-   ##
============================================
+ Coverage     62.32%   62.39%   +0.07%     
- Complexity     3293     3297       +4     
============================================
  Files           252      252              
  Lines         19467    19457      -10     
  Branches       3308     3308              
============================================
+ Hits          12133    12141       +8     
+ Misses         5717     5700      -17     
+ Partials       1617     1616       -1     

see 12 files with indirect coverage changes

[peternied]

Nicely done, a bunch of questions.