[Dev Doc] Document how after test executes to read the output from standard and error log messages associated with a test case

[peternied]

Issue

When running test cases logs are required to troubleshoot if the test cases ran as expected and what the source of a failure might have been.

Expectation

It should be clear how to get this information in DEVELOPER_GUIDE.md

Additional Context

With the recent changes to support java 17 new classes of errors started to appear during our test runs. These failures would be bubbled up into the SecurityFilter and then the output from the test would include a failure like Unexpected exception /_cat/health...{CALLSTACK}.

Unfortunately the exception cause was being sent to log.error on the OpenSearch cluster under test and would not be visible to the test output.

If we could make sure that log.error messages are included in the test output this would greatly increase debuggability. The test added in #1639 could be updated to capture this output and confirm the behavior is working as expected.

[peternied]

[Triage] This seems like a common problem with running integration tests, @peternied can you check to see if there is another way we can do this or open an issue in the OpenSearch core codebase so we can get this support for all teams.

[davidlago]

@peternied did you end up creating an issue in Core that we can link to and close this one?

[peternied]

No change to core was needed, this is possible by inspecting the junit test reports that capture both standard output and standard error from the test and opensearch process under test. We should instead update our dev documentation on how to get this information, I'm changing the description to match.