[Bug]: Fix OS integration tests for Windows platform

[rishabh6788]

Describe the bug

In the current scenario the OS integration tests have to be run manually for Windows platform as they don't run as expected. The major problem is of failure during clean-up process after the tests have run but the process fails as it is not able to delete the temp files generated during test run.

Sample Logs: WIP

To reproduce

On Windows 2019 machine checkout opensearch-build repo and run ./test.sh integ-test manifests/1.3.8/opensearch-1.3.8-test.yml --component index-management

Expected behavior

The integ test should run without failing to delete the temp file generated and should display the final result of PASS or FAILURE once the test has completed.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[zelinh]

I'm able to reproduce this issue on my Windows instance with git bash.
Here are the sample logs with command (./test.sh integ-test ./manifests/2.8.0/opensearch-dashboards-2.8.0-test.yml -p opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.8.0/7935/windows/x64/zip opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.8.0/6182/windows/x64/zip --test-run-id 1235 --component alertingDashboards)

Traceback (most recent call last):
    status = self.__setup_cluster_and_execute_test_config(config)
  File "C:\Users\Administrator\workspace\opensearch-build\src\test_workflow\integ_test\integ_test_suite_opensearch_dashbo
ards.py", line 104, in __setup_cluster_and_execute_test_config
    return self.execute_integtest_sh(endpoint, port, security, config)
  File "C:\Users\Administrator\scoop\apps\python39\3.9.13\lib\contextlib.py", line 126, in __exit__
    next(self.gen)
  File "C:\Users\Administrator\workspace\opensearch-build\src\test_workflow\test_cluster.py", line 64, in create
    cluster.terminate()
  File "C:\Users\Administrator\workspace\opensearch-build\src\test_workflow\test_cluster.py", line 85, in terminate
    self.termination_result = self.service.terminate()
  File "C:\Users\Administrator\workspace\opensearch-build\src\test_workflow\integ_test\service.py", line 65, in terminate

    self.return_code = self.process_handler.terminate()
  File "C:\Users\Administrator\workspace\opensearch-build\src\system\process.py", line 63, in terminate
    os.remove(self.stdout.name)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users
\\ADMINI~1\\AppData\\Local\\Temp\\2\\tmpix2x74us'

[zelinh]

According to my research, the NamedTemporaryFile is being accessed by python.exe.
In our implementation, we are creating self.stdout in the process.py class

opensearch-build/src/system/process.py

Line 29 in 2ade8b3

self.stdout = tempfile.NamedTemporaryFile(mode="r+", delete=False)

and we are opening the file again with open() to read the data.

opensearch-build/src/system/process.py

Lines 60 to 64 in 2ade8b3

	if self.stdout:
	self.__stdout_data__ = open(self.stdout.name, 'r').read()
	self.stdout.close()
	os.remove(self.stdout.name)
	self.stdout = None

However, checking the properties of this module NamedTemporaryFile, I'm seeing Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows). https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile
This might be the reason that python is holding these file especially on windows platform.

Let me keeping testing how we could fix this issue.

[bbarani]

Closing this issue as its fixed now.