Fix hdfs integration test

[Abacn]

Fix #30459

For some reason there are connection refused error between test_node, name_node and data_node. Might due to recent upgrade of Python official docker image (base image upgrade)

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[Abacn]

The error is

test_1      | .pkg: _optional_hooks> python /usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta
test_1      | .pkg: get_requires_for_build_sdist> python /usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta
test_1      | hdfs_integration_test: packaging backend failed (code=1), with RuntimeError: External transform wrappers have not been generated and the standard external transforms config could not be found
test_1      | <string>:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
test_1      | Traceback (most recent call last):
test_1      |   File "/usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py", line 90, in run
test_1      |     outcome = backend_proxy(parsed_message["cmd"], **parsed_message["kwargs"])
test_1      |   File "/usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py", line 32, in __call__
test_1      |     return getattr(on_object, name)(*args, **kwargs)
test_1      |   File "/app/sdks/python/target/.tox/.pkg/lib/python3.8/site-packages/setuptools/build_meta.py", line 328, in get_requires_for_build_sdist
test_1      |     return self._get_build_requires(config_settings, requirements=[])
test_1      |   File "/app/sdks/python/target/.tox/.pkg/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
test_1      |     self.run_setup()
test_1      |   File "/app/sdks/python/target/.tox/.pkg/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
test_1      |     exec(code, locals())
test_1      |   File "<string>", line 300, in <module>
test_1      |   File "<string>", line 254, in generate_external_transform_wrappers
test_1      | RuntimeError: External transform wrappers have not been generated and the standard external transforms config could not be found
test_1      | Backend: run command get_requires_for_build_sdist with args {'config_settings': None}
test_1      | b''
test_1      | Backend: Wrote response {'code': 1, 'exc_type': 'RuntimeError', 'exc_msg': 'External transform wrappers have not been generated and the standard external transforms config could not be found'} to /tmp/pep517_get_requires_for_build_sdist-_c1nbqtg.json
test_1      | .pkg: _exit> python /usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta
test_1      |   hdfs_integration_test: FAIL code 1 (11.52 seconds)
test_1      |   evaluation failed :( (11.76 seconds)
hdfs_it-non-jenkins_test_1 exited with code 1
S

See https://github.com/apache/beam/actions/runs/8100844372/job/22139914281

[Abacn]

Caused by #29834

[Abacn]

Another breakage after revert #29834:

test_1      | Traceback (most recent call last):
test_1      |   File "/usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py", line 90, in run
test_1      |     outcome = backend_proxy(parsed_message["cmd"], **parsed_message["kwargs"])
test_1      |   File "/usr/local/lib/python3.8/site-packages/pyproject_api/_backend.py", line 32, in __call__
test_1      |     return getattr(on_object, name)(*args, **kwargs)
test_1      |   File "/app/sdks/python/target/.tox/.pkg/lib/python3.8/site-packages/setuptools/build_meta.py", line 328, in get_requires_for_build_sdist
test_1      |     return self._get_build_requires(config_settings, requirements=[])
test_1      |   File "/app/sdks/python/target/.tox/.pkg/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
test_1      |     self.run_setup()
test_1      |   File "/app/sdks/python/target/.tox/.pkg/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
test_1      |     exec(code, locals())
test_1      |   File "<string>", line 257, in <module>
test_1      |   File "<string>", line 226, in copy_tests_from_docs
test_1      | RuntimeError: Could not locate yaml docs in /app/website/www/site/content/en/documentation/sdks or /app/sdks/python/apache_beam/yaml/docs.
test_1      | Backend: run command get_requires_for_build_sdist with args {'config_settings': None}
test_1      | b''
test_1      | Backend: Wrote response {'code': 1, 'exc_type': 'RuntimeError', 'exc_msg': 'Could not locate yaml docs in /app/website/www/site/content/en/documentation/sdks or /app/sdks/python/apache_beam/yaml/docs.'} to /tmp/pep517_get_requires_for_build_sdist-1vupf1cx.json

This is due to #30198. However, the change was earlier and the test succeeded at that time. What happened is that I commented out other test suites and just run hdfsIntegrationTest then install fails.

[Abacn]

Test passed: https://github.com/apache/beam/actions/runs/8103621498/job/22148603034?pr=30458

[Abacn]

passed: https://github.com/apache/beam/actions/runs/8162871528/job/22314832739?pr=30458

[Abacn]

Changed to run hdfsIntegrationTest only in lowest and highest Python version as the functionality didn't change for long but has a history of broken due to unrelated change. If it is broken at least there are green signal for Python 3.9 / 3.10 so does not undermine other breakage signal

[Abacn]

R: @ahmedabu98

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[Abacn]

both hdfsIntegration test and azureIntegrationTest passed, though there are new failures due to #30417 and reverted in #30535

[Abacn]

R: @riteshghorse

[riteshghorse]

LGTM, sounds good to me. Also, checked the python 3.8 results and it is not failing for hdfs test anymore. Thank you!

(I think the other failures are unrelated, probably sync with master)

[Abacn]

Python PreCommit passed. Python Test need rebase to pass, otherwise good for now