[DO-NOT-MERGE] Investigate 'WorkerMemoryTest.test_memory_limit' failure

[HyukjinKwon]

What changes were proposed in this pull request?

WorkerMemoryTest. test_memory_limit fails apparently on Jenkins for an unknown reason. This PR targets to investigate.

Why are the changes needed?

TBD

Does this PR introduce any user-facing change?

TBD

How was this patch tested?

TBD

[SparkQA]

Test build #116598 has finished for PR 27183 at commit 5635459.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Okay, seems it's hitting the OOM due to too small limit apparently:

Current mem limits: 18446744073709551615 of max 18446744073709551615

Setting mem limits to 1048576 of max 1048576

RPython traceback:
  File "pypy_module_pypyjit_interp_jit.c", line 289, in portal_5
  File "pypy_interpreter_pyopcode.c", line 3468, in handle_bytecode__AccessDirect_None
  File "pypy_interpreter_pyopcode.c", line 5558, in dispatch_bytecode__AccessDirect_None
out of memory: couldn't allocate the next arena
ERROR

[SparkQA]

Test build #116599 has finished for PR 27183 at commit 23da553.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HeartSaVioR]

IMHO, we don't necessarily need to set it very smaller if that's the case. It just needs to be different from default value, and we'll never conflict with such huge default value 18,446,744,073,709,551,615.

[HyukjinKwon]

Current mem limits: 18446744073709551615 of max 18446744073709551615

Setting mem limits to 1048576 of max 1048576

RPython traceback:
  File "pypy_module_pypyjit_interp_jit.c", line 289, in portal_5
  File "pypy_interpreter_pyopcode.c", line 3468, in handle_bytecode__AccessDirect_None
  File "pypy_interpreter_pyopcode.c", line 5558, in dispatch_bytecode__AccessDirect_None
out of memory: couldn't allocate the next arena
ERROR

I think we should set it high enough for now.

[SparkQA]

Test build #116602 has finished for PR 27183 at commit c596260.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

We'll drop Python 2 very soon and the current PyPy in Jenkins is compatible with Python 2. So, it won't be a big issue to skip the test for now.

[SparkQA]

Test build #116604 has finished for PR 27183 at commit 25982ec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #116606 has finished for PR 27183 at commit f46e394.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Made a PR at #27186