[SPARK-56513][INFRA] Pin meson<1.11.0 and lxml==4.9.4 in Dockerfile for PyPy to recover CI

[sarutak]

What changes were proposed in this pull request?

This PR pin meson<1.11.0 and lxml==4.9.4 to recover CI.

Why are the changes needed?

Recently, CI fails due to docker image building failure.
https://github.com/apache/spark/actions/runs/24489111484/job/71571576157

The root causes are:

• meson 1.11.0 was recently released and pandas 2.3.3 can not be built with it (BUILD: Fails with meson 1.11.0 pandas-dev/pandas#65252)
• Newer lxml was also released (https://pypi.org/project/lxml/#history). This PR pin to 4.9.4 to consistent with the version specified in .github/workflows/build_and_test.yml.

Without pinning lxml, docker build will fail with following error.

538.3 Collecting lxml
538.3   Downloading lxml-6.0.4.tar.gz (4.2 MB)
538.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 91.5 MB/s  0:00:00
538.7   Installing build dependencies: started
540.6   Installing build dependencies: finished with status 'done'
540.6   Getting requirements to build wheel: started
541.3   Getting requirements to build wheel: finished with status 'error'
541.3   error: subprocess-exited-with-error
541.3
541.3   × Getting requirements to build wheel did not run successfully.
541.3   │ exit code: 1
541.3   ╰─> [3 lines of output]
541.3       Building lxml version 6.0.4.
541.3       Building without Cython.
541.3       Error: Please make sure the libxml2 and libxslt development packages are installed.
541.3       [end of output]
541.3
541.3   note: This error originates from a subprocess, and is likely not a problem with pip.
541.3 ERROR: Failed to build 'lxml' when getting requirements to build wheel
------
Dockerfile:67
--------------------
  66 |     RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
  67 | >>> RUN echo 'meson<1.11.0' > /tmp/constraints.txt && \
  68 | >>>     PIP_CONSTRAINT=/tmp/constraints.txt pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib lxml && \
  69 | >>>     rm -f /tmp/constraints.txt
  70 |
--------------------

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed that docker build works for both dev/infra/Dockerfile and dev/spark-test-image/pypy-310/Dockerfile

Was this patch authored or co-authored using generative AI tooling?

No.

[sarutak]

docker build (apt-get install) on GA seems very slow today...

[sarutak]

Base image build seems passed.
https://github.com/sarutak/spark/actions/runs/24499564157/job/71659528745

[gaogaotiantian]

pypy support is already dropped. Let me check how we should proceed with it.

[gaogaotiantian]

I believe we can just remove all the pypy related part from infra image - they are not used anymore.

[zhengruifeng]

thanks, merged to master

[zhengruifeng]

I believe we can just remove all the pypy related part from infra image - they are not used anymore.

+1, the pypy should at least not affect the master branch

[pan3793]

should this(and other recent similar changes) be backported? I see that at least branch-4.1 and branch-4.0 infra image jobs are broken now.

[sarutak]

@pan3793

should this(and other recent similar changes) be backported? I see that at least branch-4.1 and branch-4.0 infra image jobs are broken now.

It seems that the issue resolved by #55414 is also occurring with branch-4.0 and branch-4.1.
Could you take a look at that, @LuciferYang ?

[LuciferYang]

we can backport SPARK-56540 , I'll handle this later.

[eli-schwartz]

This is unfortunate to use < rather than blocking a single "bad" release. It was fixed in meson==1.11.1

[sarutak]

Thank you for pointing it out. That has been removed.
#55505