GH-35021: [Python][CI] Use conda's gdb in test-conda-python

[kou]

Rationale for this change

If we use system gdb, we can't import build-in libraries such as math:

(gdb) source /arrow/cpp/gdb_arrow.py
Traceback (most recent call last):
  File "/arrow/cpp/gdb_arrow.py", line 20, in <module>
    import datetime
  File "/usr/lib/python3.10/datetime.py", line 12, in <module>
    import math as _math
ModuleNotFoundError: No module named 'math'

Because system gdb is built with system Python not conda's Python.

What changes are included in this PR?

Use conda's gdb instead of system gdb.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

• Closes: [Python][CI] gdb tests failing in test-conda-python-3.10 #35021

[kou]

@github-actions crossbow submit test-conda-python-*

[github-actions]

• Closes: [Python][CI] gdb tests failing in test-conda-python-3.10 #35021

[github-actions]

⚠️ GitHub issue #35021 has been automatically assigned in GitHub to PR creator.

[github-actions]

Revision: 5aa58ea

Submitted crossbow builds: ursacomputing/crossbow @ actions-7c722c7903

Task	Status
test-conda-python-3.10
test-conda-python-3.11
test-conda-python-3.7
test-conda-python-3.7-hdfs-2.9.2
test-conda-python-3.7-hdfs-3.2.1
test-conda-python-3.7-pandas-1.0
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8
test-conda-python-3.8-hypothesis
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9
test-conda-python-3.9-dask-latest
test-conda-python-3.9-dask-upstream_devel
test-conda-python-3.9-pandas-upstream_devel
test-conda-python-3.9-spark-master
test-conda-python-3.9-substrait

[jorisvandenbossche]

@kou unfortunately this made appveyor fail, conda fails trying to install gdb on windows (since that package doesn't exist for Windows)

[jorisvandenbossche]

Unfortunately, there doesn't seem to be an easy way to specify OS-specific packages in environment or requirement files with conda .. (see eg conda/conda#8089, conda/conda#11198)

So we probably need to remove gdb again from the file, and move its installation inside the docker setup script?

[kou]

Oh...
Does this work? (I can try this tomorrow. If you can't wait, feel free to try.)

diff --git a/ci/conda_env_python.txt b/ci/conda_env_python.txt
index 254acbc684..04f985c94b 100644
--- a/ci/conda_env_python.txt
+++ b/ci/conda_env_python.txt
@@ -21,7 +21,6 @@ cffi
 cython
 cloudpickle
 fsspec
-gdb
 hypothesis
 numpy>=1.16.6
 pytest
diff --git a/ci/conda_env_python_unix.txt b/ci/conda_env_python_unix.txt
new file mode 100644
index 0000000000..fab6699797
--- /dev/null
+++ b/ci/conda_env_python_unix.txt
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+gdb
diff --git a/ci/docker/conda-python.dockerfile b/ci/docker/conda-python.dockerfile
index 6a6ae73585..86d292bb67 100644
--- a/ci/docker/conda-python.dockerfile
+++ b/ci/docker/conda-python.dockerfile
@@ -22,10 +22,12 @@ FROM ${repo}:${arch}-conda-cpp
 # install python specific packages
 ARG python=3.8
 COPY ci/conda_env_python.txt \
+     ci/conda_env_python_unix.txt \
      ci/conda_env_sphinx.txt \
      /arrow/ci/
 RUN mamba install -q -y \
         --file arrow/ci/conda_env_python.txt \
+        --file arrow/ci/conda_env_python_unix.txt \
         --file arrow/ci/conda_env_sphinx.txt \
         $([ "$python" == "3.7" ] && echo "pickle5") \
         python=${python} \

[jorisvandenbossche]

Yes, that should work. Or, instead of creating a new file, just add "gdb" in conda-python.dockerfile, like

diff --git a/ci/docker/conda-python.dockerfile b/ci/docker/conda-python.dockerfile
index 6a6ae7358..0b3fbdf13 100644
--- a/ci/docker/conda-python.dockerfile
+++ b/ci/docker/conda-python.dockerfile
@@ -29,6 +29,7 @@ RUN mamba install -q -y \
         --file arrow/ci/conda_env_sphinx.txt \
         $([ "$python" == "3.7" ] && echo "pickle5") \
         python=${python} \
+        gdb \
         nomkl && \
     mamba clean --all

[ursabot]

Benchmark runs are scheduled for baseline = c70c3d5 and contender = 053acab. 053acab is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.68% ⬆️0.09%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 053acabd ec2-t3-xlarge-us-east-2
[Failed] 053acabd test-mac-arm
[Finished] 053acabd ursa-i9-9960x
[Finished] 053acabd ursa-thinkcentre-m75q
[Finished] c70c3d51 ec2-t3-xlarge-us-east-2
[Finished] c70c3d51 test-mac-arm
[Finished] c70c3d51 ursa-i9-9960x
[Finished] c70c3d51 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[kou]

Or, instead of creating a new file, just add "gdb" in conda-python.dockerfile

I think that the creating a new file approach is better because PyArrow developers can also use it to setup their environment.
Anyway, we can discuss it on #35057.