GH-35056: [Python][CI] Don't install gdb on Windows #35057

kou · 2023-04-11T21:42:15Z

Because it's not available on Windows.

Rationale for this change

The gdb conda package isn't available on Windows:

https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/46748401#L266

C:\projects\arrow>mamba create -n arrow -y -c conda-forge   --file=ci\conda_env_python.txt    --file=ci\conda_env_gandiva_win.txt --file=ci\conda_env_cpp.txt    "ccache"   "cmake"   "ninja"   "nomkl"   "pandas"   "fsspec"   "python=3.10"     || exit /B 
conda-forge/win-64                                          Using cache
conda-forge/noarch                                          Using cache
Looking for: ['boto3', 'cffi', 'cython', 'cloudpickle', 'fsspec', 'gdb', 'hypothesis', "numpy[version='>=1.16.6']", 'pytest', 'pytest-faulthandler', 'pytest-lazy-fixture', "s3fs[version='>=2021.8.0']", 'setuptools', 'setuptools_scm', "clangdev[version='<15']", "llvmdev[version='<15']", 'aws-sdk-cpp==1.10.13', "benchmark[version='>=1.6.0']", "boost-cpp[version='>=1.68.0']", 'brotli', 'bzip2', 'c-ares', 'cmake', 'flatbuffers', 'gflags', 'glog', "gmock[version='>=1.10.0']", "google-cloud-cpp[version='>=1.34.0']", 'grpc-cpp', "gtest[version='>=1.10.0']", 'libprotobuf', 'libutf8proc', 'lz4-c', 'make', 'ninja', 'orc', 'pkg-config', 'python', 'rapidjson', 're2', 'snappy', "thrift-cpp[version='>=0.11.0']", 'xsimd', 'zlib', 'zstd', 'ccache', 'cmake', 'ninja', 'nomkl', 'pandas', 'fsspec', 'python=3.10']
Could not solve for environment specs
The following package could not be installed
└─ gdb   does not exist (perhaps a typo or a missing channel).
C:\projects\arrow>set lastexitcode=1

What changes are included in this PR?

Don't install gdb on Windows.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

Closes: [Python][CI] AppVeyor job is failed because the gdb conda package isn't available on Windows #35056

Because it's not available on Windows.

github-actions · 2023-04-11T21:42:39Z

Closes: [Python][CI] AppVeyor job is failed because the gdb conda package isn't available on Windows #35056

github-actions · 2023-04-11T21:42:41Z

⚠️ GitHub issue #35056 has been automatically assigned in GitHub to PR creator.

westonpace · 2023-04-11T23:59:50Z

Do we want to add gdb to conda_env_unix to keep the number of conda files down? It may be a little imprecise.

assignUser

I agree with @westonpace but I think the solution is easier :)

assignUser · 2023-04-12T00:54:46Z

ci/docker/conda-python.dockerfile

     ci/conda_env_sphinx.txt \
     /arrow/ci/
 RUN mamba install -q -y \
        --file arrow/ci/conda_env_python.txt \
+        --file arrow/ci/conda_env_python_unix.txt \


Maybe just add gdb directly here as it is not used anywhere else? Should we add more packages/use them in multiple location we can always refactor into a new file.

I think that this approach is better. See also: #35024 (comment)

But it's not a strong opinion because I'm not a conda user. I follow opinions from developers who use conda.

kou · 2023-04-12T06:12:17Z

Do we want to add gdb to conda_env_unix to keep the number of conda files down? It may be a little imprecise.

gdb is only needed for conda-python images. conda-cpp images don't need it.

But I don't have strong opinion for this. Should I use ci/conda_env_unix.txt?

jorisvandenbossche · 2023-04-12T07:11:08Z

It's all not too important to bikeshed over, but I agree with Jacob and Weston that I find it not really worth it to create a new file for this (also as a user of those files for my local development, that makes the command to create your conda environment quite a bit longer).
gdb is also useful for C++ developers, so moving it the base conda_env_unix.txt makes sense to me (and that ensures developers get this if they use conda to setup their environment). If we don't want to do that because we would then unnecessarily install it on CI, then I would just add it to the mamba install command in the dockerfile as Jacob mentioned (and in #35024 (comment))

kou · 2023-04-12T07:17:05Z

OK. I've moved gdb to conda_env_unix.txt.

kou · 2023-04-12T07:17:28Z

@github-actions crossbow submit test-conda-python-3.10

github-actions · 2023-04-12T07:20:03Z

Revision: d1fb68a

Submitted crossbow builds: ursacomputing/crossbow @ actions-d86b248eae

Task	Status
test-conda-python-3.10

jorisvandenbossche

Thanks!

jorisvandenbossche · 2023-04-12T08:57:51Z

Appveyor is still failing, but now it's an issue in the tests (not in setting up the env). Seems to be hanging / taking too much time.

ursabot · 2023-04-13T23:13:58Z

Benchmark runs are scheduled for baseline = b8427d3 and contender = 6120345. 6120345 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.77% ⬆️0.26%] ursa-i9-9960x
[Finished ⬇️0.28% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 61203456 ec2-t3-xlarge-us-east-2
[Failed] 61203456 test-mac-arm
[Finished] 61203456 ursa-i9-9960x
[Finished] 61203456 ursa-thinkcentre-m75q
[Finished] b8427d39 ec2-t3-xlarge-us-east-2
[Failed] b8427d39 test-mac-arm
[Finished] b8427d39 ursa-i9-9960x
[Finished] b8427d39 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2023-04-13T23:14:37Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

### Rationale for this change Similar to the issue with windows in #35057 gdb is also not available for arm64, causing benchmarks to fail. This moves gdb into the dockerfile that actually needs it to prevent further changes. An alternative would be to use multi-arch-gdb but that is only available in `-c memfault` and adding on an entire new channel will only increase solve times and potentially cause problems. * Closes: #35346 Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Because it's not available on Windows. ### Rationale for this change The gdb conda package isn't available on Windows: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/46748401#L266 ```text C:\projects\arrow>mamba create -n arrow -y -c conda-forge --file=ci\conda_env_python.txt --file=ci\conda_env_gandiva_win.txt --file=ci\conda_env_cpp.txt "ccache" "cmake" "ninja" "nomkl" "pandas" "fsspec" "python=3.10" || exit /B conda-forge/win-64 Using cache conda-forge/noarch Using cache Looking for: ['boto3', 'cffi', 'cython', 'cloudpickle', 'fsspec', 'gdb', 'hypothesis', "numpy[version='>=1.16.6']", 'pytest', 'pytest-faulthandler', 'pytest-lazy-fixture', "s3fs[version='>=2021.8.0']", 'setuptools', 'setuptools_scm', "clangdev[version='<15']", "llvmdev[version='<15']", 'aws-sdk-cpp==1.10.13', "benchmark[version='>=1.6.0']", "boost-cpp[version='>=1.68.0']", 'brotli', 'bzip2', 'c-ares', 'cmake', 'flatbuffers', 'gflags', 'glog', "gmock[version='>=1.10.0']", "google-cloud-cpp[version='>=1.34.0']", 'grpc-cpp', "gtest[version='>=1.10.0']", 'libprotobuf', 'libutf8proc', 'lz4-c', 'make', 'ninja', 'orc', 'pkg-config', 'python', 'rapidjson', 're2', 'snappy', "thrift-cpp[version='>=0.11.0']", 'xsimd', 'zlib', 'zstd', 'ccache', 'cmake', 'ninja', 'nomkl', 'pandas', 'fsspec', 'python=3.10'] Could not solve for environment specs The following package could not be installed └─ gdb does not exist (perhaps a typo or a missing channel). C:\projects\arrow>set lastexitcode=1 ``` ### What changes are included in this PR? Don't install gdb on Windows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#35056 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

…ache#35348) ### Rationale for this change Similar to the issue with windows in apache#35057 gdb is also not available for arm64, causing benchmarks to fail. This moves gdb into the dockerfile that actually needs it to prevent further changes. An alternative would be to use multi-arch-gdb but that is only available in `-c memfault` and adding on an entire new channel will only increase solve times and potentially cause problems. * Closes: apache#35346 Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Because it's not available on Windows. ### Rationale for this change The gdb conda package isn't available on Windows: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/46748401#L266 ```text C:\projects\arrow>mamba create -n arrow -y -c conda-forge --file=ci\conda_env_python.txt --file=ci\conda_env_gandiva_win.txt --file=ci\conda_env_cpp.txt "ccache" "cmake" "ninja" "nomkl" "pandas" "fsspec" "python=3.10" || exit /B conda-forge/win-64 Using cache conda-forge/noarch Using cache Looking for: ['boto3', 'cffi', 'cython', 'cloudpickle', 'fsspec', 'gdb', 'hypothesis', "numpy[version='>=1.16.6']", 'pytest', 'pytest-faulthandler', 'pytest-lazy-fixture', "s3fs[version='>=2021.8.0']", 'setuptools', 'setuptools_scm', "clangdev[version='<15']", "llvmdev[version='<15']", 'aws-sdk-cpp==1.10.13', "benchmark[version='>=1.6.0']", "boost-cpp[version='>=1.68.0']", 'brotli', 'bzip2', 'c-ares', 'cmake', 'flatbuffers', 'gflags', 'glog', "gmock[version='>=1.10.0']", "google-cloud-cpp[version='>=1.34.0']", 'grpc-cpp', "gtest[version='>=1.10.0']", 'libprotobuf', 'libutf8proc', 'lz4-c', 'make', 'ninja', 'orc', 'pkg-config', 'python', 'rapidjson', 're2', 'snappy', "thrift-cpp[version='>=0.11.0']", 'xsimd', 'zlib', 'zstd', 'ccache', 'cmake', 'ninja', 'nomkl', 'pandas', 'fsspec', 'python=3.10'] Could not solve for environment specs The following package could not be installed └─ gdb does not exist (perhaps a typo or a missing channel). C:\projects\arrow>set lastexitcode=1 ``` ### What changes are included in this PR? Don't install gdb on Windows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#35056 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

…ache#35348) ### Rationale for this change Similar to the issue with windows in apache#35057 gdb is also not available for arm64, causing benchmarks to fail. This moves gdb into the dockerfile that actually needs it to prevent further changes. An alternative would be to use multi-arch-gdb but that is only available in `-c memfault` and adding on an entire new channel will only increase solve times and potentially cause problems. * Closes: apache#35346 Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Because it's not available on Windows. ### Rationale for this change The gdb conda package isn't available on Windows: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/46748401#L266 ```text C:\projects\arrow>mamba create -n arrow -y -c conda-forge --file=ci\conda_env_python.txt --file=ci\conda_env_gandiva_win.txt --file=ci\conda_env_cpp.txt "ccache" "cmake" "ninja" "nomkl" "pandas" "fsspec" "python=3.10" || exit /B conda-forge/win-64 Using cache conda-forge/noarch Using cache Looking for: ['boto3', 'cffi', 'cython', 'cloudpickle', 'fsspec', 'gdb', 'hypothesis', "numpy[version='>=1.16.6']", 'pytest', 'pytest-faulthandler', 'pytest-lazy-fixture', "s3fs[version='>=2021.8.0']", 'setuptools', 'setuptools_scm', "clangdev[version='<15']", "llvmdev[version='<15']", 'aws-sdk-cpp==1.10.13', "benchmark[version='>=1.6.0']", "boost-cpp[version='>=1.68.0']", 'brotli', 'bzip2', 'c-ares', 'cmake', 'flatbuffers', 'gflags', 'glog', "gmock[version='>=1.10.0']", "google-cloud-cpp[version='>=1.34.0']", 'grpc-cpp', "gtest[version='>=1.10.0']", 'libprotobuf', 'libutf8proc', 'lz4-c', 'make', 'ninja', 'orc', 'pkg-config', 'python', 'rapidjson', 're2', 'snappy', "thrift-cpp[version='>=0.11.0']", 'xsimd', 'zlib', 'zstd', 'ccache', 'cmake', 'ninja', 'nomkl', 'pandas', 'fsspec', 'python=3.10'] Could not solve for environment specs The following package could not be installed └─ gdb does not exist (perhaps a typo or a missing channel). C:\projects\arrow>set lastexitcode=1 ``` ### What changes are included in this PR? Don't install gdb on Windows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#35056 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

…ache#35348) ### Rationale for this change Similar to the issue with windows in apache#35057 gdb is also not available for arm64, causing benchmarks to fail. This moves gdb into the dockerfile that actually needs it to prevent further changes. An alternative would be to use multi-arch-gdb but that is only available in `-c memfault` and adding on an entire new channel will only increase solve times and potentially cause problems. * Closes: apache#35346 Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

apacheGH-35056: [Python][CI] Don't install gdb on Windows

e39bf3a

Because it's not available on Windows.

kou requested review from assignUser and raulcd as code owners April 11, 2023 21:42

kou mentioned this pull request Apr 11, 2023

GH-35021: [Python][CI] Use conda's gdb in test-conda-python #35024

Merged

github-actions bot added the awaiting committer review Awaiting committer review label Apr 11, 2023

assignUser reviewed Apr 12, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Apr 12, 2023

Use conda_env_unix.txt

d1fb68a

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 12, 2023

jorisvandenbossche approved these changes Apr 12, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Apr 12, 2023

jorisvandenbossche merged commit 6120345 into apache:main Apr 12, 2023
11 of 12 checks passed

kou deleted the python-conda-gdb-windows branch April 12, 2023 09:00

assignUser mentioned this pull request Apr 26, 2023

GH-35346: [CI][Python] Move gdb from env-file to dockerfile #35348

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-35056: [Python][CI] Don't install gdb on Windows #35057

GH-35056: [Python][CI] Don't install gdb on Windows #35057

kou commented Apr 11, 2023 •

edited by github-actions bot

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

westonpace commented Apr 11, 2023

assignUser left a comment

assignUser Apr 12, 2023

kou Apr 12, 2023

kou commented Apr 12, 2023

jorisvandenbossche commented Apr 12, 2023

kou commented Apr 12, 2023

kou commented Apr 12, 2023

github-actions bot commented Apr 12, 2023

jorisvandenbossche left a comment

jorisvandenbossche commented Apr 12, 2023

ursabot commented Apr 13, 2023

ursabot commented Apr 13, 2023

GH-35056: [Python][CI] Don't install gdb on Windows #35057

GH-35056: [Python][CI] Don't install gdb on Windows #35057

Conversation

kou commented Apr 11, 2023 • edited by github-actions bot

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

westonpace commented Apr 11, 2023

assignUser left a comment

Choose a reason for hiding this comment

assignUser Apr 12, 2023

Choose a reason for hiding this comment

kou Apr 12, 2023

Choose a reason for hiding this comment

kou commented Apr 12, 2023

jorisvandenbossche commented Apr 12, 2023

kou commented Apr 12, 2023

kou commented Apr 12, 2023

github-actions bot commented Apr 12, 2023

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jorisvandenbossche commented Apr 12, 2023

ursabot commented Apr 13, 2023

ursabot commented Apr 13, 2023

kou commented Apr 11, 2023 •

edited by github-actions bot