Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI][C++] arrow-azurefs-test times out on Ubuntu 22.04 test jobs #41462

Closed
raulcd opened this issue Apr 30, 2024 · 2 comments
Closed

[CI][C++] arrow-azurefs-test times out on Ubuntu 22.04 test jobs #41462

raulcd opened this issue Apr 30, 2024 · 2 comments
Assignees
@raulcd
Labels
Component: C++ Component: Continuous Integration Type: bug
Milestone
16.1.0

Comments

@raulcd
Copy link
Member

raulcd commented Apr 30, 2024

Describe the bug, including details regarding any error messages, version, and platform.

For the last days the jobs:

have been failing with timeout

       Start 72: arrow-azurefs-test
    Test #72: arrow-azurefs-test ...........................***Timeout 300.10 sec
Running arrow-azurefs-test, redirecting output into /build/cpp/build/test-logs/arrow-azurefs-test.txt (attempt 1/1)

      Start 72: arrow-azurefs-test
    Test #72: arrow-azurefs-test ...........................***Timeout 300.01 sec
Running arrow-azurefs-test, redirecting output into /build/cpp/build/test-logs/arrow-azurefs-test.txt (attempt 1/1)

The test-ubuntu-22.04-python-3 job also started failing on the same dates and is failing on test_fs on azure test even though the stack trace doesn't say much it seems to be the same issue:

 Fatal Python error: Aborted

Current thread 0x00007f85db68a000 (most recent call first):
  File "/arrow-dev/lib/python3.10/site-packages/pyarrow/tests/test_fs.py", line 312 in azurefs
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 877 in call_fixture_func
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 1122 in pytest_fixture_setup
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 1073 in execute
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 603 in _get_active_fixturedef
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 518 in getfixturevalue
  File "/arrow-dev/lib/python3.10/site-packages/pyarrow/tests/test_fs.py", line 452 in filesystem_config
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 884 in call_fixture_func
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 1122 in pytest_fixture_setup
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 1073 in execute
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 603 in _get_active_fixturedef
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035 in execute
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 603 in _get_active_fixturedef
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 518 in getfixturevalue
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/fixtures.py", line 683 in _fillfixtures
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/python.py", line 1630 in setup
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 512 in setup
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 159 in pytest_runtest_setup
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 129 in runtestprotocol
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/main.py", line 339 in _main
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/arrow-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/arrow-dev/lib/python3.10/site-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/arrow-dev/bin/pytest", line 8 in <module>

There were no changes related to azure when it started failing so it seems is related to infrastructure.
This also fails on the maintenance 16.1.0 branch and is a release blocker.

Component(s)

C++, Continuous Integration
@raulcd raulcd added this to the 16.1.0 milestone Apr 30, 2024
@raulcd
Copy link
Member Author

raulcd commented Apr 30, 2024

I've tried locally:

$ UBUNTU=22.04 docker compose run --rm ubuntu-cpp

and I was not able to reproduce but with:

$ ARCHERY_DEBUG=1 ARCHERY_USE_DOCKER_CLI=1 UBUNTU=22.04 archery docker run ubuntu-cpp

and (the command that archery executes):

$ docker run --rm --shm-size 2147483648 -e ARROW_MIMALLOC=ON -e CCACHE_COMPILERCHECK=content -e CCACHE_COMPRESS=1 -e CCACHE_COMPRESSLEVEL=6 -e CCACHE_DIR=/ccache -e CCACHE_MAXSIZE=1G -e SCCACHE_S3_KEY_PREFIX=sccache -v /home/raulcd/code/arrow:/arrow -v ubuntu-ccache:/ccache apache/arrow-dev:amd64-ubuntu-22.04-cpp /bin/bash -c '  /arrow/ci/scripts/cpp_build.sh /arrow /build &&\n  /arrow/ci/scripts/integration_skyhook.sh /build &&\n  /arrow/ci/scripts/cpp_test.sh /arrow /build'

I was able to reproduce, I am investigating to see if I can further debug.

raulcd added a commit to raulcd/arrow that referenced this issue May 2, 2024
@raulcd
apacheGH-41462: [CI] Temporary pin azurite to v3.29.0
a509be0
@github-actions github-actions bot mentioned this issue May 2, 2024
Merged
@raulcd raulcd mentioned this issue May 2, 2024
Open
raulcd added a commit that referenced this issue May 3, 2024
@raulcd
GH-41462: [CI] Temporary pin azurite to v3.29.0 (#41501)
c0aade5 
### Rationale for this change

install_azurite.sh is failing to install the latest version of Azurite and azure tests were failing.

### What changes are included in this PR?

Temporarily pin azurite to v3.29.0 to unblock 16.1.0 release. A follow up issue is tracked here: #41505

### Are these changes tested?

Yes via archery

### Are there any user-facing changes?

No
* GitHub Issue: #41462

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
@raulcd
Copy link
Member Author

raulcd commented May 3, 2024

Issue resolved by pull request 41501
#41501

@raulcd raulcd closed this as completed May 3, 2024
raulcd added a commit that referenced this issue May 3, 2024
@raulcd
GH-41462: [CI] Temporary pin azurite to v3.29.0 (#41501)
5c43efb 
### Rationale for this change

install_azurite.sh is failing to install the latest version of Azurite and azure tests were failing.

### What changes are included in this PR?

Temporarily pin azurite to v3.29.0 to unblock 16.1.0 release. A follow up issue is tracked here: #41505

### Are these changes tested?

Yes via archery

### Are there any user-facing changes?

No
* GitHub Issue: #41462

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
vibhatha pushed a commit to vibhatha/arrow that referenced this issue May 25, 2024
@raulcd @vibhatha
apacheGH-41462: [CI] Temporary pin azurite to v3.29.0 (apache#41501)
3bc2389 
### Rationale for this change

install_azurite.sh is failing to install the latest version of Azurite and azure tests were failing.

### What changes are included in this PR?

Temporarily pin azurite to v3.29.0 to unblock 16.1.0 release. A follow up issue is tracked here: apache#41505

### Are these changes tested?

Yes via archery

### Are there any user-facing changes?

No
* GitHub Issue: apache#41462

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raulcd raulcd

Labels
Component: C++ Component: Continuous Integration Type: bug
Projects
None yet
Milestone
16.1.0
Development

No branches or pull requests
1 participant
@raulcd