Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TEST: windows ray CI: flaky segmentation fault and "Windows fatal exception: access violation" #4735

Open
mvashishtha opened this issue Jul 29, 2022 · 1 comment
Open

TEST: windows ray CI: flaky segmentation fault and "Windows fatal exception: access violation" #4735

mvashishtha opened this issue Jul 29, 2022 · 1 comment
Labels
CI Flaky Test P3 Very minor bugs, or features we can hopefully add some day.

Comments

@mvashishtha
Copy link
Collaborator

Here's an instance from modin/pandas/test/dataframe/test_join_sort.py: https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true

Stack trace 
============================= test session starts =============================
platform win32 -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=[10](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:11) warmup=False warmup_iterations=100000)
rootdir: D:\a\modin\modin, configfile: setup.cfg
plugins: Faker-13.15.1, benchmark-3.4.1, cov-2.[11](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:12).0, forked-1.4.0, xdist-2.5.0
collected [12](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:13)998 items

Windows fatal exception: access violation

Thread 0x000019b8 (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_channel.py", line 1258 in channel_spin
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 870 in run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 890 in _bootstrap

Thread 0x000011a0 (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_channel.py", line 1258 in channel_spin
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 870 in run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 890 in _bootstrap

Thread 0x000007cc (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_channel.py", line 1258 in channel_spin
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 870 in run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 890 in _bootstrap

Thread 0x00001270 (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 306 in wait
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_common.py", line 106 in _wait_once
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_common.py", line 148 in wait
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_channel.py", line 733 in result
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\gcs_pubsub.py", line 249 in _poll_locked
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\gcs_pubsub.py", line 351 in poll
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\worker.py", line 475 in print_logs
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 870 in run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 890 in _bootstrap

Thread 0x000001e8 (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 306 in wait
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_common.py", line 106 in _wait_once
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_common.py", line 148 in wait
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_channel.py", line 733 in result
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\gcs_pubsub.py", line 249 in _poll_locked
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\gcs_pubsub.py", line 317 in poll
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\worker.py", line [13](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:14)89 in listen_error_messages
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 870 in run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 890 in _bootstrap

Thread 0x000005b4 (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 306 in wait
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_common.py", line 106 in _wait_once
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_common.py", line [14](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:15)8 in wait
  File "C:\Miniconda3\envs\modin\lib\site-packages\grpc\_channel.py", line 733 in result
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\gcs_pubsub.py", line 249 in _poll_locked
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\gcs_pubsub.py", line 385 in poll
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\import_thread.py", line 70 in _run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 870 in run
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\Miniconda3\envs\modin\lib\threading.py", line 890 in _bootstrap

Thread 0x00001b3c (most recent call first):
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\worker.py", line 364 in get_objects
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\worker.py", line 1825 in get
  File "C:\Miniconda3\envs\modin\lib\site-packages\ray\_private\client_mode_hook.py", line 105 in wrapper
  File "D:\a\modin\modin\modin\core\execution\ray\implementations\pandas_on_ray\partitioning\partition_manager.py", line 110 in get_objects_from_partitions
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\core\dataframe\pandas\partitioning\partition_manager.py", line 866 in get_indices
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 429 in _compute_axis_labels
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 2311 in <listcomp>
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 2310 in broadcast_apply_full_axis
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 1[15](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:16) in run_f_on_minimally_updated_metadata
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 1876 in apply_full_axis
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 115 in run_f_on_minimally_updated_metadata
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\core\storage_formats\pandas\query_compiler.py", line 505 in join
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\pandas\dataframe.py", line 1275 in join
  File "D:\a\modin\modin\modin\logging\logger_decorator.py", line 128 in run_and_log
  File "D:\a\modin\modin\modin\pandas\test\dataframe\test_join_sort.py", line 111 in test_join
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\python.py", line 192 in pytest_pyfunc_call
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\python.py", line 1761 in runtest
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line [16](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:17)6 in pytest_runtest_call
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line 259 in <lambda>
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line 338 in from_call
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line 258 in call_runtest_hook
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line 219 in call_and_report
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line 130 in runtestprotocol
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\runner.py", line 111 in pytest_runtest_protocol
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\main.py", line 347 in pytest_runtestloop
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\main.py", line 322 in _main
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\main.py", line 268 in wrap_session
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\main.py", line 315 in pytest_cmdline_main
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\modin\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\config\__init__.py", line 164 in main
  File "C:\Miniconda3\envs\modin\lib\site-packages\_pytest\config\__init__.py", line [18](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:19)7 in console_main
  File "C:\Miniconda3\envs\modin\lib\site-packages\pytest\__main__.py", line 5 in <module>
  File "C:\Miniconda3\envs\modin\lib\runpy.py", line 87 in _run_code
  File "C:\Miniconda3\envs\modin\lib\runpy.py", line [19](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:20)4 in _run_module_as_main
D:\a\_temp\16c1e1a0-adaf-4bff-9b1c-40fb0dbccb[25](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:26).sh: line 1:  10[32](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:33) Segmentation fault      python -m pytest modin/pandas/test/dataframe/test_join_sort.py
modin\pandas\test\dataframe\test_join_sort.py ..
Error: Process completed with exit code 1[39](https://github.com/modin-project/modin/runs/7581926651?check_suite_focus=true#step:6:40).
@mvashishtha mvashishtha added the P1 Important tasks that we should complete soon label Aug 31, 2022
@anmyachev
Copy link
Collaborator

I'm lowering the priority because I haven't encountered this problem for a long time.

@anmyachev anmyachev added P3 Very minor bugs, or features we can hopefully add some day. and removed P1 Important tasks that we should complete soon labels Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CI Flaky Test P3 Very minor bugs, or features we can hopefully add some day.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mvashishtha @anmyachev