Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

possibly a ZMQ concurrency problem #3378

Open
benclifford opened this issue Apr 20, 2024 · 0 comments
Open

possibly a ZMQ concurrency problem #3378

benclifford opened this issue Apr 20, 2024 · 0 comments
Labels
bug

Comments

@benclifford
Copy link
Collaborator

Here's a hung pytest case on my laptop:

It looks like a bunch of ZMQ stuff was happening simultanously which makes me suspicious that this is a non-thread-safe ZMQ access - all of monitoring, ZAP authorization, htex task puller and htex polling are inside ZMQ in the below traces.

benc@parsl-dev-3-12-5361:~/parsl/src/parsl$ pytest parsl/tests/ --config local
============================================================ test session starts ============================================================
platform linux -- Python 3.12.2+, pytest-7.4.4, pluggy-1.4.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /home/benc/parsl/src/parsl/parsl/tests
configfile: pytest.ini
plugins: random-order-1.1.1, typeguard-2.13.3, cov-4.1.0
collected 409 items / 2 skipped                                                                                                             

parsl/tests/test_aalst_patterns.py ssssssssssssssssssssssssss                                                                         [  6%]
parsl/tests/test_callables.py sssssssss                                                                                               [  8%]
parsl/tests/test_curvezmq.py ...........................                                                                              [ 15%]
parsl/tests/test_flux.py ssss......                                                                                                   [ 17%]
parsl/tests/test_summary.py .                                                                                                         [ 17%]
parsl/tests/test_thread_parallelism.py ss                                                                                             [ 18%]
parsl/tests/site_tests/test_provider.py s                                                                                             [ 18%]
parsl/tests/site_tests/test_site.py s                                                                                                 [ 18%]
parsl/tests/sites/test_affinity.py .                                                                                                  [ 19%]
parsl/tests/sites/test_concurrent.py .                                                                                                [ 19%]
parsl/tests/sites/test_dynamic_executor.py .                                                                                          [ 19%]
parsl/tests/sites/test_launchers.py s                                                                                                 [ 19%]
parsl/tests/sites/test_local_adhoc.py ..                                                                                              [ 20%]
parsl/tests/sites/test_worker_info.py .                                                                                               [ 20%]
parsl/tests/test_bash_apps/test_apptimeout.py ss                                                                                      [ 21%]
parsl/tests/test_bash_apps/test_basic.py sss                                                                                          [ 21%]
parsl/tests/test_bash_apps/test_error_codes.py ssss                                                                                   [ 22%]
parsl/tests/test_bash_apps/test_keyword_overlaps.py ss                                                                                [ 23%]
parsl/tests/test_bash_apps/test_kwarg_storage.py s                                                                                    [ 23%]
parsl/tests/test_bash_apps/test_memoize.py ss                                                                                         [ 23%]
parsl/tests/test_bash_apps/test_memoize_ignore_args.py ss                                                                             [ 24%]
parsl/tests/test_bash_apps/test_memoize_ignore_args_regr.py ss                                                                        [ 24%]
parsl/tests/test_bash_apps/test_multiline.py s                                                                                        [ 25%]
parsl/tests/test_bash_apps/test_pipeline.py ss                                                                                        [ 25%]
parsl/tests/test_bash_apps/test_stdout.py ssssssssss                                                                                  [ 28%]
parsl/tests/test_channels/test_large_output.py .                                                                                      [ 28%]
parsl/tests/test_checkpointing/test_periodic.py .                                                                                     [ 28%]
parsl/tests/test_checkpointing/test_python_checkpoint_1.py .                                                                          [ 28%]
parsl/tests/test_checkpointing/test_python_checkpoint_2.py .                                                                          [ 29%]
parsl/tests/test_checkpointing/test_python_checkpoint_3.py .                                                                          [ 29%]
parsl/tests/test_checkpointing/test_regression_232.py ss                                                                              [ 29%]
parsl/tests/test_checkpointing/test_regression_233.py ...                                                                             [ 30%]
parsl/tests/test_checkpointing/test_regression_239.py ss                                                                              [ 31%]
parsl/tests/test_checkpointing/test_task_exit.py .                                                                                    [ 31%]
parsl/tests/test_docs/test_from_slides.py s                                                                                           [ 31%]
parsl/tests/test_docs/test_kwargs.py ss                                                                                               [ 32%]
parsl/tests/test_docs/test_tutorial_1.py ss                                                                                           [ 32%]
parsl/tests/test_docs/test_workflow1.py s                                                                                             [ 32%]
parsl/tests/test_docs/test_workflow2.py s                                                                                             [ 33%]
parsl/tests/test_docs/test_workflow4.py sss                                                                                           [ 33%]
parsl/tests/test_error_handling/test_fail.py s                                                                                        [ 33%]
parsl/tests/test_error_handling/test_python_walltime.py ssss                                                                          [ 34%]
parsl/tests/test_error_handling/test_rand_fail.py .ssss                                                                               [ 36%]
parsl/tests/test_error_handling/test_resource_spec.py s                                                                               [ 36%]
parsl/tests/test_error_handling/test_retries.py ...                                                                                   [ 37%]
parsl/tests/test_error_handling/test_retry_handler.py .                                                                               [ 37%]
parsl/tests/test_error_handling/test_retry_handler_failure.py .                                                                       [ 37%]
parsl/tests/test_error_handling/test_serialization_fail.py .                                                                          [ 37%]
parsl/tests/test_error_handling/test_wrap_with_logs.py ..                                                                             [ 38%]
parsl/tests/test_htex/test_basic.py .                                                                                                 [ 38%]
parsl/tests/test_htex/test_connected_blocks.py .                                                                                      [ 38%]
parsl/tests/test_htex/test_cpu_affinity_explicit.py .                                                                                 [ 39%]
parsl/tests/test_htex/test_disconnected_blocks.py .                                                                                   [ 39%]
parsl/tests/test_htex/test_drain.py .                                                                                                 [ 39%]
parsl/tests/test_htex/test_htex.py .........                                                                                          [ 41%]
parsl/tests/test_htex/test_manager_failure.py ..                                                                                      [ 42%]
parsl/tests/test_htex/test_missing_worker.py .                                                                                        [ 42%]
parsl/tests/test_htex/test_multiple_disconnected_blocks.py .                                                                          [ 42%]
parsl/tests/test_htex/test_worker_failure.py .                                                                                        [ 43%]
parsl/tests/test_htex/test_zmq_binding.py ......ss....                                                                                [ 45%]
parsl/tests/test_monitoring/test_app_names.py ...                                                                                     [ 46%]
parsl/tests/test_monitoring/test_basic.py ...                                                                                         [ 47%]
parsl/tests/test_monitoring/test_db_locks.py .                                                                                        [ 47%]
parsl/tests/test_monitoring/test_fuzz_zmq.py .                                                                                        [ 47%]
parsl/tests/test_monitoring/test_htex_init_blocks_vs_monitoring.py ...                                                                [ 48%]
parsl/tests/test_monitoring/test_incomplete_futures.py Fatal Python error: Aborted

Current thread 0x00007ccffd78a6c0 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 159 in __init__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/context.py", line 362 in socket
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 250 in _create_monitoring_channel
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 275 in _command_server
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ccffcf896c0 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 975 in recv_pyobj
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 235 in task_puller
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ccffb7866c0 (most recent call first):
  File "/usr/local/lib/python3.12/threading.py", line 355 in wait
  File "/usr/local/lib/python3.12/multiprocessing/queues.py", line 251 in _feed
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ccffaf856c0 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 696 in send
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 753 in send_multipart
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/base.py", line 442 in _send_zap_reply
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/base.py", line 319 in handle_zap_message
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/thread.py", line 69 in _run
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88 in _run
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1985 in _run_once
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639 in run_forever
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672 in run_until_complete
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/thread.py", line 47 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007cd18c7d6740 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/poll.py", line 104 in poll
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 385 in start
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 680 in starter
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/local/lib/python3.12/multiprocessing/context.py", line 282 in _Popen
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 121 in start
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/executor.py", line 553 in _start_local_interchange_process
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/executor.py", line 428 in start
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 1156 in add_executors
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 190 in __init__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/typeguard/__init__.py", line 1033 in wrapper
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 1443 in load
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/typeguard/__init__.py", line 1033 in wrapper
  File "/home/benc/parsl/src/parsl/parsl/tests/test_monitoring/test_incomplete_futures.py", line 28 in test_future_representation
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 325 in _main
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/home/benc/parsl/virtualenv-3.12/bin/pytest", line 8 in <module>

Extension modules: _cffi_backend, setproctitle._setproctitle, zmq.backend.cython.context, zmq.backend.cython.message, zmq.backend.cython.socket, zmq.backend.cython._device, zmq.backend.cython._poll, zmq.backend.cython._proxy_steerable, zmq.backend.cython._version, zmq.backend.cython.error, zmq.backend.cython.utils, sqlalchemy.cimmutabledict, greenlet._greenlet, sqlalchemy.cprocessors, sqlalchemy.cresultproxy, ndcctools._cwork_queue, ndcctools._cresource_monitor, charset_normalizer.md, psutil._psutil_linux, psutil._psutil_posix, ndcctools.taskvine._cvine, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing (total: 73)

benc@parsl-dev-3-12-5361:~/parsl/src/parsl$ pytest parsl/tests/ --config local
============================================================ test session starts ============================================================
platform linux -- Python 3.12.2+, pytest-7.4.4, pluggy-1.4.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /home/benc/parsl/src/parsl/parsl/tests
configfile: pytest.ini
plugins: random-order-1.1.1, typeguard-2.13.3, cov-4.1.0
collected 409 items / 2 skipped                                                                                                             

parsl/tests/test_aalst_patterns.py ssssssssssssssssssssssssss                                                                         [  6%]
parsl/tests/test_callables.py sssssssss                                                                                               [  8%]
parsl/tests/test_curvezmq.py ...........................                                                                              [ 15%]
parsl/tests/test_flux.py ssss......                                                                                                   [ 17%]
parsl/tests/test_summary.py .                                                                                                         [ 17%]
parsl/tests/test_thread_parallelism.py ss                                                                                             [ 18%]
parsl/tests/site_tests/test_provider.py s                                                                                             [ 18%]
parsl/tests/site_tests/test_site.py s                                                                                                 [ 18%]
parsl/tests/sites/test_affinity.py .                                                                                                  [ 19%]
parsl/tests/sites/test_concurrent.py .                                                                                                [ 19%]
parsl/tests/sites/test_dynamic_executor.py .                                                                                          [ 19%]
parsl/tests/sites/test_launchers.py s                                                                                                 [ 19%]
parsl/tests/sites/test_local_adhoc.py ..                                                                                              [ 20%]
parsl/tests/sites/test_worker_info.py .                                                                                               [ 20%]
parsl/tests/test_bash_apps/test_apptimeout.py ss                                                                                      [ 21%]
parsl/tests/test_bash_apps/test_basic.py sss                                                                                          [ 21%]
parsl/tests/test_bash_apps/test_error_codes.py ssss                                                                                   [ 22%]
parsl/tests/test_bash_apps/test_keyword_overlaps.py ss                                                                                [ 23%]
parsl/tests/test_bash_apps/test_kwarg_storage.py s                                                                                    [ 23%]
parsl/tests/test_bash_apps/test_memoize.py ss                                                                                         [ 23%]
parsl/tests/test_bash_apps/test_memoize_ignore_args.py ss                                                                             [ 24%]
parsl/tests/test_bash_apps/test_memoize_ignore_args_regr.py ss                                                                        [ 24%]
parsl/tests/test_bash_apps/test_multiline.py s                                                                                        [ 25%]
parsl/tests/test_bash_apps/test_pipeline.py ss                                                                                        [ 25%]
parsl/tests/test_bash_apps/test_stdout.py ssssssssss                                                                                  [ 28%]
parsl/tests/test_channels/test_large_output.py .                                                                                      [ 28%]
parsl/tests/test_checkpointing/test_periodic.py .                                                                                     [ 28%]
parsl/tests/test_checkpointing/test_python_checkpoint_1.py .                                                                          [ 28%]
parsl/tests/test_checkpointing/test_python_checkpoint_2.py .                                                                          [ 29%]
parsl/tests/test_checkpointing/test_python_checkpoint_3.py .                                                                          [ 29%]
parsl/tests/test_checkpointing/test_regression_232.py ss                                                                              [ 29%]
parsl/tests/test_checkpointing/test_regression_233.py ...                                                                             [ 30%]
parsl/tests/test_checkpointing/test_regression_239.py ss                                                                              [ 31%]
parsl/tests/test_checkpointing/test_task_exit.py .                                                                                    [ 31%]
parsl/tests/test_docs/test_from_slides.py s                                                                                           [ 31%]
parsl/tests/test_docs/test_kwargs.py ss                                                                                               [ 32%]
parsl/tests/test_docs/test_tutorial_1.py ss                                                                                           [ 32%]
parsl/tests/test_docs/test_workflow1.py s                                                                                             [ 32%]
parsl/tests/test_docs/test_workflow2.py s                                                                                             [ 33%]
parsl/tests/test_docs/test_workflow4.py sss                                                                                           [ 33%]
parsl/tests/test_error_handling/test_fail.py s                                                                                        [ 33%]
parsl/tests/test_error_handling/test_python_walltime.py ssss                                                                          [ 34%]
parsl/tests/test_error_handling/test_rand_fail.py .ssss                                                                               [ 36%]
parsl/tests/test_error_handling/test_resource_spec.py s                                                                               [ 36%]
parsl/tests/test_error_handling/test_retries.py ...                                                                                   [ 37%]
parsl/tests/test_error_handling/test_retry_handler.py .                                                                               [ 37%]
parsl/tests/test_error_handling/test_retry_handler_failure.py .                                                                       [ 37%]
parsl/tests/test_error_handling/test_serialization_fail.py .                                                                          [ 37%]
parsl/tests/test_error_handling/test_wrap_with_logs.py ..                                                                             [ 38%]
parsl/tests/test_htex/test_basic.py .                                                                                                 [ 38%]
parsl/tests/test_htex/test_connected_blocks.py .                                                                                      [ 38%]
parsl/tests/test_htex/test_cpu_affinity_explicit.py .                                                                                 [ 39%]
parsl/tests/test_htex/test_disconnected_blocks.py .                                                                                   [ 39%]
parsl/tests/test_htex/test_drain.py .                                                                                                 [ 39%]
parsl/tests/test_htex/test_htex.py .........                                                                                          [ 41%]
parsl/tests/test_htex/test_manager_failure.py ..                                                                                      [ 42%]
parsl/tests/test_htex/test_missing_worker.py .                                                                                        [ 42%]
parsl/tests/test_htex/test_multiple_disconnected_blocks.py .                                                                          [ 42%]
parsl/tests/test_htex/test_worker_failure.py .                                                                                        [ 43%]
parsl/tests/test_htex/test_zmq_binding.py ......ss....                                                                                [ 45%]
parsl/tests/test_monitoring/test_app_names.py ...                                                                                     [ 46%]
parsl/tests/test_monitoring/test_basic.py ...                                                                                         [ 47%]
parsl/tests/test_monitoring/test_db_locks.py .                                                                                        [ 47%]
parsl/tests/test_monitoring/test_fuzz_zmq.py .                                                                                        [ 47%]
parsl/tests/test_monitoring/test_htex_init_blocks_vs_monitoring.py ...                                                                [ 48%]
parsl/tests/test_monitoring/test_incomplete_futures.py Fatal Python error: Aborted

Current thread 0x00007ccffd78a6c0 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 159 in __init__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/context.py", line 362 in socket
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 250 in _create_monitoring_channel
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 275 in _command_server
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ccffcf896c0 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 975 in recv_pyobj
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 235 in task_puller
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ccffb7866c0 (most recent call first):
  File "/usr/local/lib/python3.12/threading.py", line 355 in wait
  File "/usr/local/lib/python3.12/multiprocessing/queues.py", line 251 in _feed
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ccffaf856c0 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 696 in send
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/socket.py", line 753 in send_multipart
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/base.py", line 442 in _send_zap_reply
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/base.py", line 319 in handle_zap_message
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/thread.py", line 69 in _run
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88 in _run
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1985 in _run_once
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639 in run_forever
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672 in run_until_complete
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/auth/thread.py", line 47 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007cd18c7d6740 (most recent call first):
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/zmq/sugar/poll.py", line 104 in poll
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 385 in start
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 680 in starter
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27 in wrapped
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/local/lib/python3.12/multiprocessing/context.py", line 282 in _Popen
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 121 in start
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/executor.py", line 553 in _start_local_interchange_process
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/executor.py", line 428 in start
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 1156 in add_executors
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 190 in __init__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/typeguard/__init__.py", line 1033 in wrapper
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 1443 in load
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/typeguard/__init__.py", line 1033 in wrapper
  File "/home/benc/parsl/src/parsl/parsl/tests/test_monitoring/test_incomplete_futures.py", line 28 in test_future_representation
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 325 in _main
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/home/benc/parsl/virtualenv-3.12/lib/python3.12/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/home/benc/parsl/virtualenv-3.12/bin/pytest", line 8 in <module>

Extension modules: _cffi_backend, setproctitle._setproctitle, zmq.backend.cython.context, zmq.backend.cython.message, zmq.backend.cython.socket, zmq.backend.cython._device, zmq.backend.cython._poll, zmq.backend.cython._proxy_steerable, zmq.backend.cython._version, zmq.backend.cython.error, zmq.backend.cython.utils, sqlalchemy.cimmutabledict, greenlet._greenlet, sqlalchemy.cprocessors, sqlalchemy.cresultproxy, ndcctools._cwork_queue, ndcctools._cresource_monitor, charset_normalizer.md, psutil._psutil_linux, psutil._psutil_posix, ndcctools.taskvine._cvine, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing (total: 73)
benclifford added a commit that referenced this issue May 28, 2024
@benclifford
make interchange into a "first-order" process like process worker poo…
1733deb 
…l, not multiprocessing

any downstream packaging will need to be aware of the presence of interchange.py as a new command-line invocable script
and this might break some build instructions which do not configure installed scripts onto the path.

this PR replaces keyword arguments with argparse command line parameters. it does not attempt to make those
command line arguments differently-optional than the constructor of the Interchange class (for example, worker_ports and
worker_port_range are both mandatory, because they are both specified before this PR)

i'm somewhat uncomfortable with this seeming like an ad-hoc serialise/deserialise protocol for what was previously
effecting a dict of typed python objects... but it's what process worker pool does.

see issue #3373 for interchange specific issue

see issue #2343 for parsl general fork vs threads issue

see possibly issue #3378?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@benclifford