Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.11.5 fails with stix-shifter interface #401

Closed
subbyte opened this issue Sep 12, 2023 · 4 comments
Closed

Python 3.11.5 fails with stix-shifter interface #401

subbyte opened this issue Sep 12, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@subbyte
Copy link
Member

subbyte commented Sep 12, 2023

Describe the bug
Unit test hangs for anything that uses stix-shifter, e.g., test_cli.py or test_command_get.py::test_get_single_stixshifter_stix_bundle.

Details of the bug

Process TransmitterPool-3:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Users/subx/venv/py311/lib/python3.11/site-packages/kestrel_datasource_stixshifter/worker/transmitter.py", line 53, in run
    transmitter.start()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/synchronize.py", line 106, in __getstate__
    if self.is_fork_ctx:
       ^^^^^^^^^^^^^^^^
AttributeError: 'Lock' object has no attribute 'is_fork_ctx'

To Reproduce
pytest

Expected behavior
No error.

Environment (please complete the following information):

  • Kestrel 1.7.5
  • macOS
  • Python 3.11.5 (latest as of 2023/09/12)

Solution
This looks like a bug in Python multiprocessing library: python/cpython#108520 and it is already solved and await to be released in the next Python version.

@subbyte subbyte added the bug Something isn't working label Sep 12, 2023
@subbyte subbyte changed the title Unit test failed on macOS + Python 3.11 Python 3.11.5 fail with stix-shifter interface Sep 12, 2023
@subbyte subbyte changed the title Python 3.11.5 fail with stix-shifter interface Python 3.11.5 fails with stix-shifter interface Sep 12, 2023
@subbyte
Copy link
Member Author

subbyte commented Sep 12, 2023

Confirmed that Python 3.11.4 works on macOS

@subbyte
Copy link
Member Author

subbyte commented Oct 5, 2023

Confirmed resolved with Python 3.11.6 (released Oct 2, 2023).

@subbyte subbyte closed this as completed Oct 5, 2023
@subbyte
Copy link
Member Author

subbyte commented Oct 10, 2023

Specify 3.11.6 for unit test (github workflow) of test-kestrel-datasource-stixshifter. Please remove the .6 suffix when it is the default env for Python 3.11 in github action (probably when 3.11.7 is released).

@subbyte subbyte reopened this Oct 10, 2023
@subbyte
Copy link
Member Author

subbyte commented Apr 19, 2024

Done in #502

@subbyte subbyte closed this as completed Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant