Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.9.0: OSX Python Pickle Issue #3646

Closed
ax3l opened this issue Jun 1, 2023 · 8 comments
Closed

2.9.0: OSX Python Pickle Issue #3646

ax3l opened this issue Jun 1, 2023 · 8 comments
Assignees
@dmitry-ganyushin

Comments

@ax3l
Copy link
Contributor

ax3l commented Jun 1, 2023 •
edited
Loading

Describe the bug
This is a Python regression in 2.9.0, first seen by @liangwang0734 in conda-forge/adios2-feedstock#71

This seems to be partly a regression from a0d4586 possibly due to the change in multiprocessing on OS X since Python 3.8. Searching the error TypeError: cannot pickle '_io.TextIOWrapper' object gives a bunch of discussions.

cc @vicentebolea - thank you for offering to help with this

To Reproduce
Build and test ADIOS2 as in conda-forge/adios2-feedstock#71

Desktop (please complete the following information):

  • OS/Platform: OSX x86_64 for CPython 3.8-3.11 (all)
  • Build: LLVM Clang 14, CMake (latest), shared build

Additional context
This bug prevents us from releasing a new ADIOS2 release on conda-forge.
@ax3l
Copy link
Contributor Author

ax3l commented Jun 5, 2023 •
edited
Loading

ping @dmitry-ganyushin

Issue is in https://github.com/ornladios/ADIOS2/blob/master/testing/adios2/engine/dataman/TestDataMan1D.py with macOS since the 2.9.0 release.

@ax3l
Copy link
Contributor Author

ax3l commented Jun 5, 2023 

Traceback (most recent call last):
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/work/testing/adios2/engine/dataman/TestDataMan1D.py", line 34, in test_run
    s.start()
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/runner/miniforge3/conda-bld/adios2-split_1685656764632/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object

@ax3l
Copy link
Contributor Author

ax3l commented Jun 5, 2023 •
edited
Loading

@dmitry-ganyushin we suspect this is from a0d4586, where the test was added.

@dmitry-ganyushin
Copy link
Contributor

dmitry-ganyushin commented Jun 6, 2023 •
edited
Loading

I am looking into that. Is it only specific to OSX? I made a fast test with python 3.9.16 ubuntu, everything is fine.

@dmitry-ganyushin
Copy link
Contributor

Python 3.11.3 + Ubuntu is fine

@pnorbert
Copy link
Contributor

pnorbert commented Jun 6, 2023

@vicentebolea Can we please have ZeroMQ, DataMan, HDF5 and some compression library on for the macos tests?

@vicentebolea vicentebolea mentioned this issue Jun 14, 2023
Open
dmitry-ganyushin added a commit to dmitry-ganyushin/ADIOS2 that referenced this issue Jun 14, 2023
@dmitry-ganyushin
Fix for the issue ornladios#3646. It looks like the default for macos…
71c1e7f 
… and python 3.8+ "spawn" method does not work on macos, swith to "fork" for all situations. https://bugs.python.org/issue33725
@anagainaru anagainaru mentioned this issue Jun 20, 2023
Closed
dmitry-ganyushin added a commit to dmitry-ganyushin/ADIOS2 that referenced this issue Jul 4, 2023
@dmitry-ganyushin
Fix for the issue ornladios#3646. It looks like the default for macos…
9318869 
… and python 3.8+ "spawn" method does not work on macos, swith to "fork" for all situations. https://bugs.python.org/issue33725. Rebased on the current master.
eisenhauer added a commit that referenced this issue Jul 4, 2023
@eisenhauer
Merge pull request #3687 from dmitry-ganyushin/i3646-mult-macos
c8c5238 
Fix for the issue #3646.
@dmitry-ganyushin
Copy link
Contributor

Fixed

@ax3l
Copy link
Contributor Author

ax3l commented Jul 26, 2023 •
edited
Loading

Thanks a lot, @dmitry-ganyushin!

X-ref: #3687

vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this issue Aug 2, 2023
@eisenhauer @vicentebolea
Merge pull request ornladios#3687 from dmitry-ganyushin/i3646-mult-macos
0f2bdf5 
Fix for the issue ornladios#3646.

(cherry picked from commit c8c5238)
@vicentebolea vicentebolea mentioned this issue Aug 3, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmitry-ganyushin dmitry-ganyushin

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ax3l @pnorbert @dmitry-ganyushin