Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing.Manager() + cd leads to EOFError #890

Closed
sizmailov opened this issue Dec 17, 2019 · 6 comments
Closed

multiprocessing.Manager() + cd leads to EOFError #890

sizmailov opened this issue Dec 17, 2019 · 6 comments
Labels
Milestone

Comments

@sizmailov
Copy link

@sizmailov sizmailov commented Dec 17, 2019

Describe the bug
Change of directory + multiprocessing.Manager() entrance results to EOFError with coverage 5.0

To Reproduce

  1. What version of Python are you using?
    Python 3.8.0rc1
    It also fails on travis with Python 3.6.3

  2. What version of coverage.py are you using?

coverage debug sys
-- sys -------------------------------------------------------
                   version: 5.0
                  coverage: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/site-packages/coverage/__init__.py
                    tracer: -none-
                   CTracer: available
      plugins.file_tracers: -none-
       plugins.configurers: -none-
 plugins.context_switchers: -none-
         configs_attempted: .coveragerc
              configs_read: .coveragerc
               config_file: .coveragerc
           config_contents: '[run]\nconcurrency = multiprocessing'
                 data_file: -none-
                    python: 3.8.0rc1 (default, Oct  1 2019, 21:48:24) [GCC 7.4.0]
                  platform: Linux-4.15.0-70-generic-x86_64-with-glibc2.27
            implementation: CPython
                executable: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/bin/python
              def_encoding: utf-8
               fs_encoding: utf-8
                       pid: 11108
                       cwd: /mnt/md0/Documents/PycharmProjects/remote-runner
                      path: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/bin
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python38.zip
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/lib-dynload
                            /usr/lib/python3.8
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/site-packages
                            /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/lib/python3.8/site-packages/remote_runner-0.2.0-py3.8.egg
               environment: -none-
              command_line: /mnt/md0/Documents/PycharmProjects/remote-runner/venv-3.8/bin/coverage debug sys
           sqlite3_version: 2.6.0
    sqlite3_sqlite_version: 3.22.0
  1. --
  2. What code are you running?
import multiprocessing
import os

if not os.path.exists("./tmp"): os.mkdir("./tmp")
os.chdir("./tmp")

with multiprocessing.Manager() as manager:
    pass

print("ok")
  1. What commands did you run?
python -m coverage run repro.py
Traceback (most recent call last):
  File "repro.py", line 7, in <module>
    with multiprocessing.Manager() as manager:
  File "/usr/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
python repro.py
ok

Expected behavior
ok message

Additional context
Seems like bug was introduced with 5.0 release

@sizmailov sizmailov added the bug label Dec 17, 2019
@nedbat nedbat added this to the 5.0.1 milestone Dec 17, 2019
sizmailov added a commit to sizmailov/remote-runner that referenced this issue Dec 18, 2019
Avoid bug in coverage 5.0
nedbat/coveragepy#890
sizmailov added a commit to sizmailov/remote-runner that referenced this issue Dec 18, 2019
Avoid bug in coverage 5.0
nedbat/coveragepy#890
sizmailov added a commit to sizmailov/remote-runner that referenced this issue Dec 18, 2019
Avoid bug in coverage 5.0
nedbat/coveragepy#890
sizmailov added a commit to sizmailov/remote-runner that referenced this issue Dec 18, 2019
Avoid bug in coverage 5.0
nedbat/coveragepy#890
@nedbat nedbat modified the milestones: 5.0.1, 5.0.2 Dec 22, 2019
@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Dec 23, 2019

Hmm, something is different for me here on Mac OS. Under Python 3.8, I see the same behavior either with or without coverage, and with coverage==4.5.4 or coverage==5.0.1. I get a stack trace like this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/runpy.py", line 262, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/runpy.py", line 95, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/private/tmp/bug890/bug890.py", line 7, in <module>
    with multiprocessing.Manager() as manager:
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/managers.py", line 579, in start
    self._process.start()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "bug890.py", line 7, in <module>
    with multiprocessing.Manager() as manager:
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/pythonz/pythons/CPython-3.8.0rc1/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

If I change the code to this, it works under either:

import multiprocessing
import os

if not os.path.exists("./tmp"): os.mkdir("./tmp")
os.chdir("./tmp")

if __name__ == "__main__":
    with multiprocessing.Manager() as manager:
        pass

print("ok")

I'm not sure what the code is meant to do, and so which should go in the __main__ clause.

@sizmailov

This comment has been minimized.

Copy link
Author

@sizmailov sizmailov commented Dec 23, 2019

Thanks for spending time on it!

I'm sorry to not emphasize .coverargerc settings, it's hidden under coverage debug sys:

[run]
concurrency = multiprocessing

Without concurrency setting it doesn't throw.

The bug is 100% reproducible on all OS and python versions I've tried. I've set up a small repo to demonstrate, please see it's travis build failures.

For some reason windows builds hang forever after first failure command, therefore I cancelled windows jobs after first failure. (I'm not good at windows scripting, please tell how to fix if you know)

Note there are two scripts: bug890.py and bug890_with_main_guard.py. Bug manifests in both.

@sizmailov

This comment has been minimized.

Copy link
Author

@sizmailov sizmailov commented Dec 23, 2019

Here is build of coverage-4.5.4 branch with bug890_with_main_guard.py tests only:

https://travis-ci.com/sizmailov/reproduce-coverage-bug-890/builds/142106771

No failures.

@nedbat nedbat removed the cant-reproduce label Dec 23, 2019
@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Dec 23, 2019

Thanks, the repo helps with the details.

nedbat added a commit that referenced this issue Dec 23, 2019
nedbat added a commit that referenced this issue Dec 23, 2019
nedbat added a commit that referenced this issue Dec 23, 2019
@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Dec 24, 2019

This is now fixed in df82a96. Thanks for the detailed report.

@nedbat nedbat closed this Dec 24, 2019
@nedbat nedbat added the fixed label Dec 27, 2019
@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Jan 5, 2020

This is now available in coverage 5.0.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.