New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raw pipe example fails on Windows #68

Closed
jgehrcke opened this Issue Oct 3, 2018 · 6 comments

Comments

Projects
None yet
1 participant
@jgehrcke
Copy link
Owner

jgehrcke commented Oct 3, 2018

181003-15:03:43.936 INFO: Test with raw pipe...
Traceback (most recent call last):
  File "examples/raw_largemsg_bench.py", line 103, in <module>
    main()
  File "examples/raw_largemsg_bench.py", line 66, in main
    spawn_child_transfer(c, p, data, checksum)
  File "examples/raw_largemsg_bench.py", line 75, in spawn_child_transfer
    p = gipc.start_process(target=child, args=(childhandler, checksum))
  File "C:\Python36-x64\lib\site-packages\gipc\gipc.py", line 288, in start_process
    p.start()
  File "C:\Python36-x64\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Python36-x64\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Python36-x64\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Python36-x64\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Python36-x64\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pipe.<locals>.<lambda>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python36-x64\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Python36-x64\lib\multiprocessing\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

Taking tha raw pipe (no encoder) out of the picture for a moment shows that the part of the example code which uses the duplex pipe with the default encoder (pickle) works:

[00:01:38] python examples/raw_largemsg_bench.py
[00:01:38] 181003-15:12:17.532 INFO: Creating data ...
[00:01:41] 181003-15:12:20.456 INFO: Data size: 762.939453125 MBytes
[00:01:43] 181003-15:12:22.409 INFO: Test with default pipe...
[00:01:44] 181003-15:12:23.957 INFO: Sending data
[00:02:04] 181003-15:12:43.318 INFO: Child confirmed that it received data, checksum matches
[00:02:06] 181003-15:12:45.744 INFO: Duration: 19.361 s
[00:02:06] 181003-15:12:45.744 INFO: Rate: 39.41 MBytes/s
@jgehrcke

This comment has been minimized.

Copy link
Owner

jgehrcke commented Oct 3, 2018

Can we reproduce this in a unit test?

@jgehrcke

This comment has been minimized.

Copy link
Owner

jgehrcke commented Oct 3, 2018

This reproduces in a unit test.

@jgehrcke jgehrcke changed the title Raw duplex pipe example fails on Windows Raw pipe example fails on Windows Oct 4, 2018

@jgehrcke

This comment has been minimized.

Copy link
Owner

jgehrcke commented Oct 4, 2018

I think I understand by now that the "duplex" part is not important. It's the "raw" aspect, i.e. the special encoder and decoder callables which are problematic:

They are currently defined in the local scope of the pipe() function and therefore, on Windows, they are pickled to the child process. That does not work because function objects (also lambdas) cannot be pickled by default. One could hack something together with the more low-level marshal package. However, a cleaner approach is to define the raw (noop) encoder and decoder functions on the module level scope so that they are available through normal import (on Windows, in the absence of fork(), the way multiprocessing works is that it launches a fresh interpreter which tries to re-create as much of the parent's state as possible through regular imports, and the rest of the state is pickled).

@jgehrcke

This comment has been minimized.

Copy link
Owner

jgehrcke commented Oct 4, 2018

This raises the question about the applicability of the generic encoder/decoder concept on Windows when pipe handles are passed between processes.

We probably need to specify precisely what works and what does not work and then document some limitation.

When one needs to have custom encoder and decoder functions across processes on windows then a perfectly clean "workaround" (it's not even that, I think) is to use the raw pipe with custom (application-specific) encoder and decoder functions.

@jgehrcke

This comment has been minimized.

Copy link
Owner

jgehrcke commented Oct 4, 2018

With the bug in place this test fails on Windows:

From a209e566507be6c77e5e0d6267ca2845d021b19b Mon Sep 17 00:00:00 2001
From: Jan-Philip Gehrcke <jgehrcke@googlemail.com>
Date: Wed, 3 Oct 2018 17:22:35 +0200
Subject: [PATCH] tests: add test_raw_pipe_across_processes

---
 test/test_gipc.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/test/test_gipc.py b/test/test_gipc.py
index ea1a14b..96398cd 100644
--- a/test/test_gipc.py
+++ b/test/test_gipc.py
@@ -978,6 +978,16 @@ def test_not_callable_decoder(self):
             with pipe(encoder=lambda x: x, decoder=1) as (r, w):
                 pass
 
+    def test_raw_pipe_across_processes(self):
+        data = b'abc'
+
+        with pipe(encoder=None, decoder=None) as (r, w):
+            start_process(child_test_raw_pipe_across_processes, (r, ))
+
+
+def child_test_raw_pipe_across_processes(r):
+    assert r.get() == b'abc'
+
 
 class TestSimpleUseCases(object):
     """Test very basic usage scenarios of gipc (pure gipc+gevent).

Added this test and a fix for the problem in #67.

@jgehrcke

This comment has been minimized.

Copy link
Owner

jgehrcke commented Oct 4, 2018

We probably need to specify precisely what works and what does not work and then document some limitation. ... When one needs to have custom encoder and decoder functions across processes on windows then a perfectly clean "workaround" (it's not even that, I think) is to use the raw pipe with custom (application-specific) encoder and decoder functions.

This is a TODO item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment