Cannot close os.pipe

[saml]

Describe the bug
File descriptor returned by os.pipe() cannot be closed:

_, w = os.pipe()
os.close(w)

OSError: [Errno 9] Bad file descriptor in the fake filesystem: '12'

How To Reproduce
Using pytest:

import os
def test_1(fs):
    read, write = os.pipe()
    os.close(write)

Save above as test_foobar.py and run:

$ pytest -vvvv test_foobar.py 
======================================================================================= test session starts =======================================================================================
platform linux -- Python 3.6.5, pytest-4.2.1, py-1.8.0, pluggy-0.9.0 -- /service/nox/test-3-6/bin/python3.6
cachedir: .pytest_cache
rootdir: /service, inifile:
plugins: requests-mock-1.5.2, cov-2.6.1, pyfakefs-3.5.8
collected 1 item                                                                                                                                                                                  

test_foobar.py::test_1 FAILED                                                                                                                                                               [100%]

============================================================================================ FAILURES =============================================================================================
_____________________________________________________________________________________________ test_1 ______________________________________________________________________________________________

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7fa25d6431d0>

    def test_1(fs):
        read, write = os.pipe()
>       os.close(write)

/service/test_foobar.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/service/nox/test-3-6/lib/python3.6/site-packages/pyfakefs/fake_filesystem.py:3700: in close
    file_handle = self.filesystem.get_open_file(file_des)
/service/nox/test-3-6/lib/python3.6/site-packages/pyfakefs/fake_filesystem.py:1355: in get_open_file
    self.raise_os_error(errno.EBADF, str(file_des))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7fa25d6431d0>, errno = 9, filename = '10', winerror = None

    def raise_os_error(self, errno, filename=None, winerror=None):
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.
    
        Args:
            errno: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = self._error_message(errno)
        if (winerror is not None and sys.platform == 'win32' and
                self.is_windows_fs):
            if IS_PY2:
                raise WindowsError(winerror, message, filename)
            raise OSError(errno, message, filename, winerror)
>       raise OSError(errno, message, filename)
E       OSError: [Errno 9] Bad file descriptor in the fake filesystem: '10'

/service/nox/test-3-6/lib/python3.6/site-packages/pyfakefs/fake_filesystem.py:976: OSError
==================================================================================== 1 failed in 0.71 seconds =====================================================================================

Please provide a unit test or a minimal code snippet that reproduces the
problem.

Your enviroment

Linux-4.20.13-200.fc29.x86_64-x86_64-with
Python 3.6.5 (default, Aug 22 2018, 14:30:18) 
pyfakefs 3.5.8

[mrbean-bremen]

Thanks for the report! The problem seems to be that os.pipe creates real file descriptors, which is okay, as we don't have to fake pipes, but os.close is implemented in the fake filesystem and doesn't know about real file descriptors.
I will have a closer look somewhere next week (traveling right now).

[mrbean-bremen]

@saml - I added support for handling os.pipe() in master, please check if this fixes your problem.

[saml]

@mrbean-bremen master works with the particular testcase around closing os.pipe.

I have another testcase related to multiprocessing that fails master. Would this be a new issue?
Failure happens when I open a file and close a file in multiprocessing.

Stacktrace is:

Traceback (most recent call last):
  File "/home/saml/code/src/github.com/jmcgeheeiv/pyfakefs/test_bar.py", line 15, in do_something
    pass
  File "/home/saml/code/src/github.com/jmcgeheeiv/pyfakefs/pyfakefs/fake_filesystem.py", line 4646, in __exit__
    self.close()
  File "/home/saml/code/src/github.com/jmcgeheeiv/pyfakefs/pyfakefs/fake_filesystem.py", line 4672, in close
    self.flush()
  File "/home/saml/code/src/github.com/jmcgeheeiv/pyfakefs/pyfakefs/fake_filesystem.py", line 4712, in flush
    self._flush_related_files()
  File "/home/saml/code/src/github.com/jmcgeheeiv/pyfakefs/pyfakefs/fake_filesystem.py", line 4722, in _flush_related_files
    self.file_object == open_file.file_object and
AttributeError: 'FakePipeWrapper' object has no attribute 'file_object'

Test case is:

def do_something(result_queue):
    try:
        with open('/tmp/abc/d.txt', 'w') as f:
            pass  # this line is test_bar.py", line 15   in the above stacktrace.
    except Exception as err:
        _log.exception('bad')
        result_queue.put(err)
    else:
        result_queue.put()


def test_0(fs):
    fs.create_dir('/tmp/abc/')
    result_queue = multiprocessing.Queue()
    p = multiprocessing.Process(target=do_something, args=(result_queue,))
    p.start()
    result = result_queue.get()
    if result:
        raise result
    _log.info('result: %s', result)
    p.join()

[mrbean-bremen]

Ah ok, I missed that - thanks. Will have a look later (and yes, this is the same issue).

[mrbean-bremen]

Hm, while the exception you encountered is due to a simple error I just fixed, I'm afraid that using multiprocessing may be problematic anyway. I just tested your example under Windows, and it failed due to the usage of Windows API access to the file system, which we cannot fake. I haven't looked at the unix implementation, but I'm afraid there will be similar problems that could only be resolved by patching multiprocessing itself.

[mrbean-bremen]

@saml - did you get anywhere with that?

[saml]

Currently with master, running the same test case that involves multiprocessing hangs indefinitely (or until pytest timeout).
I'll close this and patch multiprocessing.

[tomhey]

Quick question, what's the state of multiprocessing support? I'm not quite clear given your last comment and that this bug is closed.
Is it the case that the os.pipe close issue is fixed, but there are further issue around multiprocessing?

[mrbean-bremen]

Well, multiprocessing has never been tested, so I would say it is not supported.
This never came up before, as unit tests usually do not use this - tests that are using multiprocessing are more likely on the level of module tests, and less likely will use pyfakefs.

I had a quick look at the implementation of multiprocessing, and from what I saw, there are a couple of things that certainly won't work with pyfakefs. For example, it uses memory mapping (mmap), which does not work with the fake file system, it uses functions that are currently not faked (like os.statvfs), and some OS-specific stuff that also may also be problematic. It may be possible to get it working with quite a bit of effort, but currently I don't see that happen in the foreseeable future (provided there are no volunteers for this :).

That being said, this is something that may need better documentation - there is currently only a small limitations chapter, that is probably not sufficient. I may have a go at adding a bit more information regarding problems like this one.

[mrbean-bremen]

@tomhey - I added a small chapter for this kind of problems.