Method to explicitly close memmap

[lepmik]

As far as I understand, there are currently two ways to close a memmap "file"; del fp or fp._mmap.close(). However, the former only closes the file if fp is the only reference to the memmap and the latter crashes the python interpreter if there exists another reference to the memmap.

Are there any plans of implementing a close method i.e. fp.close()? It would be much appreciated.

[mattip]

What would you expect to happen if the memmap is held by more than one np.memmap? Should it somehow be aware of the other instance and not close the resource?

I agree we should definitely not segfault, that is rarely expected behaviour

[eric-wieser]

Python solves this problem by:

• Having mmap.close() count the number of referrers to the underlying data, and throw an exception if it's not zero. This prevents memoryview(x) objects outliving the mmap
• Making all of the methods of mmap raise ValueError once it is closed.

We could emulate the first one, but given that np.memmap inherits from np.ndarray which has C-level attributes, there's no way to achieve the second. To make this work, we'd need to switch np.memmap from being a subclass to an array_like.

[lepmik]

I would expect the other references to be instances of a closed mmap, and that all methods and attributes to be unavailable. At least it should not be read/writable after closing. Throwing an error if the refcount is not zero is better behavior than the present one, but I think a more natural behavior would be to throw the error when the reference actually tries to reach the data ala:

[eric-wieser]

The mmap example I'm thinking of is:

with open('testfile', 'r+') as f:
    m = mmap.mmap(f.fileno(), 0)

mv = memoryview(m)

m.close()  # fails, because `mv` is still alive
del mv
m.close()  # ok

So there's a distinction between the direct refcount of the map object itself, and the number of "view" objects that are pointing to it.

[uellue]

A close() method turns out to be important for Windows when renaming files. This seems to be a timing issue between del and renaming if threads are active. We experienced the problem mainly if a Dask.distributed client had been started before, not under simple circumstances.

This code snippet can trigger failures since the mmap object may not be closed yet:

self._dest.flush()  # msync

# to make Windows™ happy:
self._tmp_file.close()
del self._dest  # self._dest is a numpy.memmap object

# os.rename() can fail with
# PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
os.rename(self._tmp_file.name, self._path)

dest_dir = os.path.dirname(self._path)
self.sync_dir(dest_dir)
self._tmp_file = None
self._dest = None

Closing the _mmap object fixes the error:

self._dest.flush()  # msync

# to make Windows™ happy:
self._tmp_file.close()

# self._dest is a numpy.memmap object
mm = self._dest._mmap
if mm is not None:
    mm.close()

os.rename(self._tmp_file.name, self._path)
dest_dir = os.path.dirname(self._path)
self.sync_dir(dest_dir)
self._tmp_file = None
self._dest = None

Under these circumstances it would be great to have an explicit close() method instead of using internals of numpy.memmap.

[mattip]

This is important for supporting non-refcount GC strategies like in PyPy, where it is not clear when if at all fp will be freed after calling del fp.