mmap objects used in base should expose a filename attribute

[GaelVaroquaux]

In numpy <= 1.6 it is possible to inspect arrays to find out if they come from memmapped files and which file:

>>> import numpy as np
>>> a = np.ones((10, 10))
>>> np.save('a.npy', a)
>>> b = np.load('a.npy', mmap_mode='r')
>>> c = np.asarray(b)
>>> c.base is b
True
>>> c.base.filename
'/home/varoquau/a.npy'

In 1.7b2, the 'base' of 'c' is now a reference to the Python mmap object, and thus it is now impossible to get back to original filename. This may be an important usecase in parallel computing (e.g. with multiprocessing, or IPython) to control if data is shared between processes or not.

>>> import numpy as np
>>> a = np.ones((10, 10))
>>> np.save('a.npy', a)
>>> b = np.load('a.npy', mmap_mode='r')
>>> c = np.asarray(b)
>>> c.base

I propose to use as a 'base' in the numpy memmap objects something that keeps tracks of the filename (and maybe the offset, although it can be retrieved with a bit of magic). The simplest option would probably to simply use a subclass of mmap.mmap.

I, @GaelVaroquaux, volunteer to implement such a patch, if the strategy above is deemed as acceptable.

[teoliphant]

I think it makes sense to have a standard base object interface for all NumPy arrays --- so that if base is not none it points to something with a standard interface.

In other words, I like this but for all NumPy arrays and not just memmap arrays.

Travis Oliphant
(on a mobile)
512-826-7480

On Sep 30, 2012, at 10:50 AM, Gael Varoquaux notifications@github.com wrote:

In numpy <= 1.6 it is possible to inspect arrays to find out if they come from memmapped files and which file:

import numpy as np
a = np.ones((10, 10))
np.save('a.npy', a)
b = np.load('a.npy', mmap_mode='r')
c = np.asarray(b)
c.base is b
True
c.base.filename
'/home/varoquau/a.npy'
In 1.7b2, the 'base' of 'c' is now a reference to the Python mmap object, and thus it is now impossible to get back to original filename. This may be an important usecase in parallel computing (e.g. with multiprocessing, or IPython) to control if data is shared between processes or not.

import numpy as np
a = np.ones((10, 10))
np.save('a.npy', a)
b = np.load('a.npy', mmap_mode='r')
c = np.asarray(b)
c.base
I propose to use as a 'base' in the numpy memmap objects something that keeps tracks of the filename (and maybe the offset, although it can be retrieved with a bit of magic). The simplest option would probably to simply use a subclass of mmap.mmap.

I, @GaelVaroquaux, volunteer to implement such a patch, if the strategy above is deemed as acceptable.

—
Reply to this email directly or view it on GitHub.

[GaelVaroquaux]

I think it makes sense to have a standard base object interface for all NumPy
arrays --- so that if base is not none it points to something with a standard
interface.

OK, what kind of field do you think would be useful?

[njsmith]

@teoliphant: Err.... why? Right now the interface is "this is a refcountable Python object", which matches the semantics of "base" (i.e., specifying an object that we will drop a reference to on destruction). Except for UPDATEIFCOPY, which overloads this to mean something else entirely, in which case the interface is "this is a type-and-shape-compatible ndarray". And Gael is suggesting we formalize another overload, for the memmap subclass of ndarray. Plus there are tons of codes out there that set .base to random opaque object handles and stuff. What possible common interface would all of these objects have?

[teoliphant]

If we are going to make .base always point to the underlying byte source instead of the most recent object, I think we need some common things we can expect from that object.

Another approach, though, is to just halt the traversal of base back to the first object of that subtype/subclass

I think that would fix the underlying problem Gael is struggling with without defining another buffer or array interface.

Travis

Travis Oliphant
(on a mobile)
512-826-7480

On Sep 30, 2012, at 12:34 PM, njsmith notifications@github.com wrote:

@teoliphant: Err.... why? Right now the interface is "this is a refcountable Python object", which matches the semantics of "base" (i.e., specifying an object that we will drop a reference to on destruction). Except for UPDATEIFCOPY, which overloads this to mean something else entirely, in which case the interface is "this is a type-and-shape-compatible ndarray". And Gael is suggesting we formalize another overload, for the memmap subclass of ndarray. Plus there are tons of codes out there that set .base to random opaque object handles and stuff. What possible common interface would all of these objects have?

—
Reply to this email directly or view it on GitHub.

[GaelVaroquaux]

On Sun, Sep 30, 2012 at 11:01:23AM -0700, Travis E. Oliphant wrote:

I think that would fix the underlying problem Gael is struggling with without
defining another buffer or array interface.

Probably, and it does look like a sound and light rule. Although in the
specific example I gave 'c' is of type np.ndarray, whereas its parent is
of type np.memmap. The ndarray object should be able to accept a memmap
as a parent, I believe.

[teoliphant]

On Sep 30, 2012, at 1:08 PM, Gael Varoquaux wrote:

On Sun, Sep 30, 2012 at 11:01:23AM -0700, Travis E. Oliphant wrote:

I think that would fix the underlying problem Gael is struggling with without
defining another buffer or array interface.

Probably, and it does look like a sound and light rule. Although in the
specific example I gave 'c' is of type np.ndarray, whereas its parent is
of type np.memmap. The ndarray object should be able to accept a memmap
as a parent, I believe.

So isinstance(c.base, np.ndarray) would work if c.base were a memmap so it would be acceptable according to the proposed rule.

Specifically the proposal is to set the base object of an instance to the top-most object that is still is of the same type as the instance in question.

Would this solve your underlying problem, Gael? It seem like a simple change that solves the bug but also preserves some of the same semantics people might be expecting of "base" in situations like yours.

-Travis

[GaelVaroquaux]

On Sun, Sep 30, 2012 at 11:13:48AM -0700, Travis E. Oliphant wrote:

So isinstance(c.base, np.ndarray) would work if c.base were a memmap so it
would be acceptable according to the proposed rule.

Yes.

Would this solve your underlying problem, Gael? It seem like a simple change
that solves the bug but also preserves some of the same semantics people might
be expecting of "base" in situations like yours.

I do believe that it would solve my problem. I also think that would
match a bit more the expectation of people. I don't really see a downside
of it. Maybe this change would need to be run by more people, but as far
as I am concerned, it seems a good thing, and would make me happy.

Thanks!

[certik]

This was the behavior in b2:

>>> import numpy as np
>>> a = np.ones((10, 10))
>>> np.save('a.npy', a)
>>> b = np.load('a.npy', mmap_mode='r')
>>> c = b.view()
>>> c.base.filename
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'mmap.mmap' object has no attribute 'filename'

but now after the PR #2708 was merged and backported to the 1.7.x branch by the commit a9c7b38, the behavior is:

>>> import numpy as np
>>> a = np.ones((10, 10))
>>> np.save('a.npy', a)
>>> b = np.load('a.npy', mmap_mode='r')
>>> c = b.view()
>>> c.base.filename
'/tmp/a.npy'

So this problem should be fixed and I am closing this issue. @GaelVaroquaux, please let me know if all is fixed now.

[GaelVaroquaux]

Sorry, numpy team, I have had to focus on many other things, and
completely lost track of this issue.

I'll try to come back to it. In the mean time, @ogrisel, could you also
have a look: you spent a while on these problems too.

Thanks guys!

[ogrisel]

Thanks for the heads up, I will check that it fixes our problem as soon as I can find some spare cycles.