Inheritance of Purity

[jrmlhermitte]

(As discussed here: http://stackoverflow.com/questions/42773134/inheritance-of-purity/42773643)

I have a question about inheritance of function purity. For example, consider this case:

In [1]: from dask import delayed

In [2]: myArr = delayed(np.ones, pure=True)((10,10))

In [3]: myArr
Out[3]: Delayed('ones-850ff41a84b309775d01d5f3e5c4d1c4')

In [4]: myArr = delayed(np.ones, pure=True)((10,10))

In [5]: myArr
Out[5]: Delayed('ones-850ff41a84b309775d01d5f3e5c4d1c4')

In [6]: myArr.shape
Out[6]: Delayed('getattr-de14f312-2605-4fc4-a419-c376520f73b4')

In [7]: myArr.shape
Out[7]: Delayed('getattr-9e547a62-0bc6-43e5-8d3c-cac532b73511')

In [8]: delayed(getattr, pure=True)(myArr, 'shape')
Out[8]: Delayed('getattr-5224be928bad33c9778d4f96d610cc37')

In [9]: delayed(getattr, pure=True)(myArr, 'shape')
Out[9]: Delayed('getattr-5224be928bad33c9778d4f96d610cc37')

I would expect line [6] and [7] to yield the same key, since they are accessing an attribute from a delayed instance declared pure. However, it is not the case, and I must explicitly declare the getattr method to be pure for this to work, as seen in lines [8] and [9].

As discussed in the stackoverflow link, this seems to be purely an issue of dask semantics. I would be inclined to voting for purity to be inherited. But I am a new user, so I would appreciate feedback/comments on this. Thanks!

PS: The eventual use case is for dask distributed. It seems that the way purity is treated in this case may possibly be different than the case listed here.

[shoyer]

delayed(getattr,pure=True)(myArr, 'shape',pure=True) looks like one more pure=True than necessary -- are both of those really necessary?

[jrmlhermitte]

oops yes, that was a typo, it still ran since it's treated as a kwarg to a delayed object. I corrected it, thanks!

[jrmlhermitte]

For the record, getitem seems to be forced as pure=True, as opposed to getattr:

In [6]: myArr = delayed(np.ones, pure=False)((10,10))

In [7]: myArr[0]
Out[7]: Delayed('getitem-f7b0bc2869f1fe6c1221edc2aaac0a73')

In [8]: myArr[0]
Out[8]: Delayed('getitem-f7b0bc2869f1fe6c1221edc2aaac0a73')

Would it make sense instead of choosing purity from an absolute standpoint to choose it from a relative inherited standpoint? I don't think getitem will always necessary yield something pure. (For example, could be an object with a simulated set of random images fed into some image processing algorithm that runs on image sequences, in a loop.) This may be opening a can of worms. Any discussion/feedback appreciated!

[mrocklin]

Just to be clear on terms here, a delayed object does not currently know if it is "pure" or not. Only delayed functions can be labeled as pure. I suspect that the confusion here arises because we use the function delayed both to mark a value as part of a graph and to decorate a function.

Even if we kept the label around it's not clear that the attributes of "pure" objects would themselves be pure.

[jcrist]

I'd be fine with the following:

• There's no concept of a "pure" object. Nested attribute accesses, function calls, etc... make this too hard to reason about. This should be clarified in the docs as needed.
• Attribute access is always pure. We already assume that all "magic" methods on delayed objects are pure, so we shouldn't make __getattr__ the exception (which was unintentional).
• Perhaps error if delayed is called on a non-callable object with pure=True to better indicate that pure=True only indicates that calling the resulting object is a pure operation.

Thoughts?

[mrocklin]

delayed(obj, pure=True) is currently used as a signal that we should take the trouble to hash the object down to get the key.

[jcrist]

Ah, ok, not the last one then.

[jcrist]

I applied my proposal above in #2084. I think this is internally consistent, so this approach would be my vote.

[jrmlhermitte]

Thanks, I vote for this as well :-). I like the elaborated documentation, very useful as well!