NDData and NDDataBase Refactor

[MSeifert04]

This is the refactor of NDDataBase and NDData (from #4234). More comments will follow as soon as I get the splitted parts to work.

edit 12/22/2015 to add checklist for completion:

• Update documentation to reflect changes
• Remove restriction on type of uncertainty_type (ping list notifying about this)
• Always return the uncertainty object, not the data contained in that object, even if an UnknownUncertainty is created by NDData

[MSeifert04]

Summary:
__init__:

• New optional copy parameter. The default is False which was the case before this PR.
• Implicit parameters (i.e. data parameter is another NDData instance) are always checked. This might have produced Bugs if subclasses had less requirements.
• data can now be a Masked_Quantities instance
• data with mask is only allowed if they have also a data attribute (Duck typing MaskedArrays)
• If an implicit and explicit parameter is given (i.e. data was a Quantity and a unit was given) the implicit one is ignored and a warning is raised! Before this PR that was not always done, i.e. two different units raised an Exception while two different masks raised a Warning.

uncertainty

• The uncertainty setter of NDDataBase has moved to NDData
• uncertainties get their parent_nddata attribute set in the setter. (see issue NDArithmeticMixin uncertainty propagation for multiplication/division #4152)
• The uncertainty setter now checks for an attribute uncertainty_type and if it is not present or not a string the uncertainty is wrapped in an UnknownUncertainty. (Before it raised an Exception)
• The getter uncertainty special cases UnknownUncertaintiy and returns only what was initially wanted as uncertainty but the internally stored _uncertainty is of unknown uncertainty_type.

I think this is mostly added functionality and should be backwards compatible, except for the uncertainty setter. Subclasses of NDDataBase might encounter some transitional problems since the one is now gone. All other changes only lessened restrictions (Warning instead of Exceptions).

But I now know, that some of these changes will be somehow controversial so I would be very happy about comments.

[mwcraig]

Is the idea here that eventually every NDData object would have an uncertainty, defaulting to UnknownUncertainty, eventually? Otherwise it seems like leaving the uncertainty_type a property of the uncertainty would be easier.

[mwcraig]

Ah, right, UnknownUncertainty is at the top of the file.

[MSeifert04]

The uncertainty_type property was just a convenience I used while I tested the UnknownUncertainty. I thought I deleted it before pushing - but I forgot.

[mwcraig]

@wkerzendorf -- could you please take a look at this one? Looks mostly straightforward to me, the one big change is to wrap the uncertainty in an UnknownUncertainty if the uncertainty does not have an attribute uncertainty_type.

[mwcraig]

@MSeifert04 -- once again, thanks. I gave this a quick look tonight and didn't see any problems with the code, but I'm not sure about dropping the requirement that the uncertainty have an uncertainty_type (primarily because the APE currently says it should be there).

It sounds, though, like your solution is to wrap an uncertainty that does not have an uncertainty_type in an UnkownUncertainty, right? So in the end there is always an uncertainty_type?

[MSeifert04]

@mwcraig - Yes, your summary about the UnknownUncertainty is correct.
But note that I also changed the uncertainty getter so that for UnknownUncertainty it doesn't return the instance but only the saved array of the instance. So a normal user would not suspect that the uncertainty was modified (except in some cases, i.e. the uncertainty was a list and cast to a np.ndarray). It felt a bit like cheating to be honest and if you don't like it I can revert back to the former enforced requirement.

I got the idea from #3714 where @wkerzendorf proposed just such an uncertainty and I modified it a bit to match my NDUncertainty changes.

Thanks for looking over it!

[mwcraig]

I'll have some time tomorrow to take a longer look at the others, and to put a call out on the astropy-dev list for code reviews.

I know a bunch of people are at ADASS this week...

[mwcraig]

Recommend switching this back to return self._uncertainty

[MSeifert04]

In the Hangouts meeting @wkerzendorf and @mwcraig suggested some changes. The current push reflects these:

• UnknownUncertainty is handled like any other uncertainty when accessed via the uncertainty property.
• uncertainty_type needs to be present for uncertainty.setter but we dropped the requirement for it to be a string.

I skipped the documentation update for now.

[mwcraig]

@MSeifert04 -- could you please go ahead and do the documentation changes too (should be minimal, I think, for this one)?

[MSeifert04]

@mwcraig - I edited the documentation files to reflect the changes and included a section with a summary of the changes. I'm not very happy with the wording, just let me know if anything should be edited.

[mwcraig]

This and the line below are causing the sphinx failures because there is no getter or setter method.

[mwcraig]

Won't be an issue, though, if this just moves to the changelog.

[mwcraig]

@MSeifert04 -- looks really good, just a few minor inline comments. Most of the bullet points you added to index.rst should go in CHANGES.rst instead -- please put them in the section for version 1.2.

[mwcraig]

I don't think Quantity can be masked yet?

[mwcraig]

• Capitalize "the"
• Double backticks around uncertainty_type
• "must not be anymore." -> "is no longer required to be, but..."

[mwcraig]

@eteq -- this is a change from what was originally described in the APE but seems sensible to @wkerzendorf and I. IIRC you had strong feelings about this so wanted to point it out.

[mwcraig]

@MSeifert04 -- looking good, just some minor changes in the changelog entries.

[MSeifert04]

Apart from the changes, I just found that there is a descriptor lingering in the utils for the meta: https://github.com/astropy/astropy/blob/master/astropy/utils/metadata.py#L115

Since this is completly identical to the meta handling (except that that it implements a setter which isn't present until now) in NDData: Should I include it here?

Would look something like this

from astropy.utils.metadata import MetaData
class NDData(NDDataBase):
    meta = MetaData()
    ...

[wkerzendorf]

I think this is a very good idea. and you can subclass this to make the arithmetic operations on them.

[wkerzendorf]

But this PR does not deal with meta, right? Did we merge one of them?

[mwcraig]

@MSeifert04 -- could you please open a separate issue to remind us to look at moving metadata to the one in utils?

That way we can get this merged sooner but still remember we need to come back to the issue.

[MSeifert04]

@mwcraig I've opened a seperate issue and will make a seperate PR as soon as this is merged.

[mwcraig]

@MSeifert04 -- looks like this just needs a rebase (probably related to the changelog) then it is ready to merge.

[MSeifert04]

@mwcraig - My first rebase - I hope I did not screw it up. The rebase was because of #4466.

[mwcraig]

This is where the test failures are happening:

_______________________________ test_param_meta ________________________________
    def test_param_meta():
        # everything dict-like is allowed
        with pytest.raises(TypeError):
            NDData([1], meta=3)
        nd = NDData([1, 2, 3], meta={})
        assert len(nd.meta) == 0
        nd = NDData([1, 2, 3])
>       assert isinstance(nd.meta, OrderedDict)
E       assert isinstance(OrderedDict(), OrderedDict)
E        +  where OrderedDict() = NDData([1, 2, 3]).meta

[mwcraig]

Note to self: also checked this with ccdproc and nothing broke...

[mwcraig]

🎉 Thanks again for this @MSeifert04!