New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add n-dim data base class for gammapy.irf #527

Merged
merged 12 commits into from May 25, 2016

Conversation

Projects
None yet
3 participants
@joleroi
Contributor

joleroi commented May 10, 2016

This PR introduces gammapy.utils.nddata
This module contains a base class for an NDData array. It is supposed to be subclassed by all classes in `gammapy.irf`` and maybe in other places.

An scratch EffectiveArea2D class is also part of this PR for the sake of illustration.

@joleroi joleroi added this to the 1.0 milestone May 10, 2016

@cdeil cdeil modified the milestones: 0.5, 1.0 May 10, 2016

Show outdated Hide outdated gammapy/utils/tests/test_nddata.py Outdated
Show outdated Hide outdated gammapy/utils/tests/test_nddata.py Outdated
Show outdated Hide outdated gammapy/utils/nddata.py Outdated
Show outdated Hide outdated gammapy/utils/nddata.py Outdated
Show outdated Hide outdated gammapy/utils/tests/test_nddata.py Outdated
Show outdated Hide outdated gammapy/utils/nddata.py Outdated
Show outdated Hide outdated gammapy/utils/nddata.py Outdated

@cdeil cdeil self-assigned this May 10, 2016

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil May 10, 2016

Member

I've done a superficial review and left some comments inline.

Do you want to apply it to AEFF or some other IRF in this PR, or add some simple dummy IRF in the test or as an example? (I'm not sure what is best ... either way this will be a large PR, bit it'll still be OK to review.)

Member

cdeil commented May 10, 2016

I've done a superficial review and left some comments inline.

Do you want to apply it to AEFF or some other IRF in this PR, or add some simple dummy IRF in the test or as an example? (I'm not sure what is best ... either way this will be a large PR, bit it'll still be OK to review.)

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 12, 2016

Contributor

Thanks for you comments. I added an minimal aeff example file. Next thing is to test if the NDDataArray FITS I/O works for the test datastore

cc @JouvinLea Since you're also working on IRF stuff maybe you want to have a look, too? If it comes to the worst you will have to use this class at some point 😉

Contributor

joleroi commented May 12, 2016

Thanks for you comments. I added an minimal aeff example file. Next thing is to test if the NDDataArray FITS I/O works for the test datastore

cc @JouvinLea Since you're also working on IRF stuff maybe you want to have a look, too? If it comes to the worst you will have to use this class at some point 😉

@joleroi joleroi changed the title from WIP: ND data base class for gammapy.irf to ND data base class for gammapy.irf May 12, 2016

@JouvinLea

This comment has been minimized.

Show comment
Hide comment
@JouvinLea

JouvinLea May 12, 2016

Contributor

Sorry I really didn't follow this PR... What is the purpose exactly?

Contributor

JouvinLea commented May 12, 2016

Sorry I really didn't follow this PR... What is the purpose exactly?

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 12, 2016

Contributor

I'll answer via email

Contributor

joleroi commented May 12, 2016

I'll answer via email

joleroi added some commits May 12, 2016

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 19, 2016

Contributor

@cdeil I added an example EffectiveArea2D class, that uses the NDData base class. Please review. I will continue with moving functionality from the old EffectiveAreaTable2D class in the next days.

Contributor

joleroi commented May 19, 2016

@cdeil I added an example EffectiveArea2D class, that uses the NDData base class. Please review. I will continue with moving functionality from the old EffectiveAreaTable2D class in the next days.

Show outdated Hide outdated gammapy/irf/effective_area.py Outdated
Show outdated Hide outdated gammapy/utils/nddata.py Outdated
Show outdated Hide outdated gammapy/irf/effective_area.py Outdated
Show outdated Hide outdated gammapy/irf/effective_area.py Outdated
Show outdated Hide outdated gammapy/irf/effective_area.py Outdated
----------
vmin : `~astropy.units.Quantity`, float
Lowest value
emax : `~astropy.units.Quantity`, float

This comment has been minimized.

@cdeil

cdeil May 19, 2016

Member

emax -> vmax

@cdeil

cdeil May 19, 2016

Member

emax -> vmax

Show outdated Hide outdated gammapy/utils/tests/test_nddata.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil May 19, 2016

Member

I looked at this for 10 minutes, but I would need at least an hour to read through your implementation and make good suggestions. I can do this later if you want, but not this week.

Naively I would have expected some scheme where a given IRF is a subclass and has class-level attributes to declare which axes are present and maybe some other things.
This is how e.g. Django models work or Astropy models or Astropy coordinate frames.
See https://docs.djangoproject.com/en/1.9/topics/db/models/

The axes are class-specific and not instance-specific in your implementation, right?
Then probably the axis configuration declaration for a given IRF should probably be class-level attributes like in the Django model example I linked to, no?

(The alternative would be that there is a factory function to create specific IRFs or a single class that is configurable on construction.)

Does this comment make any sense? Or should I give a code example how I would have expected the AEFF class implementation to look like?

Member

cdeil commented May 19, 2016

I looked at this for 10 minutes, but I would need at least an hour to read through your implementation and make good suggestions. I can do this later if you want, but not this week.

Naively I would have expected some scheme where a given IRF is a subclass and has class-level attributes to declare which axes are present and maybe some other things.
This is how e.g. Django models work or Astropy models or Astropy coordinate frames.
See https://docs.djangoproject.com/en/1.9/topics/db/models/

The axes are class-specific and not instance-specific in your implementation, right?
Then probably the axis configuration declaration for a given IRF should probably be class-level attributes like in the Django model example I linked to, no?

(The alternative would be that there is a factory function to create specific IRFs or a single class that is configurable on construction.)

Does this comment make any sense? Or should I give a code example how I would have expected the AEFF class implementation to look like?

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil May 19, 2016

Member

This is an example of a single class that is configurable on construction:
http://xarray.pydata.org/en/stable/data-structures.html#creating-a-dataarray

For our IRFs I don't know if one configurable class or a sub-class zoo would work better.

Member

cdeil commented May 19, 2016

This is an example of a single class that is configurable on construction:
http://xarray.pydata.org/en/stable/data-structures.html#creating-a-dataarray

For our IRFs I don't know if one configurable class or a sub-class zoo would work better.

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 19, 2016

Contributor

@cdeil thanks. you already helped a lot. Did I understand you correctly that you would have expected something like

class EffArea
    __init__(self, energy, offset, ...):
        self.energy = DataAxis(energy)
        self.offset = DataAxis(offset)

This is how I had it in mind originally. I changed to the 'weird' scheme because I have a from_table constructor in the base class (NDDataArray) that interferes with that (I already adds axes, so the subclasses cannot call this constructor). But as you say, the from_table methods should be more specific I guess (read a fixes set of columns, instead of trying to guess). I put it to the base class in order to be able to read/write any NDDataArray and not only the subclasses.

I will change this, and then add the plotting methods etc. to make the review easier, ok?

UPDATE: The NDDataArray has an axes attribute, that is use to set up the interpolator etc.This would still be present in the EffectiveArea2d. So the constructor would be more like

class EffArea
    __init__(self, energy, offset, ...):
         super.__init__()
        self.energy = self.axes[0]
        self.offset = self.axes[1]

Contributor

joleroi commented May 19, 2016

@cdeil thanks. you already helped a lot. Did I understand you correctly that you would have expected something like

class EffArea
    __init__(self, energy, offset, ...):
        self.energy = DataAxis(energy)
        self.offset = DataAxis(offset)

This is how I had it in mind originally. I changed to the 'weird' scheme because I have a from_table constructor in the base class (NDDataArray) that interferes with that (I already adds axes, so the subclasses cannot call this constructor). But as you say, the from_table methods should be more specific I guess (read a fixes set of columns, instead of trying to guess). I put it to the base class in order to be able to read/write any NDDataArray and not only the subclasses.

I will change this, and then add the plotting methods etc. to make the review easier, ok?

UPDATE: The NDDataArray has an axes attribute, that is use to set up the interpolator etc.This would still be present in the EffectiveArea2d. So the constructor would be more like

class EffArea
    __init__(self, energy, offset, ...):
         super.__init__()
        self.energy = self.axes[0]
        self.offset = self.axes[1]

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil May 19, 2016

Member

@joleroi - To be honest I don't know what I want / expect here.

The goal should be that the class is nice to use that that each individual IRF class is small, basically declarative which axes and binning are present, the boilerplate code should be in the base class.

Something like this:

In [3]: Disk2D.__init__??
Signature: Disk2D.__init__(self, amplitude=1, x_0=0, y_0=0, R_0=1, **kwargs)
Source:
            def __init__(self, *params, **kwargs):
                return super(cls, self).__init__(*params, **kwargs)

File:      ~/Library/Python/3.5/lib/python/site-packages/astropy-1.2.dev15393-py3.5-macosx-10.11-x86_64.egg/astropy/modeling/core.py
Type:      function

In [4]: Disk2D
Out[4]: 
<class 'astropy.modeling.functional_models.Disk2D'>
Name: Disk2D
Inputs: ('x', 'y')
Outputs: ('z',)
Fittable parameters: ('amplitude', 'x_0', 'y_0', 'R_0')

In [5]: from astropy.modeling.models import Disk2D

There the parameters (axes in your case) are class-level attributed (see here) and then some dark magic is generating __init__.

I now see that in Sherpa the parameters are constructed in __init__ (like you're constructing the axes in __init__ now:
https://github.com/sherpa/sherpa/blob/3dc0c33146ea5c93b0dcff07be804e25769d7642/sherpa/astro/models/__init__.py#L582

I'll try to think about this and read up on class-level attributes in Python over the weekend.

But yes ... adding tests or examples that show how your current class is used and that it works would be very helpful for review.

Member

cdeil commented May 19, 2016

@joleroi - To be honest I don't know what I want / expect here.

The goal should be that the class is nice to use that that each individual IRF class is small, basically declarative which axes and binning are present, the boilerplate code should be in the base class.

Something like this:

In [3]: Disk2D.__init__??
Signature: Disk2D.__init__(self, amplitude=1, x_0=0, y_0=0, R_0=1, **kwargs)
Source:
            def __init__(self, *params, **kwargs):
                return super(cls, self).__init__(*params, **kwargs)

File:      ~/Library/Python/3.5/lib/python/site-packages/astropy-1.2.dev15393-py3.5-macosx-10.11-x86_64.egg/astropy/modeling/core.py
Type:      function

In [4]: Disk2D
Out[4]: 
<class 'astropy.modeling.functional_models.Disk2D'>
Name: Disk2D
Inputs: ('x', 'y')
Outputs: ('z',)
Fittable parameters: ('amplitude', 'x_0', 'y_0', 'R_0')

In [5]: from astropy.modeling.models import Disk2D

There the parameters (axes in your case) are class-level attributed (see here) and then some dark magic is generating __init__.

I now see that in Sherpa the parameters are constructed in __init__ (like you're constructing the axes in __init__ now:
https://github.com/sherpa/sherpa/blob/3dc0c33146ea5c93b0dcff07be804e25769d7642/sherpa/astro/models/__init__.py#L582

I'll try to think about this and read up on class-level attributes in Python over the weekend.

But yes ... adding tests or examples that show how your current class is used and that it works would be very helpful for review.

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 20, 2016

Contributor

@cdeil
I start to understand the 2 different schemes you propose. Right now I don't know how I should make the base class work with class level attributes. All the interpolation etc. work with calls to self.axes and loops. How could this be changed to use class level attributes like self.offset? I will take 10 min. to try to understand this 😄

Otherwise I will continue with the 'construction on initialization' scheme. If we encounter severe problems later, we have to find a way to switch to the other scheme.

Contributor

joleroi commented May 20, 2016

@cdeil
I start to understand the 2 different schemes you propose. Right now I don't know how I should make the base class work with class level attributes. All the interpolation etc. work with calls to self.axes and loops. How could this be changed to use class level attributes like self.offset? I will take 10 min. to try to understand this 😄

Otherwise I will continue with the 'construction on initialization' scheme. If we encounter severe problems later, we have to find a way to switch to the other scheme.

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 23, 2016

Contributor

I worked in this with @adonath and we think the subclassing API is sufficiently elegant for now (not as fancy as astropy.models but it does the job, if someone has time it can always be improved). Axes are class level attributes defined in each subclass now.
https://github.com/gammapy/gammapy/pull/527/files#diff-cfb4e8b67c434385b17b6cd254d4c5b0R42
The order has to be defined in a separate attribute axis_names (astropy magically gets the order from the order the class level attributes are written in)

The drawback is that NDData cannot be instanciated anymore (since it does not have any axes defined).

Contributor

joleroi commented May 23, 2016

I worked in this with @adonath and we think the subclassing API is sufficiently elegant for now (not as fancy as astropy.models but it does the job, if someone has time it can always be improved). Axes are class level attributes defined in each subclass now.
https://github.com/gammapy/gammapy/pull/527/files#diff-cfb4e8b67c434385b17b6cd254d4c5b0R42
The order has to be defined in a separate attribute axis_names (astropy magically gets the order from the order the class level attributes are written in)

The drawback is that NDData cannot be instanciated anymore (since it does not have any axes defined).

joleroi added some commits May 23, 2016

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi May 25, 2016

Contributor

@cdeil I will merge this now. The example class EffectiveArea2d covers all the functionality from the old class EffectiveAreaTable2d but is not used anywhere. Next I will add a 1D effective area class based on NDDataArray. You can have a look next week and then I will remove the old irf.effective_area_table module and replace it with the new classes, if nobody raises any concerns.

Contributor

joleroi commented May 25, 2016

@cdeil I will merge this now. The example class EffectiveArea2d covers all the functionality from the old class EffectiveAreaTable2d but is not used anywhere. Next I will add a 1D effective area class based on NDDataArray. You can have a look next week and then I will remove the old irf.effective_area_table module and replace it with the new classes, if nobody raises any concerns.

@joleroi joleroi merged commit 7ae7728 into gammapy:master May 25, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@joleroi joleroi deleted the joleroi:nddata branch May 25, 2016

@cdeil cdeil changed the title from ND data base class for gammapy.irf to Add n-dim data base class for gammapy.irf Jun 9, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment