Add SkyDiffuseCube model for 3D maps

[robertazanin]

Add a class for SkyMap3d.
The test is not working because the energy interpolation is not properly working

[registerrier]

Thanks. See some suggestions inline.

[registerrier]

You might add a test to check that the Map has a single axis of type energy

[cdeil]

Yes, you could put something like this as input map validation:

if len(map.geom.axes) != 0:
    raise ValueError('Need a map with an energy axis')

axis = map.geom.axes[0]
if axis.name != 'energy':
    raise ValueError('Need a map with axis of name "energy"')

if axis.node_type != 'center':
    raise ValueError('Need a map with energy axis node_type="center"')

Usually we don't do so much input validation, but given that likely people will try to pass 2D maps or 3D maps that aren't appropriate here, I agree it's a good addition to check some things.

[registerrier]

Should give a flux unit.
Is the value expected to be integral of differential in a given energy bin? Maybe add a flag to check this no?

[cdeil]

I think norm should be a float (change that in the docstring), a unitless multiplicative normalisation factor. The diffuse model map cubes should already come with a unit and the appropriate thing (diffuse differential surface brightness e.g. in cm-2 s-1 MeV-1 sr-1).

[cdeil]

It's questionable if we want to have this norm parameter at all.
It would be worth a look how in Fermi-LAT analysis people change the diffuse model by a constant or even power-law spectrum, and how they handle that model composition and units.
We should probably do the same.

But for now, just this norm=1 no-op should give you what you need for DC1, and you should even be able to fit that norm parameter and it should come out ~ 1 .

[registerrier]

Maybe add a property to have access to the map itself.

[registerrier]

You can use the MapGeom.solid_angle(), especially if the norm is a flux unit.
It can probably be stored as a member of the class itself.

[cdeil]

The decision about what to put here exactly in evaluate is coupled to what we do in
http://docs.gammapy.org/dev/api/gammapy.cube.MapEvaluator.html

I think at the moment, we already apply solid_angle and energy integration there, so here in the diffuse 3D model evaluate, you don't have to do anything apart from interpolating the map value. However, I think the unit is currently incorrect, you could either do this:

return norm * val * u.Unit('cm-2 s-1 sr-1 MeV-1')

hardcoding to the Fermi-LAT diffuse maps, or if they set the unit correctly in the map header, you could try this:

return norm * val * u.Unit(self._map.unit)

You could test this by taking one run from CTA DC1 and computing an npred cube using the SkyModelEvaluator. You expect ~ 1 count, if it's 1e6 or 1e-6 there's a unit problem.

[registerrier]

The model should probably have a name, in case one wants to fit the norm.

[cdeil]

@robertazanin - I figures out why the map inter_by_coord wasn't giving what we wanted yesterday.

First of all, for diffuse model cubes, we need to create an energy MapAxis with node_type='center', either by passing that as an option to MapAxis or by calling Map.from_nodes. Then the map values will represent the diffuse model fluxes at the given energies.

Secondly, by default interp_by_coord on Map does nearest neighbor interpolation. Here we want to pass interp='linear' so that interpolation in energy ("linear" in log-log, i.e. actually piecewise power-law interpolation) takes place.

Here's an example one can use to understand how it works. A cube with 2 pixels in energy, with value 10 at energy 1 TeV and value 20 at energy 100 TeV:

import numpy as np
from gammapy.maps import Map, MapAxis
axis = MapAxis([1, 100], node_type='center', name="energy", unit='TeV', interp='log')
m = Map.create(npix=(4, 3), binsz=2, axes=[axis])
m.data += [[[10]], [[20]]]
coord = {'lon': 0, 'lat': 0, 'energy': 10}
pix = m.geom.coord_to_pix(coord)
val = m.interp_by_coord(coord, fill_value=0, interp='linear')

gives

>>> print(pix, val)
(array([1.5]), array([1.]), array([0.5])) [15.]

So the action items here are:

• Change the test case to such an energy axis, using the example given here.
• Add an evaluate for a sky position outside the map range
• Add an evaluate for an energy above or below the first or last node.
• Change both the 2D and the 3D class to use interp='linear'

In __init__, we could check if m.geom.axis[0].node_type is "center", and if it's "edge" error out with a good message.

A second test case should be added where you actually read a diffuse model cube file.
I tried this, and it does come with node_type="center" as it should, so you can use that one:

from gammapy.maps import Map
m = Map.read('test_datasets/unbundled/fermi/gll_iem_v02_cutout.fits')
axis = m.geom.axes[0]
print(axis)

@robertazanin - I'll leave this PR to you to finish up, or just come by on Monday and we do it together.

[adonath]

Just a few general comments maybe worth thinking about:

Why is it needed to introduce a new class for this? Can't the existing SkyDiffuseMap easily be modified to work in arbitrary dimensions? This is what I would expect it to do from it's name...

For efficiency I would also strongly suggest to not interpolate in the .evaluate() method, but rather pass an geom object on init, which does a reprojection of the diffuse model to the data geometry and just returns the diffuse map data on evaluate. The repeated interpolation is not needed as map geometries will never change during a fit...

[cdeil]

@adonath - What you suggest is not so easy. Note that the goal is to finish this today, I plan to work on this next. Basically I would suggest to postpone your suggestions to v0.9, unless you have a suggestion how to address these difficulties:

Why is it needed to introduce a new class for this? Can't the existing SkyDiffuseMap easily be modified to work in arbitrary dimensions? This is what I would expect it to do from it's name...

It probably could, but the evaluate argument handling might get complex. I think arbitrary extra axes in the model evaluation scheme will need much more thought and discussion and work.

For efficiency I would also strongly suggest to not interpolate in the .evaluate() method, but rather pass an geom object on init, which does a reprojection of the diffuse model to the data geometry and just returns the diffuse map data on evaluate. The repeated interpolation is not needed as map geometries will never change during a fit...

Is fixing the geom on which one can evaluate on model init a good idea? It would mean special-casing model init for this one model to already pass a geom. So far our logic is that model evaluate is just on coords, not passed a geom.

How about we keep this model as-is, and then special-case it in the MapEvaluator. In my mind combining model and geom in a clever way it that class's responsibility. Although I have to admit I'm not sure it's clean and easy to do the caching of the interpolator or even pre-evaluated cube on a given geom there, i.e. to store that state on MapEvaluator.

[adonath]

@cdeil With taking a Map object the model init for the diffuse model is already special cased. In fact one could even ask the user to do the reprojection beforehand and pass in a map with the right geometry already.

The two suggestion I made above go along with each other and would greatly simplify the implementation. For now I would simplify the SkyDiffuseMap to just take a map with the same geometry as the data and have only one global norm parameter...interpolation (if needed at all) could be added later as an option.

[cdeil]

So if we leave it up to the user or init to interpolate on a given geom, the evaluate becomes this, no?

def evaluate(self, *args, **kwargs):
    # Ignore any `args` or `kwargs`
    return self.parameters['norm'] * self._map.quantity

I'm still wondering if that is a good idea. I was assuming such model eval caching for a given geom is done (for many models) by the MapEvaluator and MapFit, not the model class itself.

I'll come by later to discuss ...

[adonath]

I discussed with @cdel offline and I agree it makes sense to introduce the model evaluation caching in a general way in the MapEvaluator or MapFit classes. So no further discussion from my side needed in this PR...

[cdeil]

@registerrier or @adonath - Please review.

[cdeil]

I've fixed the coord handling in 824ef28 to match what the MapEvaluator passes.

This model now works with MapEvaluator. For the maps from analysis_3d.ipynb I find that the Galactic diffuse model produces 3567 counts. I'm not sure yet if that is correct, using

for obs in obs_list:
    events = obs.events.table
    events = events[events['ENERGY'] > 0.1]
    print(obs.obs_id, np.sum(events['MC_ID'] == 2))

I find

110380 4339
111140 4019
111159 4442

i.e. 4000 IEM events per run, so for 3 runs there should be 12,000 IEM events, a factor 3 more.

So maybe something is off, but I'd be inclined to merge now, to allow everyone using this and continue debugging in master.

[cdeil]

I'm merging this in now.

@robertazanin and all - Please try it out and report any issues you find, or open a new issue or PR with suggestions to make this class better.

@adonath - You were right that it's slow. Especially for the 1DC diffuse model cube, which is 800 MB, an evaluate takes almost a second, and I presume that most of that time is spent in interpolator init, i.e. could be avoided by caching that. Making a cutout of the diffuse model is one way to improve speed for now.