sky map kde as stand alone utility

[grburgess]

First, thanks for such a great open source package!

I have a lot of various sky localization algorithms that produce posterior samples which I would like to convert to MOCs / healpix files. The SkymapKDE classes do an excellent job of this, but i don't want to make my users have to install large dependencies in general. Its not a huge deal, but if those utilities were split of in a separate package for more general use, it would be great. Is this a possibility?

[lpsinger]

The KDE stuff requires the two big C library dependencies, gsl and chealpix. But it doesn't require the many of the large Python dependencies, such as LALSuite. I can make many of these other dependencies optional using setuptools extras.

Dependency graph right now

$ poetry show --tree ligo.skymap
ligo.skymap 0.4.0 Tools for reading, writing, manipulating, and making LIGO and Virgo sky maps
├── astroplan >=0.5
│   ├── astropy >=1.3 
│   │   └── numpy >=1.13 
│   ├── numpy >=1.10 (circular dependency aborted here)
│   ├── pytz * 
│   └── six * 
├── astropy >=3.1,<4.0.1 || >4.0.1
│   └── numpy >=1.13 
├── astropy-healpix >=0.3
├── h5py *
│   ├── numpy >=1.7 
│   └── six * 
├── healpy *
│   ├── astropy * 
│   │   └── numpy >=1.13 
│   ├── matplotlib * 
│   │   ├── cycler >=0.10 
│   │   │   └── six * 
│   │   ├── kiwisolver >=1.0.1 
│   │   ├── numpy >=1.15 (circular dependency aborted here)
│   │   ├── pillow >=6.2.0 
│   │   ├── pyparsing >=2.0.3,<2.0.4 || >2.0.4,<2.1.2 || >2.1.2,<2.1.6 || >2.1.6 
│   │   └── python-dateutil >=2.1 
│   │       └── six >=1.5 (circular dependency aborted here)
│   ├── numpy >=1.13 (circular dependency aborted here)
│   ├── scipy * 
│   │   └── numpy >=1.14.5 (circular dependency aborted here)
│   └── six * (circular dependency aborted here)
├── importlib-resources *
│   └── zipp >=0.4 
├── lalsuite >=6.53
│   ├── importlib-resources * 
│   │   └── zipp >=0.4 
│   ├── ligo-segments * 
│   │   └── six * 
│   ├── lscsoft-glue * 
│   │   ├── ligo-segments * (circular dependency aborted here)
│   │   ├── numpy * 
│   │   ├── pyopenssl * 
│   │   │   ├── cryptography >=2.8 
│   │   │   │   ├── cffi >=1.8,<1.11.3 || >1.11.3 
│   │   │   │   │   └── pycparser * 
│   │   │   │   └── six >=1.4.1 (circular dependency aborted here)
│   │   │   └── six >=1.5.2 (circular dependency aborted here)
│   │   └── six * (circular dependency aborted here)
│   ├── matplotlib * 
│   │   ├── cycler >=0.10 
│   │   │   └── six * (circular dependency aborted here)
│   │   ├── kiwisolver >=1.0.1 
│   │   ├── numpy >=1.15 (circular dependency aborted here)
│   │   ├── pillow >=6.2.0 
│   │   ├── pyparsing >=2.0.3,<2.0.4 || >2.0.4,<2.1.2 || >2.1.2,<2.1.6 || >2.1.6 
│   │   └── python-dateutil >=2.1 
│   │       └── six >=1.5 (circular dependency aborted here)
│   ├── numpy >=1.7 (circular dependency aborted here)
│   ├── python-dateutil * (circular dependency aborted here)
│   └── scipy * 
│       └── numpy >=1.14.5 (circular dependency aborted here)
├── ligo-gracedb >=2.0.1
│   ├── cryptography >=1.7.2 
│   │   ├── cffi >=1.8,<1.11.3 || >1.11.3 
│   │   │   └── pycparser * 
│   │   └── six >=1.4.1 
│   ├── future >=0.15.0 
│   └── six >=1.9.0 (circular dependency aborted here)
├── ligo-segments >=1.2.0
│   └── six * 
├── matplotlib >=3.0.2
│   ├── cycler >=0.10 
│   │   └── six * 
│   ├── kiwisolver >=1.0.1 
│   ├── numpy >=1.15 
│   ├── pillow >=6.2.0 
│   ├── pyparsing >=2.0.3,<2.0.4 || >2.0.4,<2.1.2 || >2.1.2,<2.1.6 || >2.1.6 
│   └── python-dateutil >=2.1 
│       └── six >=1.5 (circular dependency aborted here)
├── networkx *
│   └── decorator >=4.3.0 
├── numpy >=1.14,<1.15.3 || >1.15.3
├── pillow >=2.5.0
├── ptemcee *
│   └── numpy * 
├── python-ligo-lw *
│   ├── ligo-segments * 
│   │   └── six * 
│   ├── lscsoft-glue * 
│   │   ├── ligo-segments * (circular dependency aborted here)
│   │   ├── numpy * 
│   │   ├── pyopenssl * 
│   │   │   ├── cryptography >=2.8 
│   │   │   │   ├── cffi >=1.8,<1.11.3 || >1.11.3 
│   │   │   │   │   └── pycparser * 
│   │   │   │   └── six >=1.4.1 (circular dependency aborted here)
│   │   │   └── six >=1.5.2 (circular dependency aborted here)
│   │   └── six * (circular dependency aborted here)
│   ├── numpy * (circular dependency aborted here)
│   ├── python-dateutil * 
│   │   └── six >=1.5 (circular dependency aborted here)
│   ├── pyyaml * 
│   └── six * (circular dependency aborted here)
├── pytz *
├── reproject >=0.3.2
│   ├── astropy >=3.2 
│   │   └── numpy >=1.13 
│   ├── astropy-healpix >=0.2 
│   ├── numpy >=1.13 (circular dependency aborted here)
│   └── scipy >=1.1 
│       └── numpy >=1.14.5 (circular dependency aborted here)
├── scipy >=0.14
│   └── numpy >=1.14.5 
└── tqdm >=4.27.0

Hypothetical dependency graph after isolating extras

$ poetry show --tree ligo.skymap
ligo.skymap 0.4.0 Tools for reading, writing, manipulating, and making LIGO and Virgo sky maps
├── astropy >=3.1,<4.0.1 || >4.0.1
│   └── numpy >=1.13 
├── astropy-healpix >=0.3
├── h5py *
│   ├── numpy >=1.7 
│   └── six * 
├── healpy *
│   ├── astropy * 
│   │   └── numpy >=1.13 
│   ├── matplotlib * 
│   │   ├── cycler >=0.10 
│   │   │   └── six * 
│   │   ├── kiwisolver >=1.0.1 
│   │   ├── numpy >=1.15 (circular dependency aborted here)
│   │   ├── pillow >=6.2.0 
│   │   ├── pyparsing >=2.0.3,<2.0.4 || >2.0.4,<2.1.2 || >2.1.2,<2.1.6 || >2.1.6 
│   │   └── python-dateutil >=2.1 
│   │       └── six >=1.5 (circular dependency aborted here)
│   ├── numpy >=1.13 (circular dependency aborted here)
│   ├── scipy * 
│   │   └── numpy >=1.14.5 (circular dependency aborted here)
│   └── six * (circular dependency aborted here)
├── importlib-resources *
│   └── zipp >=0.4 
├── matplotlib >=3.0.2
│   ├── cycler >=0.10 
│   │   └── six * 
│   ├── kiwisolver >=1.0.1 
│   ├── numpy >=1.15 
│   ├── pillow >=6.2.0 
│   ├── pyparsing >=2.0.3,<2.0.4 || >2.0.4,<2.1.2 || >2.1.2,<2.1.6 || >2.1.6 
│   └── python-dateutil >=2.1 
│       └── six >=1.5 (circular dependency aborted here)
├── numpy >=1.14,<1.15.3 || >1.15.3
├── reproject >=0.3.2
│   ├── astropy >=3.2 
│   │   └── numpy >=1.13 
│   ├── astropy-healpix >=0.2 
│   ├── numpy >=1.13 (circular dependency aborted here)
│   └── scipy >=1.1 
│       └── numpy >=1.14.5 (circular dependency aborted here)
├── scipy >=0.14
│   └── numpy >=1.14.5 
└── tqdm >=4.27.0

[grburgess]

That would be great. In the end, the package is not giant, but this would be very convenient when having to deploy on a lot of containers. If this is not too much trouble...

[lpsinger]

Update: I have opened a merge request to move most of the heavy dependencies to a setuptools extra (https://git.ligo.org/lscsoft/ligo.skymap/-/merge_requests/214), but there is a hidden dependency on the one biggest dependency, lalsuite, that is proving difficult (see https://git.ligo.org/kipp.cannon/python-ligo-lw/-/merge_requests/24).

[grburgess]

uff, did not realize this would be a huge issue. Is it worth pursuing? I can try to help some tomorrow (EU time) though I'm not familiar with the code base in any detail.

[lpsinger]

I'm willing to push on this. The changes to my code are done and ready to go, so it's only a question of whether and how soon the LALSuite issue can be resolved.

There is a long-standing issue that there is a circular runtime dependency between lalsuite (which is a C project with Python SWIG bindings, as of the past few years available in both pip and conda) and python-ligo-lw, an I/O support library for an important GW tabular data format.

For ligo.skymap, you only need lalsuite itself if you are actually running BAYESTAR to localize gravitational-wave signals. However, it's a little trickier to get the python-ligo-lw imports to be optional because they are so pervasive.

There's good reason to have both BAYESTAR and the KDE sky map postprocessing in the same project: they both rely on a Python C extension that must link against GSL.

[grburgess]

Then it is worth it as the skymap library yields beautiful IPN plots
three_det_sky_map.pdf

[lpsinger]

That is a very beautiful plot. I also really like the Ghostbusters color scheme.

[grburgess]

BTW: I do not see a software DOI or BibTeX entry for citing the package. Have you a preference?

[lpsinger]

For now, you can cite https://doi.org/10.1103/PhysRevD.93.024013, although that really describes BAYESTAR, not ligo.skymap.

[grburgess]

Cool, thanks!