Undocumented dependency packages

[avalentino]

There are three packages that are used by satpy but are not listed in the (optional) requirements list in setup.py:

• pydecorate
• glymur
• pyninjotiff

In particular pydecorate seems to be used during unit testing.

To Reproduce

$ python3 setup.py test

Expected behavior
All tests are executed without errors

Actual results
4 test failures

======================================================================
ERROR: test_add_decorate_basic_l (satpy.tests.test_writers.TestOverlays)
Test basic add_decorate usage with L data.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/satpy/satpy/tests/test_writers.py", line 599, in setUp
    self.dw = self.dec_writer.start()
  File "/usr/lib/python3.7/unittest/mock.py", line 1390, in start
    result = self.__enter__()
  File "/usr/lib/python3.7/unittest/mock.py", line 1243, in __enter__
    self.target = self.getter()
  File "/usr/lib/python3.7/unittest/mock.py", line 1413, in <lambda>
    getter = lambda: _importer(target)
  File "/usr/lib/python3.7/unittest/mock.py", line 1096, in _importer
    thing = __import__(import_path)
ModuleNotFoundError: No module named 'pydecorate'

======================================================================
ERROR: test_add_decorate_basic_rgb (satpy.tests.test_writers.TestOverlays)
Test basic add_decorate usage with RGB data.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/satpy/satpy/tests/test_writers.py", line 599, in setUp
    self.dw = self.dec_writer.start()
  File "/usr/lib/python3.7/unittest/mock.py", line 1390, in start
    result = self.__enter__()
  File "/usr/lib/python3.7/unittest/mock.py", line 1243, in __enter__
    self.target = self.getter()
  File "/usr/lib/python3.7/unittest/mock.py", line 1413, in <lambda>
    getter = lambda: _importer(target)
  File "/usr/lib/python3.7/unittest/mock.py", line 1096, in _importer
    thing = __import__(import_path)
ModuleNotFoundError: No module named 'pydecorate'

======================================================================
ERROR: test_add_overlay_basic_l (satpy.tests.test_writers.TestOverlays)
Test basic add_overlay usage with L data.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/satpy/satpy/tests/test_writers.py", line 599, in setUp
    self.dw = self.dec_writer.start()
  File "/usr/lib/python3.7/unittest/mock.py", line 1390, in start
    result = self.__enter__()
  File "/usr/lib/python3.7/unittest/mock.py", line 1243, in __enter__
    self.target = self.getter()
  File "/usr/lib/python3.7/unittest/mock.py", line 1413, in <lambda>
    getter = lambda: _importer(target)
  File "/usr/lib/python3.7/unittest/mock.py", line 1096, in _importer
    thing = __import__(import_path)
ModuleNotFoundError: No module named 'pydecorate'

======================================================================
ERROR: test_add_overlay_basic_rgb (satpy.tests.test_writers.TestOverlays)
Test basic add_overlay usage with RGB data.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/satpy/satpy/tests/test_writers.py", line 599, in setUp
    self.dw = self.dec_writer.start()
  File "/usr/lib/python3.7/unittest/mock.py", line 1390, in start
    result = self.__enter__()
  File "/usr/lib/python3.7/unittest/mock.py", line 1243, in __enter__
    self.target = self.getter()
  File "/usr/lib/python3.7/unittest/mock.py", line 1413, in <lambda>
    getter = lambda: _importer(target)
  File "/usr/lib/python3.7/unittest/mock.py", line 1096, in _importer
    thing = __import__(import_path)
ModuleNotFoundError: No module named 'pydecorate'

Environment Info:

• Linux (debian sid)
• SatPy Version: 0.11.0
• PyResample Version: 1.10.3
• Readers and writers dependencies (when relevant):
  Readers
  =======
  Environment variable PYGAC_CONFIG_FILE not set!
  abi_l1b: ok
  abi_l1b_scmi: ok
  acspo: ok
  ahi_hrit: ok
  ahi_hsd: ok
  amsr2_l1b: ok
  avhrr_l1b_aapp: ok
  avhrr_l1b_eps: ok
  avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'ConfigParser')
  avhrr_l1b_hrpt: ok
  caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
  clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
  electrol_hrit: ok
  fci_l1c_fdhsi: ok
  generic_image: ok
  geocat: ok
  ghrsst_l3c_sst: ok
  goes-imager_hrit: ok
  goes-imager_nc: ok
  grib: ok
  iasi_l2: ok
  jami_hrit: ok
  li_l2: ok
  maia: ok
  modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
  msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'glymur')
  mtsat2-imager_hrit: ok
  nucaps: ok
  nwcsaf-geo: ok
  nwcsaf-pps_nc: ok
  olci_l1b: ok
  olci_l2: ok
  omps_edr: ok
  sar-c_safe: ok
  scatsat1_l2b: ok
  seviri_l1b_hrit: ok
  seviri_l1b_native: ok
  seviri_l1b_nc: ok
  slstr_l1b: ok
  viirs_compact: ok
  viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
  viirs_l1b: ok
  viirs_sdr: ok

Writers

cf: ok
geotiff: ok
mitiff: ok
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
scmi: ok
simple_image: ok

[djhoese]

Thanks for pointing this out. The pydecorate dependency should probably be listed in the optional dependencies. Glymur I think is only used by that one reader which may not be working yet, but you are right it should be added to the optionals. The NinJo support is also iffy right now but maybe @loreclem could include it in future PRs where that is fixed.

If you want to make a PR for this I wouldn't mind that. Otherwise I'll try to get to it later this week or during travel the week after. I should probably rename the optional dependencies to use the new reader names too.

[avalentino]

Thanks @djhoese. If you plan to rename readers then probably it is better you also address the dependency declaration directly.

Just let me know if you plan to remove some of that dependences.
We are currently working on a debian package for satpy and all its dependencies.
If some dependency is going to disappear then there is no need we put effort to make debian packages for it.

[djhoese]

@avalentino The reader names have been changed in the most recent 0.11 release, the optional dependency "extras" have not been updated to reflect all of these changes.

How do you plan on handling all of these optional dependencies? Do you plan on only supporting a limited set of readers/writers in the default debian package?

[avalentino]

@djhoese the current status of the debian package can be found in [1].
In particular in [2] you can find the list of hard dependencies and recommended packages:

Depends: python3-dask,
         python3-grib,
         python3-hdf4,
         python3-numpy (>= 1:1.13),
         python3-pil,
         python3-pykdtree,
         python3-pyproj,
         python3-pyresample (>= 1.10.3),
         python3-scipy,
         python3-six,
         python3-trollimage,
         python3-trollsift,
         python3-xarray,
         python3-yaml
Recommends: python3-behave,
            python3-bs4,
            python3-gdal,
            python3-geotiepoints,
            python3-h5netcdf,
            python3-h5py,
            python3-imageio,
            python3-libtiff (>= 0.4.2),
            python3-netcdf4,
            python3-pycoast,
            python3-pygac,
            python3-pyorbital (>= 1.3.1),
            python3-pyspectral,
            python3-rasterio,
            python3-requests,
            python3-tables,
            python3-trollsched

All the above packages are already available in debian (or are in the queue to be uploaded in the main archive).

At the moment i only plan to add:

• pydecorate
• glymur
• pyninjotiff

that will be added to the list of recommended packages.

Actually some of the less important packages that are currently in recommended could me moved to the "suggested" section which is a weaker kind of relation.
Maybe you can provide some hint about that.
Probably pytables is one of the packages that could be moved to "suggested" but surely there are others.

[1] https://salsa.debian.org/debian-gis-team/satpy
[2] https://salsa.debian.org/debian-gis-team/satpy/blob/master/debian/control

[avalentino]

ah ... forgot to mention that also

• bs4
• requests
• pytables

are used somewhere but not declared in the serup .py.

[djhoese]

I think pydecorate would be good for recommended.

I personally think you should not include pyninjotiff at all. It is currently not compatible with satpy as it is right now. Or at least it hasn't been released that way yet. @loreclem may have more info on that.

As for "glymur" I'm not familiar with the package, but if it is only used in one of the readers it isn't a big deal if it isn't included.

bs4 and requests I think are optional dependencies for pyspectral (@adybbroe ?). It looks like pytables is used in the viirs_compact reader (@mraspaud ?).

[djhoese]

I'm looking through old issues and rediscovered this one. One is the current status of the debian satpy package?

With my earlier question in this thread, I was really asking: what is your philosophy on Satpy's optional dependencies as they apply to the debian package? Satpy will have a lot of dependencies if you included them all. Are you going to include as many as are available in debian right now? Or are you going to limit the dependencies to those needed by the most popular readers and writers in Satpy?

[avalentino]

Dear @djhoese currently Debian in in freeze in preparation of the new stable release, Debian 11, that should be available in few weeks (months?).
The satpy version in Debian 11 will be satpy v0.25.1.

After the release of Debian 11 I will start to work on the packaging of latest version of satpy.
I have already spotted that there are some new dependencies (e.g. donfig and pooch) which are still not available in Debian. I will need to create new packages for them before pushing the new version of satpy.

Regarding the "philosophy" for dependency management I try to follow as close as possible what is specified in the upstream setup.py:

• packages listed in the setup.py "requires" are marked as strict dependencies (Depends)
• packages listed in the setup.py "extras" are marked as "Recommended" packages (assuming that they are available as debian packages).

Sometimes I mark as "Suggested" (weaker relation) extra dependencies that are particularly large (or trigger a large number of indirect dependencies).
In any case I try to include all the possible dependencies during the build phase so to have the possibility to make a complete test.

[djhoese]

Thanks for the info. I've created #1673 to cleanup some of this.