Download RSR data in vain

[adybbroe]

Code Sample, a minimal, complete, and verifiable piece of code

import numpy as np
from pyspectral.rayleigh import Rayleigh
from pyspectral.utils import debug_on
debug_on()

msi = Rayleigh('Sentinel-2C', 'msi')

sunz = np.array([[32., 40.], [31., 41.]])
satz = np.array([[45., 20.], [46., 21.]])
ssadiff = np.array([[110, 170], [120, 180]])

refl_cor_red = msi.get_reflectance(sunz, satz, ssadiff, 'B02')

Problem description

In the case that a new satellite not yet supported with the spectral responses in PySpectral, a download will be attempted when a band name is provided to the get_reflectance method of the Rayleigh object.

To avoid trying to download RSR data in vain (cases where even after download the rsr data for the requested sensor and platform will anyway not be there) one needs to add a check if the downloaded RSR data are the latest. This kind of checking is already implemented for the atm correction LUTs.

Expected Output

So, currently with the code example above a download is attempted, even if RSR data are the most recent. As seen in the IOError below it attempts to download the data since there is no file for S2C. But, if you repeat the run it will do the same again.

Instead, it should detect that the latest RSR data have been downloaded and are available, and if the S2C RSR data are not part of the data, it can throw an IOError as is done.

Actual Result, Traceback if applicable

[INFO: 2018-06-01 13:43:43 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2018-06-01 13:43:43 : pyspectral.rayleigh] LUT filename: /home/a000680/.local/share/pyspectral/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[DEBUG: 2018-06-01 13:43:43 : pyspectral.rsr_reader] Filename: /home/a000680/.local/share/pyspectral/rsr_msi_Sentinel-2C.h5
[WARNING: 2018-06-01 13:43:43 : pyspectral.rsr_reader] No rsr file /home/a000680/.local/share/pyspectral/rsr_msi_Sentinel-2C.h5 on disk
[INFO: 2018-06-01 13:43:43 : pyspectral.rsr_reader] Will download from internet...
[DEBUG: 2018-06-01 13:43:43 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org
[DEBUG: 2018-06-01 13:43:44 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1205138/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2798376
2798376it [00:01, 1678973.88it/s]
[ERROR: 2018-06-01 13:43:46 : pyspectral.rayleigh] No spectral responses for this platform and sensor: Sentinel-2C msi
Traceback (most recent call last):
  File "/home/a000680/usr/src/pyspectral/pyspectral/rayleigh.py", line 138, in get_effective_wavelength
    rsr = RelativeSpectralResponse(self.platform_name, self.sensor)
  File "/home/a000680/usr/src/pyspectral/pyspectral/rsr_reader.py", line 94, in __init__
    raise IOError(errmsg)
IOError: pyspectral RSR file does not exist! Filename = /home/a000680/.local/share/pyspectral/rsr_msi_Sentinel-2C.h5
Files matching instrument and satellite platform: []

Versions of Python, package at hand and relevant dependencies

Python 2.7.5
Pyspectral 0.7.2 - latest master - 3937586

Thank you for reporting an issue !

[adybbroe]

Ok, I just ran this with the latest master branch of pyspectral. This should now be solved. The output running the above is:

[INFO: 2018-12-03 16:26:59 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2018-12-03 16:26:59 : pyspectral.rayleigh] LUT filename: /home/a000680/data/pyspectral/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[DEBUG: 2018-12-03 16:26:59 : pyspectral.rsr_reader] Filename: /home/a000680/data/pyspectral/rsr_msi_Sentinel-2C.h5
[WARNING: 2018-12-03 16:26:59 : pyspectral.rsr_reader] No rsr file /home/a000680/data/pyspectral/rsr_msi_Sentinel-2C.h5 on disk
[INFO: 2018-12-03 16:26:59 : pyspectral.rsr_reader] RSR data up to date, so seems there is no support for this platform and sensor
[ERROR: 2018-12-03 16:26:59 : pyspectral.rayleigh] No spectral responses for this platform and sensor: Sentinel-2C msi
Traceback (most recent call last):
  File "pyspectral/rayleigh.py", line 166, in get_effective_wavelength
    rsr = RelativeSpectralResponse(self.platform_name, self.sensor)
  File "pyspectral/rsr_reader.py", line 93, in __init__
    raise IOError(errmsg)
IOError: pyspectral RSR file does not exist! Filename = /home/a000680/data/pyspectral/rsr_msi_Sentinel-2C.h5
Files matching instrument and satellite platform: []
[ERROR: 2018-12-03 16:26:59 : pyspectral.rayleigh] Can't get effective wavelength for band B02 on platform Sentinel-2C and sensor msi