Using 'envi.open' on ENVI .sli file alters endmember spectra values (repost)

[agraham9966]

Hi Thomas,

For those who don't know, this issue has been re-posted here, since I mistakenly put this issue up on the wrong repo!

I am experiencing an issue of which I cannot conclude is an issue with a particular envi .sli file (and associated header) itself, or with something in your code using 'envi.open'.

I have provided the following files in the attached zip folder:

• 'cuprite-endmembers.sli' and .hdr: 46 endmembers total in file - EMs extracted from cuprite image from 2006
• -'envi-out-EM24.sli' and .hdr : EM24, manually exported using ENVI from 'cuprite-endmembers.sli'

The issue:
When using 'envi.open' on 'cuprite-endmembers.sli', endmember spectral values are altered or sorted at different positions at particular spectral regions. These values are not altered or sorted however, when comparing the same endmembers opened inside of ENVI (envi reads the endmember DN values correctly and differently than envi.open in spectral python).
An example here is 'EM24' (at row index position 12 of the cuprite-endmembers.sli endmember array). If I save that same endmember out from ENVI (which I have done for you and put in the zip file 'envi-out-EM24.sli') envi.open does not do anything to the spectral values. This can be verified by plotting:

In the above plot, EM24 in row 12 of 'cuprite-endmembers.sli' shows variations at 0.6554-0.6749 micrometers (or index 30-34. sli_file.spectra[12][30:35])

In summary, ENVI is able to open 'cuprite-endmembers.sli' properly without any alteration to the spectral values. But with this particular sli file, envi.open does something to the spectra, and I cannot seem to figure out the source of the issue. The only workaround is to open cuprite-endmembers.sli in envi and save the endmembers back out. Then envi.open works. But I would like to know the cause of this issue if possible.

Much thanks and sorry if the description is confusing. I found this issue hard to explain.
cuprite-endmembers.zip

[tboggs]

The issue you are encountering is due to the fact that the bands in the spectral library are not monotonically increasing in wavelength. The common reason for this is that many hyperspectral imagers use multiple detectors with slightly overlapping spectral ranges. In the spectral module, bands are not automatically sorted when reading the library (this is intentional to allow the user to decide how they want to handle the overlapping bands). It appears that ENVI automatically sorts the bands so that when you saved the spectral library from ENVI, the overlapping bands were re-ordered.

For example, in the original file (cuprite-endmembers.hdr), there is the following sequence of wavelengths (note that they do not increase monotonically):

... 0.647974 , 0.657765 , 0.667561 , 0.655292 , 0.665099 , 0.674901, ...

whereas in the library saved from ENVI (envi-out-EM24.hdr), the same set of bands are in increasing order, as follows:

... 0.647974 , 0.655292 , 0.657765 , 0.665099 , 0.667561 , 0.674901, ...

If I plot the bands from both header files in the order they appear in the header file, I get the following, which is consistent with your plot (I'm highlighting the region around the first discrepancy you circled in your figure):

import spectral as spy
import numpy as np
import matplotlib.pyplot as plt

lib1 = spy.envi.open('cuprite-endmembers.hdr')
lib2 = spy.envi.open('envi-out-EM24.hdr')
bands = np.arange(len(lib1.bands.centers))

plt.figure(figsize=(8, 4))
plt.plot(bands, lib1.spectra[12], 'k.-', label='original')
plt.plot(bands, lib2.spectra[0], 'r.--', label='ENVI')
plt.xlim(26, 36)
plt.ylim(0.275, 0.3)
plt.legend()
plt.grid()
plt.xlabel('band index')

Here is the same plot after sorting the bands from cuprite-endmembers.hdr (note that the two spectra are identical):

plt.figure(figsize=(8, 4))
plt.plot(bands, lib1.spectra[12][np.argsort(lib1.bands.centers)],
         'k.-', label='original (sorted)')
plt.plot(bands, lib2.spectra[0], 'r.--', label='ENVI')
plt.xlim(26, 36)
plt.ylim(0.275, 0.3)
plt.legend()
plt.grid()
plt.xlabel('band index')

Finally, to really see what is going on in the unsorted file, it is instructive to reproduce the first figure, plotting each spectrum as a line series with wavelength as the x coordinate:

plt.figure(figsize=(8, 4))
plt.plot(lib1.bands.centers, lib1.spectra[12], 'k.-', label='original')
plt.plot(lib2.bands.centers, lib2.spectra[0], 'r.--', label='ENVI')
plt.xlim(lib1.bands.centers[26], lib1.bands.centers[36])
plt.ylim(0.275, 0.3)
plt.legend()
plt.grid()
plt.xlabel('wavelength (um)')

Just to clarify with respect to the title of this issue, spectral does not perform any alteration of the original bands. It appears that ENVI is altering the bands when it reads the library and the altered bands are then saved by ENVI to the new file.

---

NOTE: Since sorting and filtering bands in spectral libraries is a fairly common task, I've opened issue #98 to add a filter method to the SpectralLibrary class to make it easier to perform those tasks.

---

[agraham9966]

Hi Thomas,

Amazing catch. I definitely did not notice, nor even consider the bands were not sorted! Its also good to know that envi actually does some sorting of the wavelengths.
Now all is right in the universe. Thanks for your help on this matter- and great work with spectral python.
Best,
Alex