Upgrade to use USGS Spectral Library Version 7

[lewismc]

Spectral Lib 7 is now available
https://speclab.cr.usgs.gov/spectral-lib.html

[lewismc]

I can't seem to find the equivalent version of ftp://ftpext.cr.usgs.gov/pub/cr/co/denver/speclab/pub/spectral.library/splib06.library/Convolved.libraries/s06av95a_envi.hdr

[ghost]

It looks like they have not released an AVIRIS/ENVI version of the library, but that's okay, we'll just have to use Spectral to convert it by doing something similar or identical to pycoal.mineral.AsterConversion.

Rather than making COAL users do it, we might consider converting it ourselves and then hosting it since it is (I presume) freely distributable.

[lewismc]

There will be a requirement to update the Website documentation here as well.

[lewismc]

Hi @bdegley4789 how are things coming along? Anything I can help out with over and above out meeting last week? Do you have a pull request I can review? Thanks

[bdegley4789]

@lewismc the last few days, I was working on fixing the errors I was having with gdal, which I solved. And I think I figured out what was causing the rotational bug with qgis. I added these to the installation instructions and made a pull request on the website for that. On this issue, I've looked at the links you sent but not much other than that. This will be my new priority now that the gdal stuff is out of the way. I'll get something ready for this issue either in the next day or two at least before our next meeting

[bdegley4789]

@lewismc Do you know where the Aster data files are located in usgs spectral library version 7? I keep getting an error that they aren't found when I set the path to several different locations in usgs_splib07. I set up this code to use pycoal.mineral.AsterConversion and generates s07av95a_envi.hdr and s07av95a_envi.sli

[bdegley4789]

Current code
https://github.com/bdegley4789/pycoal/blob/master/examples/example_splib07.py

[bdegley4789]

If pycoal.mineral.AsterConversion doesn't work I think we could also use Harris Geospatial Spectral Library Builder

[ghost]

@bdegley4789 Since @lewismc hasn't responded yet, I recommend you have a look at the source code of pycoal.mineral.AsterConversion. This is an experimental method expected to work only on the ASTER spectral library. The USGS Spectral Library Version 7 uses a similar but distinct format. You may also need to review spectral.AsterDatabase.create_envi_spectral_library() to see how to go from an array of samples to a resampled library. The proprietary ENVI software from Harris Geospatial would also certainly work, but if you have a look at the licensing you will see why we haven't used it.

[bdegley4789]

@browtayl Thanks. Lewis didn't responded here but we discussed this in the video chat meeting we had last Thursday. He had similar recommendations, I'll take a look at spectral.AsterDatabase.create_envi_spectral_library() I didn't know about that

[lewismc]

How are things coming along @bdegley4789 ?

[lewismc]

PING @bdegley4789

[bdegley4789]

@lewismc sorry for the late reply I had a project in another class yesterday. I am working on this by creating another convert class like we discussed. With the AsterConversion it used .db file. The usgs Spectral Version 7 is a bunch of .txt files I'm a little confused on how to work with this difference. I think I can still work around that to create the .sli and .hdr files but Im not sure if the contents with in them will be whats needed

[lewismc]

@bdegley4789 thanks, the information you need to get your started can be found at http://www.spectralpython.net/libraries.html. This will lead you to working with the SpectralLibrary Class. The output of the save function in this Class is two files: fileBaseName.hdr and fileBaseName.sli.
Any issues, let me know and we can work it through.

[lewismc]

@bdegley4789 @thomkenn please keep me updated this next week. I look forward to us making progress as a team.

[bdegley4789]

Sounds like a plan, Ill keep you updated

[bdegley4789]

Ok I worked on this issue for awhile last night and today. I found a way to get the .sli and .hdr files of all the Spectra in Spectral Library Version 7.

If I can change all the files in Spectral Library Version 7 to the format of the ASTER files and move them to one directory and then run
the AsterConversion() function in mineral.py in my most recent pull request to pycoal it does correctly output the .hdr and .sli files.

I have tested this with one Spectral Library 7 .txt file which I manually edited to have the same format as the ASTER text files. You do this by running example_splib07.py in the example directory and it will output a database file, .sli and .hdr for the one Spectral 7 file I formatted to match ASTER file format. And it appears correctly in the .hdr file

There is a slight problem I think we can work around by doing this method
Aster Files require:
Name
Particle Size
Sample No.
Wavelength
Origin
Measurement
Two Columns and the Units of Measurement

-Spectral Version 7 is all in micrometers same as ASTER
-The name will be the name of the Spectral Version 7 mineral
-On the other values though Spectral Version 7 doesn’t give that information.
However, in mineral.py we are only sorting the x column so whatever value I put in for the y column and those other values should be irrelevant.
If this works for you @lewismc , I’ll move forward and write the code to convert all the Spectral Text files to match Aster file format and then get the .hdr and .sli files for the entire spectral version 7 library

I believe this method can also be applied to solving issues #134 and #119 since those spectra are .txt files of a different format.

[bdegley4789]

Ok, I went ahead with the method described above.
I have the envi .hdr and .sli files generated for the entire USGS Spectral Library Version 7.
I have also started to run them in a mineral classification of a site to see how it will look using Spectral Version 7. I pushed all the code I wrote to do this in my most recent pull request and staged the .hdr and .sli files in https://drive.google.com/drive/folders/1YVhdLxvrZE3eC97OEXathLMJRgWt8haO

This next week I will create two more convert functions to work on the EcoSIS and ECOSTRESS spectral libraries

[lewismc]

Excellent work @bdegley4789 this is done done via #146