New data reduction procedure for Spectralon calibration imagery #289
Comments
|
The VNIR is currently completely out of service. A recent software update to implement x-axis scanning broke the data writing, LemnaTec is working on it now. I have the script ready for the calibration run so I'm hoping it can happen soon. |
|
@smarshall-bmr is VNIR back up and running? |
|
Now up and running in predawn mode for calibration |
|
VNIR is writing data again. We are working on fixing this 10,000 line scan limit right now, calibration is up next on my priority list. |
|
@solmazhajmohammadi @dlebauer The recent calibration images brought to my attention that the SWIR naming convention has diverged from the VNIR. SWIR files now have an extra date string (right after the UUID) in their names, and also are missing an underscore before the english suffix. Was there a good reason for changing the naming convention? It would be helpful if SWIR used the same convention as VNIR. Otherwise we need to re-write the extractors to handle both conventions and be backwards compatible, etc. |
|
IMHO the best way to proceed here is to remove the extra date string recently introduced into (some of) the SWIR filenames, i.e., "2017_04_15_12_05_11" in the above example, in all but the metadata.json file. That way the VNIR and SWIR will have the same naming convention. Otherwise we need to re-write the filename handling in the SWIR workflow to handle both conventions. @dlebauer should we wait for a sysadmin to change the names or start to re-write the SWIR filename handling and accept both old and new conventions? |
|
Wait on sysadmin. If you send me the appropriate command I can run it.
…On Mon, Apr 24, 2017 at 5:56 PM Charlie Zender ***@***.***> wrote:
IMHO the best way to proceed here is to remove the extra date string
recently introduced into (some of) the SWIR filenames, i.e.,
"2017_04_15_12_05_11" in the above example, in all but the metadata.json
file. That way the VNIR and SWIR will have the same naming convention.
Otherwise we need to re-write the filename handling in the SWIR workflow to
handle both conventions. @dlebauer <https://github.com/dlebauer> should
we wait for a sysadmin to change the names or start to re-write the SWIR
filename handling and accept both old and new conventions?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#289 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAcX57V9G7OXzf4-bHQRmI12DUBlHCjDks5rzSiggaJpZM4MYdQC>
.
|
|
This should work for a sufficiently priveleged user: yyyymmdd='2017-04-15'
drc_top="/projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/${yyyymmdd}"
cd ${drc_top}
for drc_sub in `ls -d ${yyyymmdd}*` ; do
echo "Renaming in directory ${drc_sub}..."
cd ${drc_top}/${drc_sub}
for fl in `ls *raw` ; do
dt_sng=`expr match "${fl}" '.*\([0-9][0-9][0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]\)\.*'`
uuid_sng=${fl:0:36}
done # !fl
echo "uuid=${uuid_sng}, dt=${dt_sng}"
for sfx in frameIndex.txt image.jpg raw raw.hdr settings.txt ; do
mv_cmd="/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
echo "/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
eval ${mv_cmd}
done # !sfx
done # !drc_subAs shown it just does the calibration day. However, all SWIR data since 2016-11-08 appear to use this new convention. @solmazhajmohammadi @LTBen is there a reason for the new convention? If not will Lemnatec please revert to the original (and current VNIR) convention? I do not understand how the hyperspectral extractor ever worked on SWIR files since 2016-11-08. AFAICT it should have reported an error. @max-zilla, I gather that the SWIR extractor was not run this whole time? I hope if it was run that it failed and gave a reasonable error message... Running the above will suffice to unblock progress on calibration. To rename every SWIR file since 2016-11-08, auto-generate the dates with |
|
@czender I think everything worked. I added looping over dates ( #!/bin/bash
cd /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/
for d in `/bin/ls -d 2016-11-0[89] 2016-11-[123]? 2016-12-?? 2017-??-??`; do
yyyymmdd=$d
drc_top="/projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/${yyyymmdd}"
cd ${drc_top}
for drc_sub in `/bin/ls -d ${yyyymmdd}*` ; do
echo "Renaming in directory ${drc_sub}..."
cd ${drc_top}/${drc_sub}
for fl in `/bin/ls *raw` ; do
dt_sng=`expr match "${fl}" '.*\([0-9][0-9][0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]\)\.*'`
uuid_sng=${fl:0:36}
done # !fl
echo "uuid=${uuid_sng}, dt=${dt_sng}"
for sfx in frameIndex.txt image.jpg raw raw.hdr settings.txt ; do
mv_cmd="/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
echo "/bin/mv ${uuid_sng}_${dt_sng}${sfx} ${uuid_sng}_${sfx}"
eval ${mv_cmd}
done # !sfx
done # !drc_sub
done # !dLet me know if it looks okay. |
|
@LTBen and @markus-radermacher-lemnatec can you please confirm that this will be fixed? |
|
The reason for different naming lays in different headwall software. LemnaTec software is only adding prefix to the file name. |
ghost
removed
the
|
Next step: @max-zilla will run SWIR extractor this week. April 15 is the priority date. @czender will modify the workflow after that. |
|
@czender I just pulled the latest hyperspectral code and queued 4/15 for VNIR (first) then SWIR. we'll see how things go over the weekend. |
|
@max-zilla - please update |
dlebauer
assigned remotesensinglab and unassigned czender, hmb1, FlyingWithJerome and smarshall-bmr
Description
The new hyperspectral imagery calibration procedure (#281) uses dedicated calibration imagery of Spectralon reflectors taken at multiple zenith angles (and possibly one day under both clear and overcast skies). This necessitates changes to the current reduction procedure, and subsequent use of, the Spectralon reflectance imagery.
Completion Criteria
The text was updated successfully, but these errors were encountered: