This repository is no longer updated, any development is now happening in the new SICE repository of the GEUS-SICE organization.

Table of Contents

• Sentinel-3 (S3) Albedo Processing Pipeline
  • Fetch S3 OLCI & SLSTR products
  • Process with SNAP
  • pySICE
  • Mosaic
  • Outputs
• Debugging & Testing
• Development Environment
  • Bash
  • Python
  • Parallel
  • GRASS
  • GPT
  • SNAP
• Misc Notes & Code Snippets
  • Generate ice mask
  • Footprint

Sentinel-3 (S3) Albedo Processing Pipeline

1. Fetch S3 OLCI & SLSTR products
2. Process OLCI & SLSTR using SNAP GPT
3. Process using pySICE
4. Build daily mosaic

The steps above are encapsulated in ./S3_wrapper.sh.

In more detail:

Fetch S3 OLCI & SLSTR products

• Download OLCI EFR and SLSTR RBT products for a specified day with ./dhusget_wrapper.sh.

Process with SNAP

• This step is a combination of running GPT on ./S3.xml with ./S3_proc.sh.
• Inputs: OLCI and SLSTR scenes
• Outputs: A folder OLCI timestamp

pySICE

• Run ./sice.py passing in one of the folders generated in the previous step.
• WARNING: This step is slow, and may take > 24 hours.

Mosaic

• As implemented in ./dm.sh and ./dm.grass.sh, for each day
  • Combine all the files from to form a mosaic
  • Mask out clouds, then…
  • When scenes overlap, use minimum SZA

Outputs

File Name	Description	Units
grain_diameter.tif	Snow grain diameter	mm
snow_specific_surface_area.tif	Snow specific surface area	m2kg-1
NDSI.tif	Normalized Difference Snow Index
NDBI.tif	Normalized Difference Blue ice Index
SCDA_final.tif	Simple Cloud Detection Algorithm
SZA.tif	Solar Zenith Angle
sza_lut.tif	Solar Zenith Angle LookUp Table
SAA.tif	Solar Azimuth Angle
OZA.tif	Observation Zenith Angle
OAA.tif	Observation Azimtuh Angle
albedo_bb_planar_nir.tif	near infrared broadband planar albedo	700-2400 nm
albedo_bb_planar_sw.tif	shortwave broadband planar albedo	300-2400 nm
albedo_bb_planar_vis.tif	visible broadband planar albedo	300-700 nm
albedo_bb_spherical_nir.tif	near infrared broadband spherical albedo	700-2400 nm
albedo_bb_spherical_sw.tif	shortwave broadband spherical albedo	300-2400 nm
albedo_bb_spherical_vis.tif	visible broadband spherical albedo	300-700 nm
r_TOA_01..21.tif	Top Of the Atmosphere reflectance (OLCI bands 1 through 21)
rBRR_01..21.tif	bottom of the atmosphere surface reflectance (OLCI bands 1 through 21)
BT_S7..9.tif	Brightness Temperatures (SLSTR bands 7 to 9)
r_TOA_S1..S5.tif	Top of the Atmosphere reflectance (SLSTR bands 1 to 5)
albedo_spectral_planar_01..21.tif	spectral planar albedo (OLCI bands 1 to 11 and 16 to 21)
albedo_spectral_spherical_01..21.tif	spectral spherical albedo (OLCI bands 1 to 11 and 16 to 21)
diagnostic_retrieval.tif	See next table
height.tif	Altitude from the SRAL instrument (Ku/C Radar Altimeter)
conc.tif	pollutant concentration (volumetric concentration)
al.tif	effective absorption length(mm)
r0.tif	reflectance of a semi-infinite non-absorbing snow layer
03.tif	total ozone product (OLCI)
03_SICE.tif	total ozone product (OLCI) corrected for ozone scattering
WV.tif	Total column Water Vapor (OLCI)
num_scenes.tif	number of scenes in daily mosaic
nume_scenes_cloudfree.tif	number of cloud free scenes in daily mosaic

Diagnostic Code	Description
0	clean snow
1	polluted snow
6	polluted snow for which r0 was calculated and not derived from observations
7	polluted snow of calculated spherical albedo in bands 1 and 2 >0.98 reprocessed as clean snow
100	sza<75, no retrival
102	TOA reflectance at band 21 < 0.1, no retrieval
104	grain_diameter < 0.1, no retrieval, potential cloud flag
-n	impossible to solve polluted snow albedo equation at band n

Slopey

The Slopey option computes effective solar/viewing angles based on ArcticDEM to obtain reflectance products accounting for slopes. This terrain correction is an option of the SICE toolchain that can be enabled in ./S3_wrapper.sh. The associated outputs have the same file names as in the table presented above.

Debugging & Testing

Change the year and day for loops in ./S3_wrapper.sh to one year and one day. E.g.

for year in 2017; do # one year
  for doy in 227 180; do # two example days
  # loop contents
  done
done

Development Environment

This work is developed with the following software versions

Bash

bash --version

GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Python

Run the following using ./SICE.yml in order to set up an identical Anaconda Python environment.

conda env create -f SICE.yml

And then run conda activate SICE to activate the SICE environment.

ResolvePackageNotFound error can be raised. In that case, run conda env export --no-builds > environment.yml instead.

Parallel

parallel --version

GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.

GRASS

grass --version 2>&1

GRASS GIS 7.4.0

Geographic Resources Analysis Support System (GRASS) is Copyright,
1999-2018 by the GRASS Development Team, and licensed under terms of the
GNU General Public License (GPL) version >=2.
 
This GRASS GIS 7.4.0 release is coordinated and produced by
the GRASS Development Team with contributions from all over the world.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.

GPT

~/local/snap/bin/gpt --diag

SNAP Release version 7.0
SNAP home: /home/kdm/local/snap/bin//..
SNAP debug: null
SNAP log level: null
Java home: /home/kdm/local/snap/jre
Java version: 1.8.0_202
Processors: 8
Max memory: 18.7 GB
Cache size: 1024.0 MB
Tile parallelism: 8
Tile size: 512 x 512 pixels

To configure your gpt memory usage:
Edit snap/bin/gpt.vmoptions

To configure your gpt cache size and parallelism:
Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism} 

SNAP

Misc Notes & Code Snippets

Generate ice mask

• Use the icemask from BedMachine v3

grass72 -c EPSG:3413 ./Gtmp
r.in.gdal input=NetCDF:~/data/Greenland/Morlighem_2017/BedMachineGreenland-2017-09-20.nc:mask output=icemask

g.region raster=icemask
g.region res=500 -ap
g.region zoom=icemask

d.mon start=wx0
d.erase
d.rast icemask

r.mapcalc "mask = if(icemask == 4, null(), icemask)" --o
d.rast mask
g.region zoom=mask

r.out.gdal -c -m input=mask output=mask.tif type=Byte createopt=COMPRESS=DEFLATE --o
exit
trash Gtmp

Buffered ice mask

grass -c mask.tif ./Gtmp
r.in.gdal input=mask.tif output=mask
# 50 cells = 25 km @ 500 m
r.mapcalc "ice = if(mask == 2, 1, null())"
r.grow input=ice output=ice_grow radius=50 new=1

r.null ice_grow null=100
r.clump input=ice_grow output=clumps
r.stats -c clumps sort=asc
for ID in $(r.stats -c clumps sort=asc | head -n7 | cut -d" " -f1); do
  r.mapcalc "ice_grow = if(clumps == ${ID}, 1, ice_grow)" --o
done
r.null ice_grow setnull=100

r.out.gdal input=ice_grow output=ice_mask_buffer.tif format=GTiff type=Byte createopt="COMPRESS=DEFLATE"
exit
trash Gtmp

Footprint

Get GL outline by drawing in Google Earth, export KML, then:

ogrinfo -al GL_outline.kml  | grep LINESTRING | sed s/\ 0//g