Skip to content

Commit

Permalink
Wrote the manual page
Browse files Browse the repository at this point in the history
  • Loading branch information
@RobiFag
RobiFag committed Jul 28, 2018
1 parent 1ad8f39 commit 5758b6c
Show file tree
Hide file tree
Showing 3 changed files with 175 additions and 60 deletions.
235 changes: 175 additions & 60 deletions i.sentinel.preproc.html
View file
Original file line number Diff line number Diff line change
@@ -1,89 +1,204 @@
<h2>DESCRIPTION</h2>
<em>i.sentinel.preproc</em> allows to download, import and apply atmospheric correction to Sentinel 2 images.
<!--p>
The implemented procedure consists essentially of values thresholds, comparisons and calculations between bands and they lead to two different
rough maps of clouds and shadows which require further improvements and elaborations (e.g. transformation from raster to vector, cleaning geometries,
removing small areas, checking topology, etc.) carried out in the different steps of the procedure.
<em>i.sentinel.preproc</em> allows to import Sentinel 2 images and perform atmospheric correction.
<p>
i.sentinel.preproc is a module for the preprocessing of Sentinel 2 images (Level-1C Single Tile product) which wraps the import and the atmospheric correction using respectively <a href="i.sentinel.import.html">i.sentinel.import</a> and <a href="i.atcorr.html">i.atcorr</a>.<br>
It works both with Sentinel 2A and 2B images.<br>
The aim is to provide a simplified module which allows importing images, which area downloaded using <a href="i.sentinel.download.html">i.sentinel.download</a> or any other sources, and performing the atmospheric correction avoiding users to provide all the required input parameters manually. In fact, regarding the atmospheric correction performed with i.atcorr one of the most challenging steps, especially for unexperienced users, is the compiling of the control file with all the required parameters to parametrize the 6S (<em>Second Simulation of Satellite Signal in the Solar Spectrum</em>) model on which i.atcorr is based.<br><br>
To run i.atcorr, users have to provide the so-called control file in which all the parameters (geometrical conditions, date, time, longitude and latitude of the center of the scene, atmospheric model, aerosol model, visibility or Aerosol Optical Depth -AOD- value, mean elevation target and bands number) have to be specified with precise syntax rules and codes.<br>
i.sentinel.preproc retrieves as many parameters as possible from the MTD_MSIL1C.xml metadata file (e.g. Geometrical conditions, data and time and bands number), longitude and latitude are automatically computed from the computational region while others like the mean target elevation above sea level from the input digital elevation model (DEM). Only a few parameters have to be provided by users who can choose the proper option from a drop-down menu thus avoiding to enter the corresponding code. In any case, i.sentinel.preproc writes a temporary control file, changing it according to the band number, following the syntax rules and codes of i.atcorr and then it runs i.atcorr for all bands.<br><br>
<br><br>
<center>
<img src="i_sentinel_preproc_GWF.png" width="30%">
<br><br>
<i>Fig: Module General WorkFlow</i>
</center>
<br><br>
When all bands have been processed by the integrated version of i.atcorr, an histogram equalization grayscale color scheme is applied.
<br><br>
If the <b>-t</b> flag is checked, a text file ready to be used as input for <a href="i.sentinel.mask">i.sentinel.mask</a> will be created. In this case a name for the output text file has to be specified.
<br><br>
NOTE: as for i.atcorr, current region settings are ignored. The region is temporary set to maximum image extent and restored at the end of the process.
<br><br>
<em><b>Important</b></em>: i.sentinel.preproc requires all the bands of a Sentinel 2 images. If the module is used only for the atmospheric correction, all bands from *_B01 to *_B12 must be imported.<br>Moreover, the original bands name has to be kept unchanged (e.g if the original name is <em>T17SPV_20180504T155911_B02</em> the imported raster map in the GIS DATABSE must be named <em>T17SPV_20180504T155911_B02</em>).
<br><br>
<h3>Import</h3>
<p>
i.sentinel.preproc allows the import of all the bands of a Sentinel 2 image. The required input is the <b>.SAFE folder</b> downloaded using i.sentinel.download or any other source (e.g. Copernicus Open Access Hub). Note that in the case that spatial reference system of input data differs from GRASS location, the input data are reprojected.<br>
Number of imported bands <b>can not</b> be reduced, all bands are automatically imported by default.<br><br>

<em><b>Important</b></em>: i.sentinel.preproc allows the import of one image at a time because the input .SAFE folder is also used to automatically identify the corresponding MTD_MSIL1C.xml file that is used during the atmospheric correction.<br><br>

<table cellspacing="2" cellpadding="2" width="100%" border="0">
<tbody>
<tr>
<td align="center" valign="bottom"><a href="i_sentinel_mask_GWF.png"><img src="i_sentinel_mask_GWF.png" width="100%"></a><br><i>Fig: Module General WorkFlow</i></td>
<td align="center" valign="bottom"><a href="i_sentinel_mask_CD.png"><img src="i_sentinel_mask_CD.png" width="100%"></a><br><i>Fig: Cloud detection procedure</i></td>
<td align="center" valign="bottom"><a href="i_sentinel_mask_SD.png"><img src="i_sentinel_mask_SD.png" width="100%"></a><br><i>Fig: Shadow detection procedure</i></td>
</tr>
</tbody>
</table>
The import can be skipped using the <b>-i</b> flag. Note that even if the import is skipped the input .SAFE folder must be specified to automatically retrieve the MTD_MSIL1C.xml file.

<h3>Atmospheric correction</h3>
<p>
The algorithm has been developed starting from rules found in literature (Parmes et. al 2017) and conveniently refined.<br>
Regarding the detection of shadows, some misclassification can occur. Often shadows and water have in fact similar reflectance
values which can lead to erroneous classification of water bodies as shadows. Therefore, in order to increase the accuracy of
the final shadow mask, a control check is implemented. Clouds and shadows are spatially intersected in order to remove misclassified areas.
This means that all those shadow geometries which do not intersect a cloud geometry are removed.
i.sentinel.preproc allows performing atmospheric correction of all bands of a Sentinel 2 scene with a single process using i.atcorr. Unlike i.atcorr, it writes the control file changing it according to the band number. The only required inputs are:
<ul>
<li><b>input_dir</b> = the *.SAFE directory where the image and metadata file are stored,
<li><b>elevation</b> = raster of a digital elevation model,
<li><b>visibility or AOD value</b> = raster of a visibility map or an AOD value (<em>see AOD section</em>),
<li><b>Atmospheric model</b> = to be choosen from the drop-down menu,
<li><b>Aerosol model</b> = to be choosen from the drop-down menu,
<li><b>suffix</b> = a suffix for the output maps name
<li><b>rescale</b> = the output range of values for the corrected bands, for example: 0-255, 0-1, 1-10000 (default value 0-1).
</ul>
<br><br>
The module writes the control file automatically starting from the input above.

<div align="center" style="margin: 10px">
<a href="i_sentinel_mask_CS.png">
<img src="i_sentinel_mask_CS.png" width="30%">
</a><br>
<i>Fig: Module General WorkFlow</i>
</div-->
<!--center>
<img src="i_sentinel_mask_CS.png" width="30%">
<br>
<i>Fig: Module General WorkFlow</i>
</center-->
<h4>Control file</h4>
<p>
i.atcorr requires a control file to parametrize the 6S algorithm on which it is based.

<!--p>
All necessary input bands (blue, green, red, nir, nir8a, swir11, swir12) must be imported in GRASS and specified one by one or using a text file.
The text file has to be written following the syntax below: <em>variable=your_map</em>
<p>Below an example of the control file, taken from the i.atcorr manual page, of a Sentinel 2A image:

<div class="code"><pre>
blue=<em>your_blue_map</em>
green=<em>your_green_map</em>
red=<em>your_red_map</em>
nir=<em>your_nir_map</em>
nir8a=<em>your_nir8a_map</em>
swir11=<em>your_swir11_map</em>
swir12=<em>your_swir12_map</em>
25 - geometrical conditions = Sentinel 2A
5 4 19.737 -78.727 35.748 - month day hh.ddd longitude latitude ("hh.ddd" is in decimal hours GMT)
2 - atmospheric model = midlatitude summer
1 - aerosols model = continental
0 - visibility [km] (aerosol model concentration)
0.07 - AOD at 550nm
-0.124 - mean target elevation above sea level [km]
-1000 - sensor height (here, sensor on board a satellite)
167 - sensor band = Sentinel2A Blue band B2
</pre></div>

Tha variables names (blue, green, red, nir, nir8a, swir11, swir12) have to be written precisely like in the example above (e.g. not Blue, nor BLUE but blue),
no spaces or special characters are permitted.
Using i.sentinel.preproc the only parameters from the list above that users have to provide are: atmospheric model, aerosol model, visibility or AOD value. The others are automatically retrieved from the metadata file, input elevation map and bands.
<br><br>
<ol>
<b><li><b> Geometrical conditions</b>
<p>
The geometrical condition of the satellite are read from the MTD_MSIL1C.xml file and converted to the corresponding i.atcorr code, 25 for Sentinel 2A mission and 26 for Sentinel 2B.
<br><br>
<b><li> Date, time, longitude and latitude </b>
<p>
Date (month and day) and time are read from the MTD_MSIL1C.xml file. The date (with the format YYYY-MM-DDTHH:MM:SSZ) is converted in a standard format and only the month and the day are selected and added to the control file.<br><br>
Time is already in Greenwich Mean Time (GMT), as i.atcorr requires, and it's automatically converted to decimal hours.<br>
Longitude and latitude are computed from the computational region and converted to WGS84 decimal coordinates.
<br><br>
<b><li> Atmospheric model</b>
<p>
Only some options are available:
<ul>
<li>Automatic
<li>No gaseous absorption
<li>Tropical
<li>Midlatitude summer
<li>Midlatitude winter
<li>Subarctic summer
<li>Subarctic winter
<li>Us standard 62
</ul><br><br>
Users can choose the proper option from a drop-down menu. The desired model is automatically converted to the corresponding code and added to the control file.
<br><br><em><b>Automatic</b> option</em><br>
The default option is <em>Automatic</em> which consists in the automatic identification of the proper atmospheric model for the input image. The <em>Automatic</em> option reads the latitude of the center of the computational region and uses it to choose between Midlatitude (15.00 > lat <= 45.00), Tropical (-15.00 > lat <= 15.00) and Subarctic (45.00 > lat <= 60.00) for Northern Hemisphere (obviously it also works for the Southern Hemisphere). Then, the month from the acquisition date is used to distinguish summer or winter in case of Midlatitude or Subarctic model. Once the proper atmospheric model is identified, it is converted to the corresponding code and added to the control file.<br>
Note that this is a simplified and standardized method to identify the atmospheric model. Obviously, it is possible to choose other options from those available.
<br><br>
<b><li> Aerosol model</b><br>
<p>
The final outputs are two different vector maps, one for clouds and one for shadows.
Also in this case, only some options are available and users have to select the desired one from the drop-down menu, then it is converted to the corresponding code and added to the control file.
<ul>
<li>no aerosols
<li>continental model
<li>maritime model
<li>urban model
<li>shettle model for background desert aerosol
<li>biomass burning
<li>stratospheric model
</ul><br><br>
No automatic procedure has been implemented in this case.
<br><br>
<b><li> Visibility or AOD </b><br>
<p>
The metadata file (MTD_TL.xml) is required only if both masks (cloud and shadow)
are computed. The module retrieves from this file the sun azimuth and zenith necessary for the shadow mask cleaning phase
<em>(see the schema above)</em>
By default, i.sentinel.preproc uses the input visibility map to estimate a visibility value to be added in the control file. If no visibility map is available for the processed scene, it is possible to use an estimated Aerosol Optical Depth (AOD) value checking the <b>-a</b> flag.<br>
If the -a flag is checked and a visibility map is provided, the visibility will be ignored and no mean visibility value will be computed and added to the control file. Whereas, if the -a flag isn't checked and an AOD value is provided it will be ignored and not added to the control file.<br><br> In the same way, if the -a flag is checked and a visibility map is provided it will be excluded from atmospheric correction process.<br><br>
<b> AOD</b><br><br>
The AOD value can be specified by users (e.g. <tt>aod_value=0.07</tt>) or automatically retrieved from an AERONET file to be given as input instead of the AOD value.<br>
i.sentinel.preproc reads the AERONET file, identify the closest available date to the scene date and compute AOD at 550nm using the closest upper and lower wavelength to 550 (e.g. 500nm and 675nm) and applying the Angstrom coefficient. <br><br>
The type of AERONET file is a Combined file for All Points (Level 1.5 or 2.0)<br>
To download this kind of file:<br>
<ol>
<li>Go to <a href=http://aeronet.gsfc.nasa.gov/cgi-bin/webtool_opera_v2_inv>http://aeronet.gsfc.nasa.gov</a>
<li>Choose the site you want to get data from
<li>Tick the box near the bottom labelled as 'Combined file (all products without phase functions)'
<li>Choose either Level 1.5 or Level 2.0 data. Level 1.5 data is unscreened, so contains far more data meaning it is more likely for users to find data near your specified time
<li>Choose 'All Points' under Data Format
<li>Download the file
<li>Unzip (the file has a .dubovik extension)
</ol>
<br><br>
Then, giving this file as input (e.g. <tt>aeronet_file=your_path/*.dubovik</tt>), the AOD at 550nm will be automatically computed and added to the control file.<br><br>
NOTE: as i.atcorr manual explain, if an AOD value is provided a value 0 for the visibility has to be entered with the AOD value in the following line. Obviously, i.sentinel.preproc takes into account this syntax rule and automatically adds a 0 value for visibility (or -1 if AOD=0) if an AOD value is provided (through both <tt>aod_value</tt> and <tt>aeronet_file</tt>).
<br><br>
<li><b> Mean target elevation above sea level </b><br>
<p>
If flag <b>-s</b> is given all selected bands are rescaled using the specified scale factor [<b>scale_fac</b>=<em>integer</em>]. By default the scale factor is set to 10000,
the QUANTIFICATION_VALUE from the metadata of Sentinel 2 images.
Mean target elevation above sea level is automatically estimated from the input digital elevation model. According to the rules for writing the contol file of i.atcorr, the mean elevation value is added as a negative value and converted in kilometers (e.g. if mean=121 in the control file it will be written in [-km], i.e., -0.121).
<br><br>
<li><b> Sensor height </b><br>
<p>
The module takes the current region settings into accout. To ignore the current region and set it from the whole image, the flag <b>-r</b> has to be given.
Since the sensor is on board a satellite, the sensor height is automatically set to -1000.
<br><br>
<li><b> Sensor band </b><br>
<p>
The module allows to compute only the cloud mask or both cloud and shadow masks. If flag <b>-c</b> is given, only the cloud procedure will be performed. The computation
of cloud mask is mandatory for shadow mask creation. In fact cloud map is used during the cleaning phase of the shadow mask in order to remove misclassifications.<-->
The number of the band changes automatically according to the band that is processed at that moment. The module reads the name of the band and converts it into the corresponding code.
</ol>

<h2>EXAMPLE</h2>
<!--div class="code">
i.sentinel.mask -r -s input_file=/home/input_bands.txt cloud_mask=cloud_sen2 shadow_mask=shadow_sen2 mtd_file=/home/MTD_TL.xml scale_fac=1000
</div>
<p>
-r to set the computational region to the maximum image extente and -s to rescale the input bands with the specified scale factor (in this case 1000)
The example illustrates how to run i.sentinel.preproc for a Sentinel 2A image (S2A_MSIL1C_20180504T155911_N0206_R097_T17SPV_20180504T194416.SAFE) in the North Carolina location.
<div class="code"><pre>
i.sentinel.preproc -a -t input_dir=/path/S2A_MSIL1C_20180504T155911_N0206_R097_T17SPV_20180504T194416.SAFE elevation=elevation atmospheric_model=Automatic aerosol_model="Continental model" aod_value=0.07 suffix=cor text_file=/path/input_cloud_mask.txt
<pre></div>
<p>
Here is the control file automatically written for Band 02 of the input scene
<div class="code"><pre>
25
5 4 19.737 -78.727 35.748
2 -The Automatic option identified the Midlatitude summer as the proper model for the scene
1
0 -The visibility is set to 0 with AOD in the following line
0.07
-0.121
-1000
167
</pre></div>
Here is the output text file ready to be used as input for i.sentinel.mask (-t flag)
<div class="code"><pre>
blue=T17SPV_20180504T155911_B02_cor
green=T17SPV_20180504T155911_B03_cor
red=T17SPV_20180504T155911_B04_cor
swir11=T17SPV_20180504T155911_B11_cor
nir=T17SPV_20180504T155911_B08_cor
swir12=T17SPV_20180504T155911_B12_cor
nir8a=T17SPV_20180504T155911_B8A_cor
</pre></div>

<div align="center" style="margin: 10px">
<a href="i_sentinel_preproc_ES.png">
<img src="i_sentinel_preproc_ES.png" width="1126" height="906" alt="i.atcorr example" border="0">
</a><br>
<i>Figure: Sentinel-2A Band 02</i>
</div>

<h2>IMPORTANT NOTES</h2>
<ul>
<li>i.sentinel.preproc integrates a simplyfied version of both modules (i.sentinel.import and i.atcorr), only some options are available. For instance, if it's necessary a strong customization (e.g. definition of your own atmospheric or aerosol model), please refer to i.atcorr.<br><br>
<li>At the moment i.sentinel.preproc works <b>only</b> with the Level-1C Single Tile product which name follows the New Compact Naming Convention: MMM_MSIL1C_YYYYMMDDTHHMMSS_Nxxyy_Rooo_ProductDiscriminator.SAFE (e.g. S2A_MSIL1C_20170527T102031_N0205_R065_T32TMQ_20170527T102301.SAFE). This convention was introducd by the begining of December 2016 (see <a href="https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/naming-convention">ESA Sentinel User Guide</a> for further information)
</ul>

<h2>REFERENCE</h2>
<h2>FOLLOW UP</h2>
<ul>
<li>Parmes et. al 2017</li>
</ul-->
<li>Add support for the Old format Naming Convention (e.g. S2A_OPER_PRD_MSIL1C_PDMC_20160930T155112_R079_V20160930T095022_20160930T095944.SAFE)
<li>Implement download functionality avoiding dependencies
<li>Integrate the Topographic Correction
</ul>

<h2>SEE ALSO</h2>

<em>
<a href="i.sentinel.download.html">i.sentinel.download</a>,
<a href="i.sentinel.import.html">i.sentinel.import</a>,
<a href="i.sentinel.import.html">i.atcorr</a>,
<a href="i.atcorr.html">i.atcorr</a>,
<a href="i.sentinel.import.html">i.sentinel.mask</a>,
<a href="r.import.html">r.import</a>,
<a href="r.extenal.html">r.external</a>
Expand Down
Binary file added i_sentinel_preproc_ES.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added i_sentinel_preproc_GWF.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 5758b6c

Please sign in to comment.