i.sentinel.mask: returncode error #85

veroandreo · 2020-01-30T12:27:40Z

Name of the addon
i.sentinel.mask

Describe the bug
When using the module to mask clouds and cloud shadows in a S2 scene, I get the following error:

[the rest skipped...]
--- Start removing misclassification from the shadow mask ---
--- Data preparation... ---
WARNING: Values in column <cat> will be overwritten
WARNING: Number of centroids exceeds number of areas: 24 > 22
WARNING: Number of duplicate centroids: 2
WARNING: Values in column <cat> will be overwritten
WARNING: Values in column <cat> will be overwritten
--- Reading mean sun zenith and azimuth from metadata file to compute
clouds shift ---
--- the mean sun Zenith is: 28.732 deg ---
--- the mean sun Azimuth is: 82.028 deg ---
--- Start computing the east and north clouds shift at steps of 100m of
clouds height---
Traceback (most recent call last):
  File "/home/veroandreo/.grass7/addons/scripts/i.sentinel.mask", line 638, in <module>
    main()
  File "/home/veroandreo/.grass7/addons/scripts/i.sentinel.mask", line 561, in main
    quiet=True, stderr=subprocess.DEVNULL)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py", line 499, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/home/veroandreo/software/grass79-dev/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py", line 392, in handle_errors
    returncode=returncode)
grass.exceptions.CalledModuleError: Module run v.overlay v.overlay --o --q ainput=addcat_5922 binput=cl_shift_5922 operator=and output=overlay_5922 ended with error
Process ended with non-zero return code 1. See errors in the (error) output.

To Reproduce
Steps to reproduce the behaviour:

Download S2 scene with UUID: 541021f8-63f5-4e9d-ba28-425c4c8451df

i.sentinel.download settings=your_sentinel_file output=tmp uuid=541021f8-63f5-4e9d-ba28-425c4c8451df

Import into a UTM 21S location (create with grass79 -c EPSG:32721 $HOME/grassdata/utm21s)

i.sentinel.import input=tmp/ pattern='B(02_1|03_1|04_1|08_1|8A_2)0m' extent=input memory=500

Apply cloud masking

# note the following will only work if no other S2 scene is present in the mapset, otherwise you'll have to write the map names or narrow the search pattern
blue=`g.list type=raster pattern="*B02_1*"`
green=`g.list type=raster pattern="*B03_1*"`
red=`g.list type=raster pattern="*B04_1*"`
nir=`g.list type=raster pattern="*B08_1*"`
nir8a=`g.list type=raster pattern="*B8A_2*"`
swir11=`g.list type=raster pattern="*B11_2*"`
swir12=`g.list type=raster pattern="*B12_2*"`

i.sentinel.mask -s blue=$blue green=$green red=$red swir11=$swir11 nir=$nir swir12=$swir12 nir8a=$nir8a cloud_mask=cloud shadow_mask=shadow mtd_file=`find tmp/ -name 'MTD_TL.xml'` scale_fac=10000

System description (please complete the following information):

Operating System: fedora 31
GRASS GIS version 79 (dev)
Further software components
- proj=5.2.0
- gdal=2.3.2
- geos=3.7.1
- sqlite=3.30.0
- python=3.7.4

The text was updated successfully, but these errors were encountered:

RobiFag · 2020-01-30T14:02:47Z

H vero,
I have just tested i.sentinel.mask using your image and the process ended without errors.

Steps:

Download the image using i.sentinel.download
Import into a WGS84/UTM21 location using r.import
Apply cloud mask

i.sentinel.mask -s 
blue=T21JYM_20200126T134201_B02_10m@test_vero 
green=T21JYM_20200126T134201_B03_10m@test_vero 
red=T21JYM_20200126T134201_B04_10m@test_vero 
nir=T21JYM_20200126T134201_B08_10m@test_vero 
nir8a=T21JYM_20200126T134201_B8A_20m@test_vero 
swir11=T21JYM_20200126T134201_B11_20m@test_vero 
swir12=T21JYM_20200126T134201_B12_20m@test_vero 
cloud_mask=v_cloud 
cloud_raster=r_cloud 
shadow_mask=v_shadow 
shadow_raster=r_shadow 
cloud_threshold=50000 
shadow_threshold=10000 
mtd_file=S2A_MSIL2A_20200126T134201_N0213_R124_T21JYM_20200126T154908\S2A_MSIL2A_20200126T134201_N0213_R124_T21JYM_20200126T154908.SAFE\GRANULE\L2A_T21JYM_A024002_20200126T134204\MTD_TL.xml

System description:

Operating System: Windows8
GRASS GIS version 7.8.2 (2019) - installed using OSGeo4W 64bit installer
PROJ: 6.2.1
GDAL/OGR: 3.0.2
GEOS: 3.8.0
SQLite: 3.29.0
Python: 3.7.0

veroandreo · 2020-01-30T14:07:11Z

Yes, I forgot to mention (actually I didn't think it was relevant then) that the error arises when setting the computational region to a subset of the full scene. If I, like you, run the process for the full scene, then it finishes without problems.

Do you think the reported problem is just something particular to that region subset or something more general for the module that won't work for scene subsets?

ninsbl · 2020-01-30T14:16:58Z

Just a wild guess: The problem might be that there are no clouds or shaddows in the smaller region?

@veroandreo : could you try to delete , stderr=subprocess.DEVNULL from https://github.com/OSGeo/grass-addons/blob/master/grass7/imagery/i.sentinel/i.sentinel.mask/i.sentinel.mask.py#L561 ?
Then you should get the full error output.

RobiFag · 2020-01-30T14:20:08Z

I have just tested the module setting as region and it works fine. Is it possible that you choose a region in which the module does not detect clouds or more probably shadows?
My region is:

projection: 1 (UTM)
zone:       -21
datum:      wgs84
ellipsoid:  wgs84
north:      7159480
south:      7119380
west:       708540
east:       773140
nsres:      20
ewres:      20
rows:       2005
cols:       3230
cells:      6476150

veroandreo · 2020-01-30T14:44:52Z

Just a wild guess: The problem might be that there are no clouds or shadows in the smaller region?

Actually, the other way around... Within the subset there was only one shadow (red) identified and a couple of clouds (blue) with their respective shadows that were missed (area in the square).

This is the region setting I was using (Iguazú falls area in case you wonder :))

g.region -p n=7169584.87 w=739426.81 e=761177.37 s=7152969.62 align=`g.list type=raster pattern="*B02_1*"`

@RobiFag could such cases be considered somehow easily within the module?

RobiFag · 2020-01-30T15:03:31Z

So only a shadow is detected in your computational region and no clouds.
If I remember correctly, I had already added some check about the area of clouds and shadows detected but you should get a clearer message (e.g 'No clouds have been detected').
Maybe something has changed in the code @ninsbl ? (e.g. a82876e#diff-86a1994a46407a9b6bd9d1b786d7869fL378-R389 )

RobiFag · 2020-01-30T16:49:04Z

Maybe I have understood..in Vero's case some clouds are detected but are then removed by the cleaning geometries procedure (remove small areas). Actually there was a check in the code about the detected clouds but it considered only the temporary vector cloud map and not the cleaned one. Moreover the if statement regarding the cloud check was removed (a82876e#diff-86a1994a46407a9b6bd9d1b786d7869fL378-R389)

This is a possible code side solution:

else:
        gscript.message(_('--- Cleaning geometries ---'))
        gscript.run_command('v.clean',
            input=tmp["cloud_v"],
            output=cloud_mask,
            tool='rmarea',
            threshold=cloud_threshold)
        info_c_clean = gscript.parse_command('v.info',
            map=cloud_mask,
            flags='t')
        if info_c_clean['areas'] == '0':
            gscript.warning(_('No clouds have been detected'))
            check_cloud = 0
        else:
            check_cloud = 1
    gscript.message(_('--- Finish cloud detection procedure ---'))
    # End of Clouds detection

    if (options['shadow_mask'] or options['shadow_raster']) and check_cloud == 1:

In this way Vero's case ends without error but with the warning message 'no clouds detected

WARNING: All subsequent operations will be limited to the current computational region
WARNING: Any rescale factor has been applied
--- Start computing maximum values of bands ---
T21JYM_20200126T134201_B02_10m_double@test_vero
T21JYM_20200126T134201_B03_10m_double@test_vero
T21JYM_20200126T134201_B04_10m_double@test_vero
T21JYM_20200126T134201_B08_10m_double@test_vero
T21JYM_20200126T134201_B8A_20m_double@test_vero
T21JYM_20200126T134201_B11_20m_double@test_vero
T21JYM_20200126T134201_B12_20m_double@test_vero
--- Computed maximum value: dict_values([0.7832, 0.8584, 0.8288, 1.0008, 0.7023, 0.9381, 1.3597]) ---
--- Statistics have been computed! ---
--- Start clouds detection procedure ---
--- Computing cloud mask... ---
--- Converting raster cloud mask into vector map ---
Estrazione delle aree in corso ...
Writing areas...
Si sta ricreando la topologia per il vettore <cloud_v_6924@test_vero>...
Registrando primitive...
Costruendo aree...
Isole allegate...
Centroidi allegati...
r.to.vect completo.
--- Cleaning geometries ---
--------------------------------------------------
Tool: Threshold
Remove small areas: 50000
--------------------------------------------------
Copying features...
Sto ricostruendo parti della topologia...
Si sta ricreando la topologia per il vettore <v_cloud_new3@test_vero>...
Registrando primitive...
Costruendo aree...
Isole allegate...
Centroidi allegati...
--------------------------------------------------
Tool: Remove small areas
654 areas of total size 314723 removed
--------------------------------------------------
Sto ricostruendo la topologia per la mappa vettoriale di output...
Si sta ricreando la topologia per il vettore <v_cloud_new3@test_vero>...
Registrando primitive...
WARNING: No clouds have been detected
--- Finish cloud detection procedure ---
WARNING: No shadow mask will be computed

Maybe you can try with smaller cloud and shadow thresholds!

veroandreo · 2020-01-30T23:07:58Z

[skipping the rest]

WARNING: No clouds have been detected
--- Finish cloud detection procedure ---
WARNING: No shadow mask will be computed

Why no shadow mask computed? What if, like in this case, only cloud shadow is present in the scene or region subset? Are shadows dependent on finding clouds first then?

Maybe you can try with smaller cloud and shadow thresholds!

Right! I have not modified them. Which is the unit of these thresholds? Square meters? If yes, we could add that in the description of the options to make it clearer

RobiFag · 2020-01-31T08:27:23Z

The shadow mask, in this case, is not computed since the cloud mask is mandatory for shadow mask creation. In fact, the cloud map is used during the cleaning phase of the shadow mask in order to remove misclassifications (see the manual for further details about the algorithm of cloud and shadow detection).

Regarding the thresholds, the unit is square meters (v.clean - tool=rmarea).

RobiFag · 2020-01-31T10:17:00Z

I have already made some changes to the code (see #86 ). Now the computation of the shadow mask is independent of the clouds mask. If no clouds are detected, the shadow mask is computed but the procedure of cleaning misclassifications from shadow is not performed.

veroandreo · 2020-01-31T17:39:49Z

Sounds good to me! Thanks for such a prompt response and all the explanations :)

ninsbl · 2020-03-17T22:13:57Z

Can the ticket be closed with this PR merged:
#86
?

veroandreo · 2020-03-18T11:19:07Z

Can the ticket be closed with this PR merged:
#86
?

Yes, I think so. Thanks again both of you 😊

veroandreo added bug Something isn't working new addon PR contains a new addon or issue proposes one labels Jan 30, 2020

RobiFag mentioned this issue Jan 30, 2020

i.sentinel.mask: improved control checks about the number of clouds and shadows detected #86

Merged

veroandreo closed this as completed Mar 18, 2020

neteler removed the new addon PR contains a new addon or issue proposes one label Dec 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

i.sentinel.mask: returncode error #85

i.sentinel.mask: returncode error #85

veroandreo commented Jan 30, 2020

RobiFag commented Jan 30, 2020 •

edited

Loading

veroandreo commented Jan 30, 2020

ninsbl commented Jan 30, 2020

RobiFag commented Jan 30, 2020

veroandreo commented Jan 30, 2020

RobiFag commented Jan 30, 2020

RobiFag commented Jan 30, 2020

veroandreo commented Jan 30, 2020 •

edited

Loading

RobiFag commented Jan 31, 2020

RobiFag commented Jan 31, 2020

veroandreo commented Jan 31, 2020

ninsbl commented Mar 17, 2020

veroandreo commented Mar 18, 2020

i.sentinel.mask: returncode error #85

i.sentinel.mask: returncode error #85

Comments

veroandreo commented Jan 30, 2020

RobiFag commented Jan 30, 2020 • edited Loading

veroandreo commented Jan 30, 2020

ninsbl commented Jan 30, 2020

RobiFag commented Jan 30, 2020

veroandreo commented Jan 30, 2020

RobiFag commented Jan 30, 2020

RobiFag commented Jan 30, 2020

veroandreo commented Jan 30, 2020 • edited Loading

RobiFag commented Jan 31, 2020

RobiFag commented Jan 31, 2020

veroandreo commented Jan 31, 2020

ninsbl commented Mar 17, 2020

veroandreo commented Mar 18, 2020

RobiFag commented Jan 30, 2020 •

edited

Loading

veroandreo commented Jan 30, 2020 •

edited

Loading