Burst inconsistency when unpacking scenes across dates of stack

[shitong01]

Hi ISCE developers,

Recently, we processed a coregistered SLC stack over cities in Europe and found that at times, the bursts processed across dates in a stack may be inconsistent. This seems to be an edge case occurs specifcially when we have a mix of S1A and S1B acquisitions in the stack.

This is an example of a stack we created over Bucharest >
Start Date: 20200503
End Date: 20200527
Master Date: 20200527

stackSentinel.py command:

stackSentinel.py -s data -d ./dem/demLat_N44_N45_Lon_E025_E027.dem.wgs84 -a .AuxDir/ -o ./orbits -b '44.3112 44.5513 25.9251 26.2722' -W slc -m 20200527 -C geometry -n '2'

SLCs used in ./data folder:

S1A_IW_SLC__1SDV_20200503T160840_20200503T160907_032403_03C07F_86EB.zip
S1A_IW_SLC__1SDV_20200503T160905_20200503T160932_032403_03C07F_AA8C.zip
S1B_IW_SLC__1SDV_20200509T160757_20200509T160824_021507_028D58_29EF.zip
S1B_IW_SLC__1SDV_20200509T160821_20200509T160848_021507_028D58_FEDB.zip
S1A_IW_SLC__1SDV_20200515T160841_20200515T160908_032578_03C5FB_8471.zip
S1A_IW_SLC__1SDV_20200515T160905_20200515T160932_032578_03C5FB_6A9E.zip
S1B_IW_SLC__1SDV_20200521T160757_20200521T160824_021682_029289_17E0.zip
S1B_IW_SLC__1SDV_20200521T160822_20200521T160849_021682_029289_4958.zip
S1A_IW_SLC__1SDV_20200527T160841_20200527T160908_032753_03CB48_F2D5.zip
S1A_IW_SLC__1SDV_20200527T160906_20200527T160933_032753_03CB48_A2F1.zip

In the ./slaves folder:

|-- 20200503
|   |-- IW2
|   |   |-- burst_01.slc
|   |   |-- burst_01.slc.vrt
|   |   |-- burst_01.slc.xml
|   |   |-- burst_02.slc
|   |   |-- burst_02.slc.vrt
|   |   |-- burst_02.slc.xml
|   |   |-- burst_03.slc
|   |   |-- burst_03.slc.vrt
|   |   |-- burst_03.slc.xml
|   |   |-- burst_04.slc
|   |   |-- burst_04.slc.vrt
|   |   `-- burst_04.slc.xml
|   `-- IW2.xml
|-- 20200509
|   |-- IW2
|   |   |-- burst_01.slc
|   |   |-- burst_01.slc.vrt
|   |   |-- burst_01.slc.xml
|   |   |-- burst_02.slc
|   |   |-- burst_02.slc.vrt
|   |   |-- burst_02.slc.xml
|   |   |-- burst_03.slc
|   |   |-- burst_03.slc.vrt
|   |   `-- burst_03.slc.xml
|   `-- IW2.xml
|-- 20200515
|   |-- IW2
|   |   |-- burst_01.slc
|   |   |-- burst_01.slc.vrt
|   |   |-- burst_01.slc.xml
|   |   |-- burst_02.slc
|   |   |-- burst_02.slc.vrt
|   |   |-- burst_02.slc.xml
|   |   |-- burst_03.slc
|   |   |-- burst_03.slc.vrt
|   |   |-- burst_03.slc.xml
|   |   |-- burst_04.slc
|   |   |-- burst_04.slc.vrt
|   |   `-- burst_04.slc.xml
|   `-- IW2.xml

In the ./master folder:

master
|-- IW2
|   |-- burst_01.slc
|   |-- burst_01.slc.vrt
|   |-- burst_01.slc.xml
|   |-- burst_02.slc
|   |-- burst_02.slc.vrt
|   |-- burst_02.slc.xml
|   |-- burst_03.slc
|   |-- burst_03.slc.vrt
|   |-- burst_03.slc.xml
|   |-- burst_04.slc
|   |-- burst_04.slc.vrt
|   `-- burst_04.slc.xml
`-- IW2.xml

As you can see, the bursts extracted over IW2 alternates between 3 and 4 bursts, depending if its S1A (4 bursts) or S1B (3 bursts). Due to this inconsistency, This causes an issue when we start pairing our dates for calculations (e.g. coherence etc):

A deeper look into this burst inconsistency occurs because ISCE does an approximate computation of the burst boundary (Sentinel1.py and BurstSLC.py) to decide if a burst should be processed or not. This approximated boundary is not precise (has a huge buffer) and inconsistent between S1A and S1B:

(click on image and zoom in for clearer view)

• Magenta box
  • approximated burst boundaries computed by BurstSLC.py for 20200527's burst 4 (B4) captured by S1A during unpacking.
• Cyan box
  • approximated burst boundaries computed by BurstSLC.py for 20200509's burst 4 (B4) captured by S1B during unpacking.
• White box
  • bbox defined in stackSentinel.py.
• Lower right figure
  • The magenta box for B4 in 20200527 overlaps the white box, but the cyan box for 20200509 does not. Hence B4 is included for unpacking for 20200527 (master) but not 20200509.
• Of the 70 cities we processed around the world, we see this in 7 cities (5/7 in Europe).

We see this effect on our fully geocoded paired products if there is:

1. Mix of S1A and S1B acquisitions in stack
2. Any of the slave dates have less bursts than master date

Despite this, the ISCE stack proc. algorithm to filter bursts ensures that the bursts extracted cover 100% of the bbox defined in stackSentinel.py. So, if we just limit our geocoding region to only cover the area defined by the bbox of stackSentinel.py, the area with this issue will be cropped out.

Some users, however, would like to make use of all data that was processed by the stack processor, hence I am reflecting this issue here for reference. Hopefully, this issue could be addressed when unpacking the data at run_1 and run_2.

Attached is the KMZ for the above example of burst boundaries (from plotBursts.py), approximated burst boundaries, and bbox defined.

bucharest.kmz.zip

[shitong01]

FYI, a temporary resolve we made for this is in extract_stack_valid_region through editing extractCommonValidRegion.py, where it identifies the common bursts between master-slave date pairs for run_6_merge to include after coregistration. However, the current implementation decides the bounds for extraction based on overlaps of each local master-slave pair and does not do it globally (slave-slave) for all dates across the whole stack.

Edited extractCommonValidRegion.py:
https://github.com/earthobservatory/isce2/blob/d3ae179f05d9f3f625dd60e45870ccdd63acb087/contrib/stack/topsStack/extractCommonValidRegion.py

Diffs:
https://github.com/isce-framework/isce2/compare/v2.3.3...earthobservatory:gekko-stack-processor?expand=1#diff-6e3ba28d77dfe64d6eba790b5577f105

^ This change checks for and only includes valid common overlap bursts amongst all dates (Globally) and creates metadata that is used for merging the valid selection in run_6_merge.

[piyushrpt]

The solution you present to handle it during merging is a good one. I would make that an option as described later.

From the plots, the bounding box does exactly what it is supposed to do - cover the area that was requested completely; so things are working as expected. The issue you bring up is more of a convenience.

There are 2 options:

1. The issues you are pointing out can be managed entirely by setting up the right bbox to begin with. This is probably best handled as a pre-processing option that refines the bbox rather than complicating the workflows. From the screenshots, it is clear that if the bbox was setup right - this issue would not have occurred. I would recommend just making these kinds of plots right up front, which ISCE's sentinel-1 parser allows you to easily do and then refine the bbox before deploying the stack processing.

This should probably be done all the time- the same case can occur with a single burst showing up in another swath in some scenes - say at the east/west edge like the one you pointed out earlier - http://earthdef.caltech.edu/boards/4/topics/3283?r=3289#message-3289? There are tools like plotBursts.py which have all the components needed to put together such a preprocessing tool. We often just do this manually for a date, and draw the bbox such that the edges are not too close to the burst boundaries and also pick desired swaths. However, this can be easily automated as well. You only need the annotation files for doing this. It would be even better, if you contribute back such capability.

2. The second option is what you present which is also a good one. You are probably working in regions with consistent coverage and have small regions of interest. For larger and wider area stacks over Asia / S America / Africa, the chances of having inconsistent spatial coverage is high and users would want to get as much coverage as possible and handle missing data at the time-series stage. Reference date is typically picked with best spatial coverage and all other images are coregistered to it with slices missing at the top and bottom. I would say that the easiest thing to do would be able add global common bursts as an option for extracting valid region but not make it mandatory as there are other use cases where that behavior is not desired.

[jdmillstein]

Hi - I am having this exact issue but the modification of [extractCommonValidRegion.py] doesn't render a fix. Do you have any other suggestions, as my bounding box is at a really weird angle to form a good bbox.