Final GUNW 3.0 Packaging Updates

[cmarshak]

This represents the final packaging updates requested by @dbekaert and ACCESS team. Note: the python files I modified were reformatted by ruff (via VS code) and so there appears to be more changes than what is done. Below, I have highlight important changes and spell-out important questions that can be answered before merging.

Here is a sample GUNW run by this branch: link. The command to generate the above product is:

OMP_NUM_THREADS=4
isce2_topsapp --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F \
                                 S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7  \
              --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \
                                 S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2  \
              --frame-id 25502 \
              > topsapp_img.out 2> topsapp_img.err\

Here is the browse imagery:

Note: ionosphere and SET corrections are now default so the correction arguments are not needed in the above command.

Ionosphere

@mgovorcin @dbekaert:

Please review this modification to the staging of the water mask for the ionosphere here. Below are screenshots of the ionosphere layers (ionosphereBurstRamps and ionosphere, respectively) in the product (which can also be seen directly in the GUNW linked above):


I can provide additional ISCE files or you can check the folder on our shared server leffe in /mnt/leffe-data2/cmarshak/Jan2024-topsapp-revision.

Global attributes for relevant mean values

@sssangha @mgovorcin @dbekaert

In an email from David last week, we agree to have global attributes of the following:

• Mean (filtered) coherence masked with water
• Mean (filtered) coherence
• Mean perpendicular baseline*
• Mean parallel baseline*
• Mean incidence angle
• Mean azimuth angle (with ARIA convention) i.e. adding 90 degrees to ISCE2 raster

They are now as global attributes (copied from the panoply netcdf viewer):

:mean_filtered_coherence_without_water_mask = 0.3126283f; // float
:mean_filtered_coherence_with_water_mask = 0.32342613f; // float
:mean_incidence_angle = 28.281992f; // float
:mean_azimuth_angle = -99.18769836425781; // double
:mean_parallel_baseline = 79.44112223102859; // double
:mean_perpendicular_baseline = -130.66025246650725; // double

All the values except for the baselines can be obtained by averaging a band from the ISCE2 raster. The relevant paths (in the merged folder) and the band are captured here. For taking the mean of geocoded ISCE2 rasters, the code is here. I created a simple integration test by staging the merged directory in s3 and using those rasters to generate the means provided. If there are corrections or changes or concerns, please make specific use of this staged data so we can all be on the same page.

For extracting baseline data, you can see this code here that extracts the baselines from the relevant topsProc.xml file and computes an average for perpendicular and parallel baseline values (there are 6 total for each value). An integration test of the extraction process is here in which the baselines are extracted from this sample topsProc.xml.

Water Mask

The water mask used in all the workflows is the ESA Land Cover 2021 map (via tile-mate).

For the ionosphere, as noted above, the code is here. For the browse imagery, it is captured here. For the mean value of the masked coherence, it reuses the browse masking and the relevant lines of code are found here.

To see some improvement of the browse imagery from this PR compare it to the browse image generated the current dev branch:

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[cmarshak]

@mgovorcin - the water mask is obtained through ESA world cover 2021.

I wrote a package (similar to dem-stitcher) that merges the tiles. The permanent water class is what I am using.

Here is a basic example in the demo: https://github.com/OPERA-Cal-Val/tile-mate/blob/dev/notebooks/Basic_Demo.ipynb

And here is how it is used in the code: https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/final_packaging_updates/isce2_topsapp/localize_mask.py#L27-L40

[dbekaert]

@mgovorcin Please review impact in the ionospheric area.

@cmarshak we should document the limitations of our ancillaries somewhere. i.e. DEM, water mask (max latitude bounds etc). Did we also confirm there are no gaps on the water masks etc. We had some issues with the DEM and patched it earlier too,.

[dbekaert]

I all this looks fine to me please do some testing to confirm this is all as expected.
Look at GUNW created from ASC,DESC and also for the north and southern hemisphere.

[dbekaert]

LGTM

[dbekaert]

@mgovorcin Please open these global variables and ensure this is meeting your expectations.

[cmarshak]

@dbekaert asked for the following:

1. to check whether the GUNW cubes (for angles) and comparable rasters (coherence) had similar means to those computed from the ISCE2 geocoded rasters as done for the global variable. This has been checked using the GUNW above and is now included as an integration test, comparing the respective means from the GUNW with those from the geocoded rasters output from ICSE2 and ensuring they are within acceptable range for the particular example. For coherence, the mean values are within .01. For angles, the means are within 1 degree.
2. Include mean_unfiltered_coherence_with/without_water_mask as global variables. These too are apart of the integration test from 1.

For larger scale studies, let's use hyp3 after this has been merged.

[cmarshak]

Also, the water_mask attribute now looks like:

:water_mask = "water_mask_derived_from_esa_world_cover_2021_10m.geo";

[cmarshak]

@cmarshak we should document the limitations of our ancillaries somewhere. i.e. DEM, water mask (max latitude bounds etc). Did we also confirm there are no gaps on the water masks etc. We had some issues with the DEM and patched it earlier too,

Regarding DEM and water mask availability, if the auxiliary datasets required by the plugin are not available over a particular area, each separate package raises an exception:

For the water mask: https://github.com/OPERA-Cal-Val/tile-mate/blob/dev/src/tile_mate/exceptions.py#L5
For the dem: https://github.com/ACCESS-Cloud-Based-InSAR/dem-stitcher/blob/dev/dem_stitcher/exceptions.py#L1

Each plugin will get as much of the tiles that are available over a requested area and raises warnings if the requested bounds are outside of the tiles that are available.

I think this is a good approach.

[mgovorcin]

@cmarshak

1. Global Variables look good to me.

@dbekaert : One minor component, take it or leave it, think might be more instructive for common user to use standard convention for azimuth angle(which would be angle from North) rather than ISCE2 (from east), e.g. this example has azimuth -99deg w.r.t East that is 351deg w.r.t North (ascending track).

2. As long as water mask is in the right format which it appears to be, ionosphere should not be affected.
   @cmarshak I would also test it over the land without the water, to make sure there are not artifacts.
   Do not know how this mask look like, but remember that SRTM WBD did not have tiles over the land far from coast that were filled by using txt file within ISCE2.

3. Testing : can you make a small stack (~10 GUNWs) over the same aoi, so I can try out to ingest new global variables in ariatools-contrib

@cmarshak changes looks great to me, thanks

[cmarshak]

@dbekaert : One minor component, take it or leave it, think might be more instructive for common user to use standard convention for azimuth angle(which would be angle from North) rather than ISCE2 (from east), e.g. this example has azimuth -99deg w.r.t East that is 351deg w.r.t North (ascending track).

@mgovorcin - I was told as long as we are consistent (even if consistently wrong) that's all that matters.

That's what this test illustrates:

https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/tests/test_packaging.py#L77-L103

Specifically, the mean values captured as global variables have the same means approximately as obtained from the rasters/cubes in the GUNW itself.