displaying the correct area

[falkamelung]

When a subset is processed the entire frame is shown in the overview window. Would it be possible to show only the area of processed? Here two examples:

1st example (Mauna Loa):

https://insarmaps.miami.edu/start/19.6610/-154.7935/8.0000?flyToDatasetCenter=false&startDataset=CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481

I think we need to check for the existence of the mintpy.subset.lalo attribute (and miaplypy.subset.lalo which has preference)) options and use if given:

iinfo.py ./MaunaLoaCskDT91/mintpy/CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481.he5 | grep sub
  mintpy.save.hdfEos5.subset                  yes
  mintpy.subset.lalo                          19.40:19.51,-155.64:-155.50

Second example (Miami):
129.213.120.104/start/25.78/-80.3/14.0?flyToDatasetCenter=true&startDataset=S1_IW3_048_0081_0082_20150921_20211112_SM_PS

Other notes:

Occasionally the subset area is properly displayed. This could be the mintpy.save.hdfEos5.subset = yes option when creating the S1* files, but I need to find an example to check. See this example for a blue subset box:

[falkamelung]

To better address this I have prepared 2 datasets and found that mintpy.save.hdfEos5.subset does not have any effect. So if mintpy.subset.lalo or minopy.subset.lalo are given the frame of a subset area should be displayed. There are two possibilities. One is to use the coordinates in the subset.lalo attribute. This will show a horizontal rectangle, but this is not quite correct as the rectangle should be inclined. The second one is to get the corner coordinates from the latitude and longitude fields given in geometryRadar.h5. I think this would show the inclined rectangles. But I am not quite sure and there are complexities with the subsetting.

So it would be good to support both options so I can carefully try which one works better and then we set this one as default.

• subset0: mintpy.subset.lalo selected. The area covered is not shown properly.

/data/HDF5EOS/subset0unittestGalapagosSenDT128/mintpy

mintpy.subset.lalo                = -0.86:-0.81,-91.19:-91.13  
#mintpy.save.hdfEos5.subset        = yes  

https://insarmaps.miami.edu/start/-0.5535/-90.9440/9.0500?flyToDatasetCenter=false&startDataset=S1_IW1_128_0596_0597_20160605_XXXXXXXX_subset0

• subset: mintpy.subset.lalo selected, mintpy.save.hdfEos5.subset I= yes:

mintpy.subset.lalo                = -0.86:-0.81,-91.19:-91.13    
mintpy.save.hdfEos5.subset        = yes  

[stackTom]

Is it possible to visualize these using mintpy viewer? That way we can choose the right attribute and I can add it to the site.

[falkamelung]

Yes, absolutely. If we have a S1* file in radar coordinates it is easy. The first/last column/row values give the lat/longs of the edges. If the S1* file is in geo-coordinates it is more complicated. We probably should take the smallest longitude/latitude largest latitude/longitude but it may be difference for descending (southward traveling) and ascending (northward traveling) orbits (sensor always looks to right).

- geometry file in radar coordinates
view.py geometryRadar.h5

- S1 file in geo-coordinates:*

subsetunittestGalapagosSenDT128/mintpy
view.py S1_IW1_128_0596_0597_20160605_XXXXXXXX_S00887_S00783_W091207_W091105_subset.he5 latitude

- S1 file in radar coordinates (Miami):*

view.py S1_IW3_048_0081_0082_20160927_20210621_N25862_N25907_W080152_W080108_PS.he5 latitude
view.py S1_IW3_048_0081_0082_20160927_20210621_N25862_N25907_W080152_W080108_PS.he5 latitude longitude

[stackTom]

So the mintpy library has no way of extracting this? Right now we just use scene_footprint and data_footprint.

[falkamelung]

How do we currently do it: Are we using data_footprint, and if not given scene_footprint? I suspect that MintPy has a bug and does not alway properly populate these attributes. Here where these variables are in MintPy (only save_hdf5eos)

 grep data_footprint *  */*  */*/*  */*/*/* */*/*/*/*

docs/hdfeos5.md:+   **data_footprint:** WKT formatted polygon outlining the area covered by the data [precise; for geocoded file only; auto-grabbed from Y/X_FIRST/STEP]. This is temporary and should be merged into scene_footprint.
mintpy/save_hdfeos5.py:        unavco_meta['data_footprint'] = "POLYGON((" + ",".join(
mintpy/save_hdfeos5.py:        print('Input file is not geocoded, no data_footprint without X/Y_FIRST/STEP info.')

 grep scene_footprint *  */*  */*/*  */*/*/* */*/*/*/*

docs/hdfeos5.md:+   **data_footprint:** WKT formatted polygon outlining the area covered by the data [precise; for geocoded file only; auto-grabbed from Y/X_FIRST/STEP]. This is temporary and should be merged into scene_footprint.
docs/hdfeos5.md:+   **scene_footprint:** WKT formatted polygon outlining the area covered by the data [coarse; auto-grabbed from LON/LAT_REF1/2/3/4]

mintpy/save_hdfeos5.py:    unavco_meta['scene_footprint'] = "POLYGON((" + ",".join(

[falkamelung]

There is a bug for the footprint display. The footprint is displayed differently depending on how many nearly-identical datasets are ingested:

Here an example with two datasets. Both files have both footprints:

info.py MaunaLoaCskDT91/mintpy/CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481.he5 | grep foot
  data_footprint                              POLYGON((-155.66571208496094 19.5403611328125,-155.48131208496093 19.5403611328125,-155.48131208496093 19.3675611328125,-155.66571208496094 19.3675611328125,-155.66571208496094 19.5403611328125))
  scene_footprint                             POLYGON((-155.2838673797385 19.626161537381275,-155.3552354077705 19.170555714889275,-155.76423518623707 19.24529066803371,-155.66825789547767 19.696110361106044

info.py MaunaLoaHighResCskDT91/mintpy/CSK_SM_091_0000_20201007_20210706.he5 | grep foot
  data_footprint                              POLYGON((-155.76978383789063 19.698523385620117,-155.28958383789063 19.698523385620117,-155.28958383789063 19.171523385620116,-155.76978383789063 19.171523385620116,-155.76978383789063 19.698523385620117))
  scene_footprint                             POLYGON((-155.2896716107029 19.628400383926436,-155.36080927352575 19.17169429989824,-155.76966094184132 19.24639314353436,-155.67377575294623 19.698287975508965,-155.2896716107029 19.628400383926436))

I started off without any CSK_SM_091* file.
Then I ingested:
MaunaLoaCskDT91/mintpy/CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481.he5
The data_footprint is not shown:

Then I removed it and ingested: CSK_SM_091_0000_20201007_20210706
Again, the data foot print is not shown:

Then I ingested the first file again ('CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481') and now the data_footprint is shown.:

So it shows different/more footprints if more than one file for the same orbit is in the system.

To what I understand, It should display data_footprint only if `subset` is given.Else scene_footprint. For the file where no subset is used the attribute dow not exist:

info.py MaunaLoaHighResCskDT91/mintpy/CSK_SM_091_0000_20201007_20210706.he5 | grep subset

For the subsetted with subset.lalo is given:

info.py MaunaLoaCskDT91/mintpy/CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481.he5 | grep subset
mintpy.save.hdfEos5.subset yes
mintpy.subset.lalo 19.40:19.51,-155.64:-155.50

[stackTom]

I've fixed the ingest script so it ingests the mintpy.subset.Lalo attribute if it's there. Can you ingest a couple of datasets so I can test? I'd do it but I will be busy until I get home around 7 pm today.

[falkamelung]

I re-ingested three datasets. I hope this worked. I used the version if you do s.bop

129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481

129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_048_0081_0082_20150921_20211112_SM_PS

129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_048_0081_0082_20150921_20211112_SM

[stackTom]

I only see subset Lalo in 1 area (one that I ingested when I first sent the email). Did you git pull? Maybe that’s why the ones you did didn’t get infested. Alfredo

…

Sent from my iPhone

On Aug 16, 2022, at 10:44 AM, Falk Amelung ***@***.***> wrote:  I re-ingested three datasets. I hope this worked. I used the version if you do s.bop 129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481 129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_048_0081_0082_20150921_20211112_SM_PS 129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_048_0081_0082_20150921_20211112_SM — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[falkamelung]

I re-ingested another one in the case I did something stupid. Please try those two:

129.213.120.104/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481

http://129.213.120.104/start/25.8878/-80.1175/12.9119?flyToDatasetCenter=false&startDataset=S1_IW3_048_0081_0082_20191111_20220710_PS&minScale=-0.4&maxScale=0.4

[stackTom]

Can you try? Might be working. Didn't completely understand your message. It is pretty clear, but before we had more logic for determining subsets. Not sure if the above will break it.

[stackTom]

For example, we have lost the grouping of subsets now (first picture has subsets grouped, second picture is with new logic and subsets aren't grouped. Maybe we just need these datasets to be reingested with mintpy.subset.lalo?):

[falkamelung]

1. I am a bit confused on whether it works. Today in the morning, I saw it working for Hawaii, but now it is not working anymore? See example for Miami ( Miami2SenAT48/minopy_Surfside_2019-2022/single_reference_network/S1_IW3_048_0081_0082_20191111_20220710_PS.he5
   Did you switch back to the previous version because the grouping is gone?

2. I don't understand why the grouping is gone. I thought everything with the same orbit and frame number is grouped ?

ls CSK_SM_010_0000_* 
ls S1_IW2_124_0059_0063_*

[falkamelung]

We actually should not display the mintpy.subset.lalo coordinates but use it only to determine whether a dataset is a subset. If it its, we should display data_footprint

info.py ./MaunaLoaCskDT91/mintpy/CSK_SM_091_0000_20180916_XXXXXXXX_N19368_N19540_W155666_W155481.he5 | grep footprint
  data_footprint                              POLYGON((-155.66571208496094 19.5403611328125,-155.48131208496093 19.5403611328125,-155.48131208496093 19.3675611328125,-155.66571208496094 19.3675611328125,-155.66571208496094 19.5403611328125))
  scene_footprint                             POLYGON((-155.2838673797385 19.626161537381275,-155.3552354077705 19.170555714889275,-155.76423518623707 19.24529066803371,-155.66825789547767 19.696110361106044,-155.2838673797385 19.626161537381275))

./info.py KilaueaSenDT87/mintpy/S1_IW12_087_0527_0531_20180902_XXXXXXXX.he5 | grep foot
  data_footprint                              POLYGON((-156.28374045410158 20.23051779785156,-154.39894045410156 20.23051779785156,-154.39894045410156 18.804917797851562,-156.28374045410158 18.804917797851562,-156.28374045410158 20.23051779785156))
  scene_footprint                             POLYGON((-154.41127740074518 19.931689920895348,-154.64631706760304 18.80632677313162,-156.27163071971506 19.104601599484138,-156.04853475648636 20.228506003547817,-154.41127740074518 19.931689920895348))

Unfortunately, the high-res S1_* files don't have data_footprint:

info.py Miami2SenAT48/minopy_SM/S1_IW3_048_0081_0082_20150921_20211112_SM2_PS.he5 | grep foot
  scene_footprint                             POLYGON((-80.27288426610589 25.573931501092392,-80.37628346800055 26.08861098124306,-79.5870234098245 26.21499965809593,-79.48708567150678 25.700746626289984,-80.27288426610589 25.573931501092392))

We have to modify save_hdf5eos.py to fill it. Unfortunately my Python is not good enough to get changes quickly approved by Yunjun. Can you try? Here how to run it:

cd /data/HDF5EOS/Miami2SenAT48/minopy_SM
save_hdfeos5.py timeseries_*.h5 --tc maskTempCoh.h5 --asc maskTempCoh.h5 -m  maskPS.h5 -g corrected_geometry/geometryRadar.h5 -t smallbaselineApp.cfg --suffix SM2_PS

h5file=S1_IW3_048_0081_0082_20150921_20211112_SM2_PS.he5
rm -rf JSON ; hdfeos5_2json_mbtiles.py $h5file ./JSON; json_mbtiles2insarmaps.py -u insaradmin -p Insar123 --host insarmaps.miami.edu -P rsmastest -U rsmas\@gmail.com --json_folder JSON --mbtiles_file JSON/${h5file%.*}.mbtiles ; echo "$URL/start/25.78/-80.3/11.0?flyToDatasetCenter=true&startDataset=${h5file%.*}"

(if you are successful there is another minor modification needed for save_hdfeos5, I have to look it up, though)

The alternative is to populate data_footprint in hdf5_2_json?

[stackTom]

Yes, I undid the changes last night. I have fixed it now so that the grouping is still there. Can you check and let me know if it's correct?

Also, "S1_IW3_048_0081_0082_20191111_20220710_PS" has mintpy.subset.lalo yet it has no data_footprint.

*Edit: Oh, I realize this is a high res file as you mentioned above.

[falkamelung]

Yes, the grouping looks good again. Thank you!

I looked a bit at the code for data_footprint:
https://github.com/insarlab/MintPy/blob/36a083521b39d9fb557741f9c5d703b7e42448c3/mintpy/save_hdfeos5.py#L242-L245

I would think that it is possible to get the data_footprint from the min and max from the lat/long arrays.

Maybe we should first convince ourselves that it works, and before doing a PR to mintpy we ask Yunjun whether he would accept it.

[stackTom]

If the grouping looks good and the new logic looks good as well, can we close this issue and open a new one to add data_footprint to save_hdfeos5?

[falkamelung]

yes, sure.
I am actually wondering whether we always should display data_footprint instead of scene_footprint. It could be that we did not have it when we started. Do you remember?
On the other hand, "don't fix it if it ain't broke".

[stackTom]

I don't remember. I agree we shouldn't change it if it's working.

Closing this and will open new issue for modifying the mintpy script.

[stackTom]

I don't remember. I agree we shouldn't change it if it's working.

Closing this and will open new issue for modifying the mintpy script in the mintpy repo.

[stackTom]

Yes, the grouping looks good again. Thank you!

I looked a bit at the code for data_footprint:

https://github.com/insarlab/MintPy/blob/36a083521b39d9fb557741f9c5d703b7e42448c3/mintpy/save_hdfeos5.py#L242-L245

I would think that it is possible to get the data_footprint from the min and max from the lat/long arrays.

Maybe we should first convince ourselves that it works, and before doing a PR to mintpy we ask Yunjun whether he would accept it.

I think it should work. @yunjunz what do you think? Should I open a new issue and implement this?

[yunjunz]

I did not read all comments in this issue, as it's pretty long.

Should I open a new issue and implement this?

If "this" means "specifying the data_footprint for datasets in the radar coordinates", yes, it makes sense to me, as we are now ingesting radar-coded products. Please go for it.