Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ASF HyP3 Support #540

Closed
forrestfwilliams opened this issue Mar 22, 2021 · 60 comments
Closed

ASF HyP3 Support #540

forrestfwilliams opened this issue Mar 22, 2021 · 60 comments

Comments

@forrestfwilliams
Copy link
Collaborator

The ASF HyP3 on-demand InSAR service is great option for creating interferograms, but MintPy is not currently set up to ingest these products. I propose creating a prep_hyp3.py workflow that would allow for the ingestion of these products. I'm happy to perform this work, but could use guidance on preparing metadata.

@forrestfwilliams
Copy link
Collaborator Author

HyP3 product are projected to the local UTM zone. I imagine this could cause issues for the LAT/LON_START and LAT/LON_REF1/2/3/4 attributes. Can MintPy accommodate this, or must they all be reprojected?

@yunjunz
Copy link
Member

yunjunz commented Mar 23, 2021

No need for re-projection. MintPy should be able to handle X/Y_STEP = meters and its corresponding X/Y_FIRST.

Since most of the operations in mintpy do not care about coordinates, special attention to the coordinate system is only needed during geocoding and plotting in map coordinates, as far as I remember. Both of these two functionalities have been tested before, but not maintained via regular testing, so some functionalities may not work out of the box. But they should be fixed easily.

@yunjunz
Copy link
Member

yunjunz commented Mar 23, 2021

A HyP3 + MintPy support would be awesome @forrestfwilliams. I am not familiar with HyP3 results. Here is a general guideline to prepare datasets for MintPy: https://mintpy.readthedocs.io/en/latest/FAQs/. I am more than happy to answer any question you may have related to mintpy.

@forrestfwilliams
Copy link
Collaborator Author

Thanks @yunjunz! Currently HyP3 only provides the average perpendicular baseline, and not the baseline for the top and bottom of the interferogram. Should I see if I can get this data, or is it permissible to have the same average baseline for both P_BASELINE_TOP_HDR and P_BASELINE_BOTTOM_HDR?

@yunjunz
Copy link
Member

yunjunz commented Mar 23, 2021

Use the same value is perfectly fine.

@forrestfwilliams
Copy link
Collaborator Author

forrestfwilliams commented Mar 23, 2021

Hi @yunjunz, I've successfully created a prep_hyp3.py file that generates the needed rsc files for each interferogram. Hyp3 provides interferograms as tifs that can be viewed using view.py, but once an rsc file is generated for an interferogram, I believe that readfile.read assumes that the interferogram is now in a binary format, which causes an error and the message: "Unknown InSAR processor: hyp3". Do you have any thoughts on how to correct this?

@forrestfwilliams
Copy link
Collaborator Author

Disregard the previous comment, I was able to track down the error.

@yunjunz
Copy link
Member

yunjunz commented Mar 23, 2021

The discussion on the GMTSAR support (#452) could be a good reference here. We have implemented it here: https://github.com/yunjunz/MintPy/tree/gmtsar.

forrestfwilliams added a commit to forrestfwilliams/MintPy that referenced this issue Mar 24, 2021
HyP3 is an on-demand service for producing Sentinel-1 interferograms.
prep_hyp3.py provides a utility to produce MintPy-readable rsc files
so that HyP3 data can be loaded in MintPy.

See Issue: insarlab#540
forrestfwilliams added a commit to forrestfwilliams/MintPy that referenced this issue Mar 24, 2021
Adds hyp3 to load_data.py processor list, and sets up reference
to prep_hyp3 when hyp3 is the processor.

See Issue: insarlab#540
forrestfwilliams added a commit to forrestfwilliams/MintPy that referenced this issue Mar 24, 2021
@yunjunz
Copy link
Member

yunjunz commented Apr 20, 2021

Here is the to-do list I could think of for full support of HyP3+MintPy workflow:

  1. The 2D matrix of pixel-wised incidence angle [required for tropospheric correction and DEM error correction].
  2. The connected components for each interferogram [optional but recommended; required for unwrapping error correctoin]
  3. A 2D matrix of water/land mask [optional but recommended]
  4. Add attributes: HEIGHT (satellite altitude in m) and EARTH_RADIUS, both of them are currently hardwired in the code here.

@talogan
Copy link

talogan commented Apr 21, 2021

Thanks for the to-do list. We are going to start looking into this soon.

I note that the first link in this issue is to HyP3 version 1, which is our beta system. We've recently released our InSAR GAMMA process in HyP3 version 2, a more stable system that can be accessed through the ASF Data Search engine Vertex, through an API, or though a python SDK.

Documentation for HyP3 v2 is available at the HyP3 docs website.

@forrestfwilliams
Copy link
Collaborator Author

@talogan and @yunjunz, thank you for continuing this discussion! I'm looking forward to fully integrating HyP3 products into MintPy, and am happy to keep contributing as we continue this process.

@yunjunz
Copy link
Member

yunjunz commented Apr 22, 2021

Thank you @talogan for pointing out the link, it hasd been fixed in #545. Please feel free to add more detailed descriptions/tutorials/links via PRs. Both the example directory structure and the prep_hyp3.py help msg are good places for them.

@talogan
Copy link

talogan commented Apr 28, 2021

ASF is working in the following:

  • we've added the spacecraft height and earth radius at the nadir to our metadata
  • we're adding the option to include an incidence angle map in the product bundle
  • we're adding the option to select water masking
  • we're adding the option to include the DEM into the product bundle

Unless I missed something, the only items we're missing are the connected components and the slant range distance. I have questions about these:

  • GAMMA's MCF phase unwrapping implementation doesn't mention connected components. Does that apply to MCF? I know that it does for snaphu, but I couldn't find any mention in the GAMMA documentation of connected components.

  • For the slant range distance, you need only the first slant range and the slant range spacing, I believe. Do you actually need a raster image, or would the start, [middle, end] slant range plus the range pixel spacing suffice? It seems to me that the slant range will be the same for every line in the SAR file.

@yunjunz
Copy link
Member

yunjunz commented Apr 28, 2021

You are right. There is no connected components product from GAMMA phase unwrapping, as far as I know.

As for the slant range distance, we could calculate it based on the 2D incidence angle, accurate spacecraft height and earth radius (https://github.com/insarlab/MintPy/blob/main/mintpy/utils/utils0.py#L139). So it's okay to not have it.

For further clarification, we are using the geocoded products from HyP3, so the pixel-wised 2D slant range distance can not be calculated/predicted from the few parameters. However, it will still be good to have those parameters available for other purposes.

@talogan
Copy link

talogan commented May 14, 2021

As of today, the HyP3 InSAR products include the following (the options are not yet available in the Vertex, but are accessible via the other interfaces):

  • Spacecraft height, Earth radius, Slant range near, Slant range center, and Slant range far are now in the .txt metadata file
  • an option to include the incidence angle map into the product bundle now exists
  • an option to include the DEM file used during processing now exists.

We're still working on water masking.

One point of clarification @forrestfwilliams - would you prefer that the water mask be applied to the InSAR pairs when they are processed by HyP3, or would you prefer to just get a mask file and not have it already applied to the data?

@yunjunz
Copy link
Member

yunjunz commented May 16, 2021

This is great @talogan!

Just jump in for the water mask :)

I would think an option to download a water mask is definitely desired. This would be similar to the DEM and incidence angle.

A separate option in the processing to apply the water mask before phase unwrapping would be great too.

@forrestfwilliams
Copy link
Collaborator Author

Hi @talogan this is all great, thanks! Echoing @yunjunz, I think it would be great to have the water mask applied before unwrapping, but just receiving the water mask would be great as well!

@forrestfwilliams
Copy link
Collaborator Author

Hi @yunjunz and @talogan, I've taken a quick look at updating MintPy so that it takes full advantages of the updates to HyP3, but some other projects have taken priority and I won't be able to devote much time to this until June. Hope this isn't too long for you all!

@yunjunz
Copy link
Member

yunjunz commented May 18, 2021

Thank you for looking into this again @forrestfwilliams.

@cirrusasf
Copy link
Contributor

I am trying to make the mintpy work against hyp3. @forrestfwilliams mentioned you have already did a prep_hyp3. May I take a look at it. I followed the guide to arrange the hyp3 output. When I run, I got stuck at step “correct_troposphere”. It complains that Exception: Assertion failed: bbox.north() <= Latitude::NORTH_POLE in check

, which comes from the statement result = self._api("%s/resources/%s" % (self.url, name), request, "POST"). The dem.tif and hyp3 tif files have the projection "WGS 84 / UTM zone 53N". Does this error means that we need longitude/latitude in degree?

@yunjunz
Copy link
Member

yunjunz commented May 19, 2021

Hi @cirrusasf, the error you show seems from the PyAPS code, I assume that you have successfully ingested incidenceAngle to geometryGeo.h5 file.

To your question, yes, PyAPS works on lat/lon only (tropo_pyaps3.py#L714-L716). So some changes are needed in tropo_paps3.py to prepare the pixelwised lat/lon. It's should straightforward to grab the UTM coordinates of the 4 corners, convert them to lat/lon and generate the pixelwised lat/lon from it.

@cirrusasf
Copy link
Contributor

cirrusasf commented May 19, 2021 via email

@yunjunz
Copy link
Member

yunjunz commented May 19, 2021

@cirrusasf It's committed and available already.

@cirrusasf
Copy link
Contributor

cirrusasf commented May 19, 2021 via email

@forrestfwilliams
Copy link
Collaborator Author

@yunjunz, @talogan, @cirrusasf, the new version of prep_hyp3.py will correctly read any HyP3 tif file that begins with the HyP3 naming convention S1xy-aaaaaaaaTbbbbbb_ggggggggThhhhhh_pponnn_INTzz_u_def_ssss and ends with .tif. As long as future water mask/incidence angle/connected components files stick to this formatting, we shouldn't have to make any major changes!

@cirrusasf
Copy link
Contributor

cirrusasf commented Jun 11, 2021 via email

@yunjunz
Copy link
Member

yunjunz commented Jun 13, 2021

There is a good description of the definition of incidence angle from UNAVCO: https://www.unavco.org/education/professional-development/short-courses/course-materials/insar/2008-insar-course-materials/sar_summary.pdf.

@cirrusasf
Copy link
Contributor

cirrusasf commented Jun 15, 2021 via email

@cirrusasf
Copy link
Contributor

Can anybody in MintPy team take a look at our PR insarlab/MintPy-tutorial#22? This gives a tutorial to run mintpy with HyP3 data.

@rbiessel
Copy link

rbiessel commented Jun 21, 2021

Maybe this deserves its own issue, but there are a lot of instances where MintPy assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. In addition, the GPS stations will not load and the input reference point latlon is not converted. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy.

@cirrusasf
Copy link
Contributor

cirrusasf commented Jun 21, 2021 via email

@rbiessel
Copy link

rbiessel commented Jun 21, 2021

Hi Rowan Biessel, I am a little bit confused about what you talked about. Can you point out what the exact problem is. For example, which part of the program can not go through? What does the output giff look like? Jiang

On Mon, Jun 21, 2021 at 11:48 AM Rowan Biessel @.***> wrote: Maybe this deserves its own issue, but there are a lot of instances where MintpY assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#540 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWCWJBICUSWKV42UBQLTT6JPXANCNFSM4ZT26Z7A .

A command such as save_gdal.py velocity.h5 --of Gtiff save_gdal will set the projection of the output geotiff as EPSG:4326 no matter what. Meanwhile, the geotransform associated with velocity.h5 will be in UTMs. As a result, the geotiff won't render at all in QGIS. save_gdal.py needs to read the UTM zone from velocity.h5's attributes and use that instead of 4326.

For example, something like this:

from pyproj import CRS

try:
  zone = h5_file.attrs['UTM_ZONE']
  n_or_s = 'north' if 'N' in zone else 'south'
  zone_n = zone.replace('N', '').replace('S', '')
  crs = CRS.from_string(f'+proj=utm +zone={zone_n} +{n_or_s}')
  projection = crs.to_authority()[1]
except:
  projection = 4326 # If the interferograms are  in longitude/latitude already, the UTM_ZONE attribute won't exist 

...

outRasterSRS = osr.SpatialReference()
outRasterSRS.ImportFromEPSG(int(projection))
outRaster.SetProjection(outRasterSRS.ExportToWkt())

...

It is for sure easier to just reproject HyP3 products to EPSG:4326 before running MintPy

@cirrusasf
Copy link
Contributor

cirrusasf commented Jun 21, 2021 via email

@cirrusasf
Copy link
Contributor

cirrusasf commented Jul 12, 2021 via email

@asjohnston-asf
Copy link
Contributor

@yunjunz Regarding incidence angles, after more review, the lv_theta.tif GeoTIFF we offer via the include_look_vectors option may be sufficient. That GeoTIFF is generated using GAMMA's look_vector utility:

look_vector calculates SAR look-vector direction (pointing towards SAR) in map geometry.
Based on the SAR imaging and DEM geometries the look vector is calculated for each pixel of the DEM/map section specified in the DEM parameter file. The local topography is taken into account.
The look vector elevation angle is defined as the angle between the horizontal surface and the look vector with positive angles indicating sensor positions above the surface.

Elevation is already taken into account, but our lv_theta angle is measuring up from horizontal, rather than down from vertical. I believe a simple pi/2 - lv_theta conversion would give the angle desired by MintPy. Is there a convenient way for MintPy to perform that conversion? Would you prefer ASF include an additional GeoTIFF with the conversion already applied?

We also notice that the impact of elevation on the angle map in your Galapagos example is significantly greater than when we see from HyP3 products over the same area. I'll post an example HyP3 product shortly.

@yunjunz
Copy link
Member

yunjunz commented Jul 30, 2021

@asjohnston-asf the conversion could be applied in mintpy at stackDict.py#L619 since it already handles the radian to degree conversion.

@yunjunz
Copy link
Member

yunjunz commented Aug 14, 2021

Hi ASF team (@cirrusasf, @talogan, @asjohnston-asf, et al.), it seems like the "Heading" attribute in the example dataset that @cirrusasf put here: https://jzhu-hyp3-dev.s3.us-west-2.amazonaws.com/hyp3-mintpy/Ridgecrest.zip is not right: For Sentinel-1 descending track, it should be around -168, while the *.txt file shows 193.0934889.

Update: my miscalculation, the value is right and normal (with 360 in difference). Please ignore my above msg.

@asjohnston-asf
Copy link
Contributor

asjohnston-asf commented Aug 23, 2021

FYI only, I created a sample data set for HyP3 using the same 98 S1 acquisitions (288 interferograms) as the Fernandina volcano data set for ARIA:

wget https://asj-hyp3-dev.s3.us-west-2.amazonaws.com/galapagos.zip
unzip galapagos.zip
cd galapagos
smallbaselineApp.py hyp3.cfg

Not intended to be hosted there long-term, but I've made it available for now in case anyone wants to look at a HyP3 time series for a familiar data set.

@yunjunz yunjunz added this to the UTM coordinate support milestone Sep 19, 2021
@yunjunz
Copy link
Member

yunjunz commented Oct 28, 2021

Close this issue as all the proposed changes are completed! Kudos to all!

@yunjunz yunjunz closed this as completed Oct 28, 2021
@forrestfwilliams
Copy link
Collaborator Author

Happy to see this completed!

@jhkennedy
Copy link
Collaborator

🥳

@yunjunz do you have plans for a release soon since a lot of this (and a lot more!) is main but not yet released? With a release we can get everything in the conda-forge install as well.

@yunjunz
Copy link
Member

yunjunz commented Oct 28, 2021

@jhkennedy yes, a release is planned for next month, for the exact reason you said.

There are a few things on the to-do list before that: https://github.com/insarlab/MintPy/wiki/To-do-list#version-132

@XJQiao
Copy link

XJQiao commented Dec 10, 2021

Hi guys,

I am trying to play with Mintpy and Hyp3 on Google Colab these days.

I am using "!smallbaselineApp.py mintpy/RidgecrestSenDT71.txt", but got stuck at the step of "modify network" (as shown in the end). Any hints on this issue?

Appreciated!

Best,
Jun

******************** step - modify_network ********************
WARNING: "coherence" is missing in file /content/RidgecrestSenDT71/inputs/ifgramStack.h5
Input data seems to be geocoded. Lookup file not needed.
generate /content/RidgecrestSenDT71/waterMask.h5 from /content/RidgecrestSenDT71/inputs/geometryGeo.h5 for conveniency
create HDF5 file: /content/RidgecrestSenDT71/waterMask.h5 with w mode
create dataset /waterMask of bool       in size of (1125, 1500)         with compression=None
finished writing to /content/RidgecrestSenDT71/waterMask.h5

modify_network.py /content/RidgecrestSenDT71/inputs/ifgramStack.h5 -t /content/RidgecrestSenDT71/smallbaselineApp.cfg
No lookup table info range/lat found in files.
read options from template file: smallbaselineApp.cfg
No input option found to remove interferogram
Keep all interferograms by enable --reset option
--------------------------------------------------
reset dataset 'dropIfgram' to True for all interferograms for file: /content/RidgecrestSenDT71/inputs/ifgramStack.h5
All dropIfgram are already True, no need to reset.

plot_network.py /content/RidgecrestSenDT71/inputs/ifgramStack.h5 -t /content/RidgecrestSenDT71/smallbaselineApp.cfg --nodisplay -d coherence -v 0.2 1.0
read options from template file: smallbaselineApp.cfg
read temporal/spatial baseline info from file: /content/RidgecrestSenDT71/inputs/ifgramStack.h5
calculating spatial mean of coherence in file /content/RidgecrestSenDT71/inputs/ifgramStack.h5 ...
read mask from file: waterMask.h5
Traceback (most recent call last):
  File "/usr/local/bin/smallbaselineApp.py", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1270, in main
    app.run(steps=inps.runSteps)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1038, in run
    self.run_network_modification(sname)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 434, in run_network_modification
    mintpy.plot_network.main(iargs)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/plot_network.py", line 245, in main
    inps = read_network_info(inps)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/plot_network.py", line 167, in read_network_info
    checkAoi=False)[0]
  File "/usr/local/lib/python3.7/dist-packages/mintpy/utils/utils1.py", line 256, in spatial_average
    threshold=threshold)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/objects/stack.py", line 794, in spatial_average
    dset = f[datasetName]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/group.py", line 288, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'coherence' doesn't exist)"

@forrestfwilliams
Copy link
Collaborator Author

It looks like your ifgramStack.h5 may have failed to load correctly and you don't have the required coherence dataset. Could you run info.py inputs/ifgramStack.h5 and paste the output here?

@XJQiao
Copy link

XJQiao commented Dec 10, 2021

@forrestfwilliams Appreciated you!

After I rerun the code once again this morning, the bug was gone. I guess I passed a wrong path of input files, leading to no output file of ifgramStack.

Then, I encountered the next issue (attached in the end). I will look into this later, which seems due to no output file of avgPhaseVelocity.h5.

Have a nice day!

Best,
Jun

******************** step - quick_overview ********************
Input data seems to be geocoded. Lookup file not needed.

temporal_average.py /content/RidgecrestSenDT71/inputs/ifgramStack.h5 --dataset unwrapPhase -o /content/RidgecrestSenDT71/avgPhaseVelocity.h5 --update
output file: /content/RidgecrestSenDT71/avgPhaseVelocity.h5
--------------------------------------------------
update mode: ON
1) output file /content/RidgecrestSenDT71/avgPhaseVelocity.h5 NOT exist.
run or skip: run.
calculate the temporal average of unwrapPhase in file /content/RidgecrestSenDT71/inputs/ifgramStack.h5 ...
Traceback (most recent call last):
  File "/usr/local/bin/smallbaselineApp.py", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1270, in main
    app.run(steps=inps.runSteps)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1044, in run
    self.run_quick_overview(sname)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 497, in run_quick_overview
    mintpy.temporal_average.main(iargs)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/temporal_average.py", line 102, in main
    ut.temporal_average(inps.file, datasetName=inps.datasetName, outFile=inps.outfile)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/utils/utils1.py", line 355, in temporal_average
    dataMean = ifgramStack(File).temporal_average(datasetName=datasetName)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/objects/stack.py", line 933, in temporal_average
    ref_val = dset[ifgram_flag, self.refY, self.refX]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/dataset.py", line 777, in __getitem__
    selection = sel.select(self.shape, args, dataset=self)
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/selections.py", line 82, in select
    return selector.make_selection(args)
  File "h5py/_selector.pyx", line 272, in h5py._selector.Selector.make_selection
  File "h5py/_selector.pyx", line 183, in h5py._selector.Selector.apply_args
TypeError: Indexing arrays must have integer dtypes

@ditafaith
Copy link

HyP3 product are projected to the local UTM zone. I imagine this could cause issues for the LAT/LON_START and LAT/LON_REF1/2/3/4 attributes. Can MintPy accommodate this, or must they all be projected?

Hi @forrestfwilliams and @yunjunz

It seems this issue is still relevant since we can't create a subset using lat/lon in MintPy using HyP3 products.
Also, when we visualize the end results.
Anyway, will ASF produce the connected components file to support MintPy perform unwrapping error correction?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants