-
Notifications
You must be signed in to change notification settings - Fork 256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ASF HyP3 Support #540
Comments
HyP3 product are projected to the local UTM zone. I imagine this could cause issues for the LAT/LON_START and LAT/LON_REF1/2/3/4 attributes. Can MintPy accommodate this, or must they all be reprojected? |
No need for re-projection. MintPy should be able to handle Since most of the operations in mintpy do not care about coordinates, special attention to the coordinate system is only needed during geocoding and plotting in map coordinates, as far as I remember. Both of these two functionalities have been tested before, but not maintained via regular testing, so some functionalities may not work out of the box. But they should be fixed easily. |
A HyP3 + MintPy support would be awesome @forrestfwilliams. I am not familiar with HyP3 results. Here is a general guideline to prepare datasets for MintPy: https://mintpy.readthedocs.io/en/latest/FAQs/. I am more than happy to answer any question you may have related to mintpy. |
Thanks @yunjunz! Currently HyP3 only provides the average perpendicular baseline, and not the baseline for the top and bottom of the interferogram. Should I see if I can get this data, or is it permissible to have the same average baseline for both P_BASELINE_TOP_HDR and P_BASELINE_BOTTOM_HDR? |
Use the same value is perfectly fine. |
Hi @yunjunz, I've successfully created a prep_hyp3.py file that generates the needed rsc files for each interferogram. Hyp3 provides interferograms as tifs that can be viewed using view.py, but once an rsc file is generated for an interferogram, I believe that readfile.read assumes that the interferogram is now in a binary format, which causes an error and the message: "Unknown InSAR processor: hyp3". Do you have any thoughts on how to correct this? |
Disregard the previous comment, I was able to track down the error. |
The discussion on the GMTSAR support (#452) could be a good reference here. We have implemented it here: https://github.com/yunjunz/MintPy/tree/gmtsar. |
HyP3 is an on-demand service for producing Sentinel-1 interferograms. prep_hyp3.py provides a utility to produce MintPy-readable rsc files so that HyP3 data can be loaded in MintPy. See Issue: insarlab#540
Adds hyp3 to load_data.py processor list, and sets up reference to prep_hyp3 when hyp3 is the processor. See Issue: insarlab#540
Here is the to-do list I could think of for full support of HyP3+MintPy workflow:
|
Thanks for the to-do list. We are going to start looking into this soon. I note that the first link in this issue is to HyP3 version 1, which is our beta system. We've recently released our InSAR GAMMA process in HyP3 version 2, a more stable system that can be accessed through the ASF Data Search engine Vertex, through an API, or though a python SDK. Documentation for HyP3 v2 is available at the HyP3 docs website. |
Thank you @talogan for pointing out the link, it hasd been fixed in #545. Please feel free to add more detailed descriptions/tutorials/links via PRs. Both the example directory structure and the prep_hyp3.py help msg are good places for them. |
ASF is working in the following:
Unless I missed something, the only items we're missing are the connected components and the slant range distance. I have questions about these:
|
You are right. There is no connected components product from GAMMA phase unwrapping, as far as I know. As for the slant range distance, we could calculate it based on the 2D incidence angle, accurate spacecraft height and earth radius (https://github.com/insarlab/MintPy/blob/main/mintpy/utils/utils0.py#L139). So it's okay to not have it. For further clarification, we are using the geocoded products from HyP3, so the pixel-wised 2D slant range distance can not be calculated/predicted from the few parameters. However, it will still be good to have those parameters available for other purposes. |
As of today, the HyP3 InSAR products include the following (the options are not yet available in the Vertex, but are accessible via the other interfaces):
We're still working on water masking. One point of clarification @forrestfwilliams - would you prefer that the water mask be applied to the InSAR pairs when they are processed by HyP3, or would you prefer to just get a mask file and not have it already applied to the data? |
This is great @talogan! Just jump in for the water mask :) I would think an option to download a water mask is definitely desired. This would be similar to the DEM and incidence angle. A separate option in the processing to apply the water mask before phase unwrapping would be great too. |
Thank you for looking into this again @forrestfwilliams. |
I am trying to make the mintpy work against hyp3. @forrestfwilliams mentioned you have already did a prep_hyp3. May I take a look at it. I followed the guide to arrange the hyp3 output. When I run, I got stuck at step “correct_troposphere”. It complains that Exception: Assertion failed: bbox.north() <= Latitude::NORTH_POLE in check , which comes from the statement result = self._api("%s/resources/%s" % (self.url, name), request, "POST"). The dem.tif and hyp3 tif files have the projection "WGS 84 / UTM zone 53N". Does this error means that we need longitude/latitude in degree? |
Hi @cirrusasf, the error you show seems from the PyAPS code, I assume that you have successfully ingested To your question, yes, |
Hi Insarlab/Mintpy,
Thank you for your reply. I noticed that there is prep_hyp3.py. I want to
play this first to see if I can make mintPy work against hyp3 data.
Jiang
…On Tue, May 18, 2021 at 6:00 PM Zhang Yunjun ***@***.***> wrote:
Hi @cirrusasf <https://github.com/cirrusasf>, the error you show seems
from the PyAPS code, I assume that you have successfully ingested
incidenceAngle to geometryGeo.h5 file.
To your question, yes, PyAPS works on lat/lon only (
tropo_pyaps3.py#L714-L716
<https://github.com/insarlab/MintPy/blob/main/mintpy/tropo_pyaps3.py#L714-L716>).
So some changes are needed in tropo_paps3.py to prepare the pixelwised
lat/lon. It's should straightforward to grab the UTM coordinates of the 4
corners, convert them to lat/lon and generate the pixelwised lat/lon from
it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWHUQIC2MMES4IK3SCDTOMLTPANCNFSM4ZT26Z7A>
.
|
@cirrusasf It's committed and available already. |
Hi Insarlab/Mintpy,
I got stuck at
pa.ECMWFdload(date_list2dload, hour, grib_dir,
model=tropo_model,
snwe=snwe,
flist=grib_files2dload)
it can not download ERA5 data. Perhaps snake needs in degree, not in meter?
Jiang
…On Wed, May 19, 2021 at 9:57 AM Zhang Yunjun ***@***.***> wrote:
It's committed and available already.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWBNO3AA2IHLUGLSBBLTOP3XTANCNFSM4ZT26Z7A>
.
|
@yunjunz, @talogan, @cirrusasf, the new version of |
Hi Insarlab/Mintpy,
I am a little bit confused. The inc_map.tif that the hyp3 produced is the
pixel-wise inc angle tif file. Does mintpy need the inc_angle tif files
from every ifm?
I put the following in the custom.cfg
##---------incAngleFile:
mintpy.load.incAngleFile =
/media/jzhu4/data/hyp3-mintpy/case2/*/*inc_map_clip.tif
the smallbaselineApp can produce the output.
You told me our inc_map.tif was wrong. We will check what exactly the inc
angle is defined in our inc_map.tif.
Jiang
…On Thu, Jun 10, 2021 at 4:44 PM Zhang Yunjun ***@***.***> wrote:
It should be a pixel-wise incidence angle, such as the example below from
the SanFranSenDT42 example dataset from ARIA
<https://github.com/insarlab/MintPy/blob/main/docs/demo_dataset.md#sentinel-1-on-san-francisco-bay-with-aria>
.
[image: inc_angle]
<https://user-images.githubusercontent.com/13143710/121613806-30b01400-ca12-11eb-98ed-300aa54ea025.png>
If you could consider the impact of elevation for each pixel, such as the
example below from the FernandinaSenDT128 example dataset from
ISCE/topsStack
<https://github.com/insarlab/MintPy/blob/main/docs/demo_dataset.md#sentinel-1-on-fernandina-with-isce>,
that would be even better.
[image: inc_angle2]
<https://user-images.githubusercontent.com/13143710/121614376-61447d80-ca13-11eb-941c-e7a8cb9b873a.png>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWAGOEQSGIGW4R24JATTSFL53ANCNFSM4ZT26Z7A>
.
|
There is a good description of the definition of incidence angle from UNAVCO: https://www.unavco.org/education/professional-development/short-courses/course-materials/insar/2008-insar-course-materials/sar_summary.pdf. |
Hi Insarlab/Mintpy,
I pushed the PR insarlab/MintPy-tutorial#22. it is
about the notebook we created to demo how the mintpy work with hyp3 data.
Please take a look at it.
Thank you,
Jiang
…On Thu, Jun 10, 2021 at 4:12 PM Zhang Yunjun ***@***.***> wrote:
Sounds great @cirrusasf <https://github.com/cirrusasf>. For jupyter
notebooks, please push to MintPy-tutorial
<https://github.com/insarlab/MintPy-tutorial> repo.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWGHD6RMA5WUH5OK5ZLTSFIFBANCNFSM4ZT26Z7A>
.
|
Can anybody in MintPy team take a look at our PR insarlab/MintPy-tutorial#22? This gives a tutorial to run mintpy with HyP3 data. |
Maybe this deserves its own issue, but there are a lot of instances where MintPy assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. In addition, the GPS stations will not load and the input reference point latlon is not converted. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy. |
Hi Rowan Biessel,
I am a little bit confused about what you talked about. Can you point out
what the exact problem is. For example, which part of the program can not
go through? What does the output giff look like?
Jiang
…On Mon, Jun 21, 2021 at 11:48 AM Rowan Biessel ***@***.***> wrote:
Maybe this deserves its own issue, but there are a lot of instances where
MintpY assumes a WGS84 projection regardless of the projection of the input
files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work
correctly because they use the UTM zone's geotransform with EPSG 4326. My
current hack-around is to just reproject all of the HyP3 tiffs to lat/lon
before running mintpy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWCWJBICUSWKV42UBQLTT6JPXANCNFSM4ZT26Z7A>
.
|
A command such as For example, something like this:
It is for sure easier to just reproject HyP3 products to EPSG:4326 before running MintPy |
Hi Rowan,
Thank you for clarification. I suggest modifying the save_gdal.py or other
conversion/display/save scripts to make sure they can intake data with geo
projection other than EPSG:4326. I don't suggest converting ifm files and
then ingesting them to MintPy.
Jiang
On Mon, Jun 21, 2021 at 2:46 PM Rowan Biessel ***@***.***>
wrote:
… Hi Rowan Biessel, I am a little bit confused about what you talked about.
Can you point out what the exact problem is. For example, which part of the
program can not go through? What does the output giff look like? Jiang
… <#m_-2245035496201012529_>
On Mon, Jun 21, 2021 at 11:48 AM Rowan Biessel *@*.***> wrote: Maybe this
deserves its own issue, but there are a lot of instances where MintpY
assumes a WGS84 projection regardless of the projection of the input files.
For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly
because they use the UTM zone's geotransform with EPSG 4326. My current
hack-around is to just reproject all of the HyP3 tiffs to lat/lon before
running mintpy. — You are receiving this because you were mentioned. Reply
to this email directly, view it on GitHub <#540 (comment)
<#540 (comment)>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AO3CPWCWJBICUSWKV42UBQLTT6JPXANCNFSM4ZT26Z7A
.
A command such as save_gdal.py velocity.h5 --of Gtiff save_gdal will set
the projection of the output geotiff as EPSG:4326 no matter what.
Meanwhile, the geotransform associated with velocity.h5 will be in UTMs.
save_gdal.py needs to read the UTM zone from velocity.h5's attributes and
use that instead of 4326.
For example, something like this:
from pyproj import CRS
zone = h5_file.attrs['UTM_ZONE']
n_or_s = 'north' if 'N' in zone else 'south'
zone_n = zone.replace('N', '').replace('S', '')
crs = CRS.from_string(f'+proj=utm +zone={zone_n} +{n_or_s}')
projection = crs.to_authority()[1]
...
outRasterSRS = osr.SpatialReference()
outRasterSRS.ImportFromEPSG(int(projection))
outRaster.SetProjection(outRasterSRS.ExportToWkt())
...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWF6MG736DCX2EESABDTT66NZANCNFSM4ZT26Z7A>
.
|
Hi Jianbao,
I believe that new mintpy version includes the fix that I provided. If you
grasp the newest version of mintpy. It will works with hyp3 and
download the ERA correctly.
Jiang
…On Sat, Jul 10, 2021 at 7:48 AM JianbaoSun ***@***.***> wrote:
Hi, Jiang,
I got the same ERA5 download issue when using HyP3 data. Mintpy works well
when GAM is not used for tropospheric corrections. Could you please let me
know what is your final solution? Thanks!
Sun
------------------------------
WARNING: downloading failed for 3 times, stop trying and continue.
------------------------------
update mode: ON
output file: inputs/ERA5.h5
1. output file either do NOT exist or is NOT newer than all GRIB files.
run or skip: run
open geometry file: geometryGeo.h5
reading incidenceAngle data from file: inputs/geometryGeo.h5 ...
reading height data from file: inputs/geometryGeo.h5 ...
------------------------------
create HDF5 file: inputs/ERA5.h5 with w mode
create dataset : date of |S1 in size of (0,) with compression = None
create dataset : timeseries of <class 'numpy.float32'> in size of (0, 953,
1202) with compression = None
close HDF5 file: inputs/ERA5.h5
------------------------------
calculating absolute delay for each date using PyAPS (Jolivet et al.,
2011; 2014) ...
number of grib files used: 0
Traceback (most recent call last):
File "/home/sun/MintPy_tools/MintPy/mintpy/tropo_pyaps3.py", line 821, in
main(sys.argv[1:])
File "/home/sun/MintPy_tools/MintPy/mintpy/tropo_pyaps3.py", line 793, in
main
calc_delay_timeseries(inps)
File "/home/sun/MintPy_tools/MintPy/mintpy/tropo_pyaps3.py", line 716, in
calc_delay_timeseries
prog_bar = ptime.progressBar(maxValue=num_date, print_msg=~inps.verbose)
File "/home/sun/MintPy_tools/MintPy/mintpy/objects/progress.py", line 53,
in *init*
self.reset()
File "/home/sun/MintPy_tools/MintPy/mintpy/objects/progress.py", line 58,
in reset
self.update_amount(0) # Build progress bar string
File "/home/sun/MintPy_tools/MintPy/mintpy/objects/progress.py", line 74,
in update_amount
percentDone = (diffFromMin / np.float(self.span)) * 100.0
ZeroDivisionError: float division by zero
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#540 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AO3CPWF6TR3IXSLH4QT3HLTTXBTUNANCNFSM4ZT26Z7A>
.
|
@yunjunz Regarding incidence angles, after more review, the
Elevation is already taken into account, but our We also notice that the impact of elevation on the angle map in your Galapagos example is significantly greater than when we see from HyP3 products over the same area. I'll post an example HyP3 product shortly. |
@asjohnston-asf the conversion could be applied in mintpy at stackDict.py#L619 since it already handles the radian to degree conversion. |
Hi ASF team (@cirrusasf, @talogan, @asjohnston-asf, et al.), it seems like the "Heading" attribute in the example dataset that @cirrusasf put here: Update: my miscalculation, the value is right and normal (with 360 in difference). Please ignore my above msg. |
FYI only, I created a sample data set for HyP3 using the same 98 S1 acquisitions (288 interferograms) as the Fernandina volcano data set for ARIA:
Not intended to be hosted there long-term, but I've made it available for now in case anyone wants to look at a HyP3 time series for a familiar data set. |
Close this issue as all the proposed changes are completed! Kudos to all! |
Happy to see this completed! |
🥳 @yunjunz do you have plans for a release soon since a lot of this (and a lot more!) is |
@jhkennedy yes, a release is planned for next month, for the exact reason you said. There are a few things on the to-do list before that: https://github.com/insarlab/MintPy/wiki/To-do-list#version-132 |
Hi guys, I am trying to play with Mintpy and Hyp3 on Google Colab these days. I am using "!smallbaselineApp.py mintpy/RidgecrestSenDT71.txt", but got stuck at the step of "modify network" (as shown in the end). Any hints on this issue? Appreciated! Best,
|
It looks like your |
@forrestfwilliams Appreciated you! After I rerun the code once again this morning, the bug was gone. I guess I passed a wrong path of input files, leading to no output file of ifgramStack. Then, I encountered the next issue (attached in the end). I will look into this later, which seems due to no output file of avgPhaseVelocity.h5. Have a nice day! Best,
|
Hi @forrestfwilliams and @yunjunz It seems this issue is still relevant since we can't create a subset using lat/lon in MintPy using HyP3 products. |
The ASF HyP3 on-demand InSAR service is great option for creating interferograms, but MintPy is not currently set up to ingest these products. I propose creating a prep_hyp3.py workflow that would allow for the ingestion of these products. I'm happy to perform this work, but could use guidance on preparing metadata.
The text was updated successfully, but these errors were encountered: