UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format #384

taliboliver · 2020-06-22T23:16:45Z

Description of proposed changes

This PR adds UAVSAR support (#376). The detailed changes include:

add YYYYMMDDThhmm time string support to import to isce processed uavsar coregistered interferogram stack into mintpy.
add get_time_str_format() for time format identification in utils/ptime.py and objects/stack.py
adapt utils.readfile.py and ifgram_inversion.py to read the new time format

Reminders

Pass Codacy code review (green)
Pass Circle CI test (green)
Make sure that your code follows our style. Use the other functions/files as a basis.
If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
If adding new functionality, add a detailed description to the documentation and/or an example.

to import to mintpy isce processed uavsar coregistered stack data function get_time string format function in both ptime and stack readfile and ifgram_inversion were adapted to read the new time format

yunjunz · 2020-06-22T23:59:54Z

To-do list:

1. add the updated prep_isce.py with uavsar support (@taliboliver)
2. pass the tests for the uavsar stack and existing example datasets (@taliboliver and @yunjunz).
3. remove the duplicated objects.stack.get_time_str_format() and use ptime.utils.get_time_str_format() only. This will change the module dependency of stack.py and need to update the docs/api/module_hierarchy.md (@yunjunz).
4. ❎ update the time string format from YYYYMMDDTHHMMSS as suggested by @hfattahi (@yunjunz). Let's leave this to another PR because the current format works and should be saved for a simpler PR content.
5. replace all instances of dt(*time.strptime()[0:5]) to dt(*time.strptime()[0:6]) to include seconds. Find all instances via igrep ")\[0:5\]" (@yunjunz).

to be consistent with previous convention.

consider the hour/min/sec info while calculating the temporal baseline info in the following functions: + yyyymmdd2years() + date_list2tbase() + date_list2vector()

yunjunz · 2020-06-23T03:39:15Z

@taliboliver MintPy is currently using the time unit of years for the inversion steps, i.e. network inversion, dem error correction, and velocity estimation.

For UAVSAR, with a pair of images of 20 mins away, tbase = 10 / (365.25 * 24 * 60) = 1.9e-05, which is already very close to the resolution of np.float32 (1e-06). Since we are using float32 for all the least-squares estimation due to the performance (input data with float32 is 2x faster than the one with float64), this could lead to the precision leak.

>>> import numpy as np
>>> np.finfo(np.float32)
finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)

Thus, we should use the unit of days for all inversion steps. Storing the final result in the unit of years, such as the velocity file, is still fine. Let's check this carefully after the above to-do list is done.

Then we should remove those float64 design matrix in stack.py.

UPDATE: float64 is not needed because the current float32 is precise enough, as shown below.

mintpy/objects/stack.py

+ replace dt(*time.strptime) with dt.strptime() to be able to grab second info from input date string + remove unsed time module

adjust docs/api/module_hierarchy.md accordingly.

Update made to isce_utils.py to recognize isce uavsar "data" metafile format. Changes allow for correct data prep_isce.py use and load_data.py successfully.

from the initial prep_fringe.py PR.

changes made to date_str_format to read year without century and typo corrected in line 283.

…avsar

obj/sensor: + add uavsar to global varialbe SENSOR_NAMES + update standardSensorNames and get_unavco_mission_name() in alphebet order with UAV as the mission short name utils/ptime.get_date_str_format() + add both YYYYMMDDTHHMM and YYMMDDTHHMM

use chunk_size with variable number of pixel determined by the number of ifgrams for a fixed memory usage, more robust. less printout message for calc temp coh.

obj/stack: roll back the design matrix data type to float32 becuase the test result with both Sentinel-1 of 12 days re-visit time and UAVSAR of 10 mins re-visit time are precise enough with max residual less than 0.002 mm. simulation: + use lower case for all labels + move the old sim_variable_timeseries() to sim_variable_timeseries_v1() + add sim_variable_timeseries() that takes num_date so that it can generate the exact displacement list regardless of re-visit time. utils/network.select_pairs_sequential(): re-write to support any list of date, in str, datetime.datetime obj, etc. utils/ptime.get_date_str_format(): support YYYYMMDDTHHMMSS format

yunjunz · 2020-06-29T02:34:50Z

Here are the evaluation results of whether the precision of using the unit of years in float32 is enough for all three inversion problems (invert_network, correct_topography, velocity) in smallbaselineApp.py. All of them give the same level of estimation precision, thus, yes, they are precise enough. So we could keep using the previous form of design matrices.

1. invert_network

2. correct_topography

3. velocity

yunjunz · 2020-06-30T22:44:39Z

I got a dbm.gnu module not found error while reading the masterShelve/data file, processed using stripmapStack. @taliboliver could successfully run these lines. Any idea why would this happen? @hfattahi, @dbekaert and @piyushrpt.

I am using one conda environment with isce2 and mintpy installed. Talib is using brew installed isce2 and conda environment with mintpy installed.

import os
import isce
import isceobj
import shelve

fname = '/Users/yunjunz/Downloads/data'
print('file exists: {}'.format(os.path.isfile(fname)))
with shelve.open(fname, flag='r') as f:
    frame = f['frame']
print(frame.PRF, frame.radarWavelegth)

The full error message:

file exists: True
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-1-5139a7ff3399> in <module>
      5 
      6 fname = '/Users/yunjunz/Downloads/data'
----> 7 with shelve.open(fname, flag='r') as f:
      8     frame = f['frame']
      9 print(frame.PRF, frame.radarWavelegth)

~/tools/miniconda3/lib/python3.7/shelve.py in open(filename, flag, protocol, writeback)
    241     """
    242 
--> 243     return DbfilenameShelf(filename, flag, protocol, writeback)

~/tools/miniconda3/lib/python3.7/shelve.py in __init__(self, filename, flag, protocol, writeback)
    225     def __init__(self, filename, flag='c', protocol=None, writeback=False):
    226         import dbm
--> 227         Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
    228 
    229 

~/tools/miniconda3/lib/python3.7/dbm/__init__.py in open(file, flag, mode)
     89     elif result not in _modules:
     90         raise error[0]("db type is {0}, but the module is not "
---> 91                        "available".format(result))
     92     else:
     93         mod = _modules[result]

error: db type is dbm.gnu, but the module is not available

After modifying the dbm/gnu.py following the StackOverflow, the error message changed to:

file exists: True
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-5-a6c425a4aa35> in <module>
      6 fname = '/Users/yunjunz/Downloads/data'
      7 print('file exists: {}'.format(os.path.isfile(fname)))
----> 8 with shelve.open(fname, flag='r') as f:
      9     frame = f['frame']
     10 print(frame.PRF, frame.radarWavelegth)

~/tools/miniconda3/lib/python3.7/shelve.py in open(filename, flag, protocol, writeback)
    241     """
    242 
--> 243     return DbfilenameShelf(filename, flag, protocol, writeback)

~/tools/miniconda3/lib/python3.7/shelve.py in __init__(self, filename, flag, protocol, writeback)
    225     def __init__(self, filename, flag='c', protocol=None, writeback=False):
    226         import dbm
--> 227         Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
    228 
    229 

~/tools/miniconda3/lib/python3.7/dbm/__init__.py in open(file, flag, mode)
     92     else:
     93         mod = _modules[result]
---> 94     return mod.open(file, flag, mode)
     95 
     96 

error: [Errno 2] No such file or directory

obj/stack: roll back the design matrix data type to float32 becuase the test result with both Sentinel-1 of 12 days re-visit time and UAVSAR of 10 mins re-visit time are precise enough with max residual less than 0.002 mm. simulation: + use lower case for all labels + move the old sim_variable_timeseries() to sim_variable_timeseries_v1() + add sim_variable_timeseries() that takes num_date so that it can generate the exact displacement list regardless of re-visit time. utils/network.select_pairs_sequential(): re-write to support any list of date, in str, datetime.datetime obj, etc. utils/ptime.get_date_str_format(): support YYYYMMDDTHHMMSS format

…r/384

piyushrpt · 2020-07-01T03:52:48Z

I had some code to generate python3-gdbm bindings and had given it to @sssangha and @dbekaert at some point I think. You should be able to something similar to build gdbm if it is missing from your platform. Then you wont need to make that subtle change. I'm guess the error is because the database is not actually dbm.

piyushrpt · 2020-07-01T03:57:38Z

It was essentially wrapping a setup.py around this file https://github.com/python/cpython/blob/master/Modules/_gdbmmodule.c that included linking to libgdbm. You will get the lib when you install gdbm from conda.

yunjunz · 2020-07-01T05:22:37Z

BTW, for the masterShelve/data.dat from ALOS-2 stripmapStack products that I processed, the above shelve.open() went through successfully. Not sure if this is related with ContinuumIO/anaconda-issues#7356 (comment):

The lack of _gdbm means the standard library's shelve doesn't work between anaconda and non-anaconda Pythons.

piyushrpt · 2020-07-01T05:52:17Z

I dont think its an incompatibility issue. Just that the gdbm module is optional since it has dependency on libgdbm and wasn't part of the python build. If you build one yourself with that C file, it should work.

yunjunz · 2020-07-01T23:09:53Z

Is it possible to share this code of generating python3-gdbm bindings here? @piyushrpt @dbekaert @sssangha. Thank you in advance.

piyushrpt · 2020-07-02T00:10:01Z

I don't have access to my old email. Essentially, the setup.py did this

${CC} -I include_path -L lib_path _gdbmmodule.c -lgdbm -o gdbm.so

include_path to contain gdbm.h
lib_path to contain libgdbm

You can then drop the gdbm.so in any location you want.

yunjunz · 2020-07-02T05:38:36Z

This shelve issue passed through if using python environment via macports, after sudo port install py37-gdbm. In python environment via conda, it's still there.

yunjunz

Looks all good to me now!

The only issue left is the python-gdbm binding in the conda python env, which is not an issue due to this PR, thus, I think it's fine to merge it.

piyushrpt · 2020-07-02T05:46:27Z

You can use the macports recipe to build the _gdbm extension as well.
https://github.com/macports/macports-ports/blob/master/python/py-gdbm/files/setup-py3k.py

Replace "__PREFIX__" with str corresponding to $CONDA_PREFIX

yunjunz · 2020-07-02T06:15:19Z

@piyushrpt I git cloned cpython, which contains _gdbmmodule.c; prepared the setup.py from macports by replacing the prefix with the value of $CONDA_PREFIX as you suggested; run python3 setup.py build. Here is the error message. Any suggestions to proceed?

(base) yunjunz:~/tools/utils/cpython/Modules>$ python3 setup_gdbm.py build
running build
running build_ext
building '_gdbm' extension
creating build
creating build/temp.macosx-10.9-x86_64-3.7
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/yunjunz/tools/miniconda3/include -arch x86_64 -I/Users/yunjunz/tools/miniconda3/include -arch x86_64 -I/Users/yunjunz/tools/miniconda3/include -I/Users/yunjunz/tools/miniconda3/include/python3.7m -c _gdbmmodule.c -o build/temp.macosx-10.9-x86_64-3.7/_gdbmmodule.o
In file included from _gdbmmodule.c:56:
...
_gdbmmodule.c:543:5: error: use of undeclared identifier 'METH_METHOD'
    _GDBM_GDBM_KEYS_METHODDEF
    ^
./clinic/_gdbmmodule.c.h:99:59: note: expanded from macro '_GDBM_GDBM_KEYS_METHODDEF'
    {"keys", (PyCFunction)(void(*)(void))_gdbm_gdbm_keys, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, _gdbm_gdbm_keys__doc__},
                                                          ^
_gdbmmodule.c:544:5: error: use of undeclared identifier 'METH_METHOD'
    _GDBM_GDBM_FIRSTKEY_METHODDEF
    ^
./clinic/_gdbmmodule.c.h:132:67: note: expanded from macro '_GDBM_GDBM_FIRSTKEY_METHODDEF'
    {"firstkey", (PyCFunction)(void(*)(void))_gdbm_gdbm_firstkey, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, _gdbm_gdbm_firstkey__doc__},
                                                                  ^
_gdbmmodule.c:545:5: error: use of undeclared identifier 'METH_METHOD'
    _GDBM_GDBM_NEXTKEY_METHODDEF
    ^
./clinic/_gdbmmodule.c.h:169:65: note: expanded from macro '_GDBM_GDBM_NEXTKEY_METHODDEF'
    {"nextkey", (PyCFunction)(void(*)(void))_gdbm_gdbm_nextkey, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, _gdbm_gdbm_nextkey__doc__},
                                                                ^
_gdbmmodule.c:546:5: error: use of undeclared identifier 'METH_METHOD'
    _GDBM_GDBM_REORGANIZE_METHODDEF
    ^
./clinic/_gdbmmodule.c.h:207:71: note: expanded from macro '_GDBM_GDBM_REORGANIZE_METHODDEF'
    {"reorganize", (PyCFunction)(void(*)(void))_gdbm_gdbm_reorganize, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, _gdbm_gdbm_reorga...
                                                                      ^
_gdbmmodule.c:547:5: error: use of undeclared identifier 'METH_METHOD'
    _GDBM_GDBM_SYNC_METHODDEF
    ^
./clinic/_gdbmmodule.c.h:239:59: note: expanded from macro '_GDBM_GDBM_SYNC_METHODDEF'
    {"sync", (PyCFunction)(void(*)(void))_gdbm_gdbm_sync, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, _gdbm_gdbm_sync__doc__},
                                                          ^
...
36 warnings and 5 errors generated.
error: command 'gcc' failed with exit status 1

piyushrpt · 2020-07-02T06:35:13Z

Looks like it needs an additional .h file as well now - https://github.com/macports/macports-ports/blob/e8949c965569e476add2628228231d96fc8c634b/python/py-gdbm/Portfile#L122

since we will use isce for it.

welcome · 2020-07-02T18:02:11Z

🎉 🎉 🎉 Congrats on merging your first pull request! We here at behaviorbot are proud of you! 🎉 🎉 🎉

yunjunz · 2020-07-02T18:03:05Z

Here is an overview of what got changed by this pull request:

Clones added
============
- mintpy/simulation/simulation.py  2
         

Clones removed
==============
+ mintpy/utils/network.py  -2

See the complete overview on Codacy

zhuhaxixiong · 2021-09-22T16:58:28Z

my mintpy was in base environment ,i must active conda in shell before use mintpy smallbaselineApp.py ,after i exit the base i can use prep_isce.py to generate the dara.rsc file

yunjunz self-requested a review June 22, 2020 23:30

yunjunz marked this pull request as draft June 22, 2020 23:32

yunjunz mentioned this pull request Jun 22, 2020

unavsar support in obj/stack.py #383

Closed

UAVSAR support via YYYYMMDDThhmm time format

9823cd0

to import to mintpy isce processed uavsar coregistered stack data function get_time string format function in both ptime and stack readfile and ifgram_inversion were adapted to read the new time format

yunjunz force-pushed the uavsar branch from 462ff41 to 9823cd0 Compare June 22, 2020 23:44

yunjunz added 7 commits June 22, 2020 17:29

ifg_inv: revert tbase to the unit of year

0f9ef45

to be consistent with previous convention.

Update load_data.py

1f5195d

Update load_data.py

ab6631f

potential bug fix in ptime.get_date_str_format()

835756c

ptime: add seconds info for all time diff calc

571e65c

consider the hour/min/sec info while calculating the temporal baseline info in the following functions: + yyyymmdd2years() + date_list2tbase() + date_list2vector()

Update readfile.py

762d913

bugs fix

79ac97f

Update stack.py

0085c7b

yunjunz reviewed Jun 23, 2020

View reviewed changes

mintpy/objects/stack.py Outdated Show resolved Hide resolved

yunjunz reviewed Jun 23, 2020

View reviewed changes

mintpy/objects/stack.py Outdated Show resolved Hide resolved

yunjunz added 2 commits June 22, 2020 21:06

replace dt(*time.strptime) with dt.strptime()

99f3911

+ replace dt(*time.strptime) with dt.strptime() to be able to grab second info from input date string + remove unsed time module

stack: use ptime.get_date_str_format()

5d4d3ff

adjust docs/api/module_hierarchy.md accordingly.

yunjunz changed the title ~~UAVSAR support via YYYYMMDDThhmm time format~~ UAVSAR support Jun 23, 2020

taliboliver and others added 10 commits June 23, 2020 08:16

Update isce_utils.py

998c63f

Update made to isce_utils.py to recognize isce uavsar "data" metafile format. Changes allow for correct data prep_isce.py use and load_data.py successfully.

prep_isce: remove unnecessary code for date12

cc9b228

from the initial prep_fringe.py PR.

Update ptime.py

8887af1

changes made to date_str_format to read year without century and typo corrected in line 283.

Merge branch 'uavsar' of https://github.com/taliboliver/MintPy into u…

20e845e

…avsar

sensor: add uavsar

a540c84

obj/sensor: + add uavsar to global varialbe SENSOR_NAMES + update standardSensorNames and get_unavco_mission_name() in alphebet order with UAV as the mission short name utils/ptime.get_date_str_format() + add both YYYYMMDDTHHMM and YYMMDDTHHMM

update docs/hdfeos5.md for UAV

7b423b0

sensor: add PLATFORM name uav

7ce028e

ifg_inv: adaptive printout msg

11b3b57

use chunk_size with variable number of pixel determined by the number of ifgrams for a fixed memory usage, more robust. less printout message for calc temp coh.

Update stack.py

fb5db4d

yunjunz marked this pull request as ready for review June 30, 2020 22:42

yunjunz requested review from dbekaert and hfattahi June 30, 2020 22:42

yunjunz added 2 commits June 30, 2020 19:49

Merge branch 'uavsar' of https://github.com/taliboliver/MintPy into p…

a084b17

…r/384

ports.txt: add py37-gdbm

142d1b9

yunjunz approved these changes Jul 2, 2020

View reviewed changes

yunjunz changed the title ~~UAVSAR support~~ UAVSAR support processed with stripmapStack via YYMMDDTHHMMSS time format Jul 2, 2020

yunjunz changed the title ~~UAVSAR support processed with stripmapStack via YYMMDDTHHMMSS time format~~ UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format Jul 2, 2020

remove uavsar processor in load_data.py

a058cc6

since we will use isce for it.

yunjunz merged commit b4e00aa into insarlab:master Jul 2, 2020

yunjunz mentioned this pull request Nov 6, 2022

ISCE2 StripmapStack load_data looking for ionUnwFile #891

Closed

yunjunz mentioned this pull request Aug 31, 2023

Add function to create CF-compliant arrays/metadata for HDF5 stacks #1073

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format #384

UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format #384

taliboliver commented Jun 22, 2020 •

edited by yunjunz

Loading

yunjunz commented Jun 22, 2020 •

edited

Loading

yunjunz commented Jun 23, 2020 •

edited

Loading

yunjunz commented Jun 29, 2020

yunjunz commented Jun 30, 2020 •

edited

Loading

piyushrpt commented Jul 1, 2020

piyushrpt commented Jul 1, 2020

yunjunz commented Jul 1, 2020 •

edited

Loading

piyushrpt commented Jul 1, 2020

yunjunz commented Jul 1, 2020 •

edited

Loading

piyushrpt commented Jul 2, 2020

yunjunz commented Jul 2, 2020

yunjunz left a comment

piyushrpt commented Jul 2, 2020 •

edited

Loading

yunjunz commented Jul 2, 2020 •

edited

Loading

piyushrpt commented Jul 2, 2020

welcome bot commented Jul 2, 2020

yunjunz commented Jul 2, 2020

zhuhaxixiong commented Sep 22, 2021

UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format #384

UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format #384

Conversation

taliboliver commented Jun 22, 2020 • edited by yunjunz Loading

yunjunz commented Jun 22, 2020 • edited Loading

yunjunz commented Jun 23, 2020 • edited Loading

yunjunz commented Jun 29, 2020

1. invert_network

2. correct_topography

3. velocity

yunjunz commented Jun 30, 2020 • edited Loading

piyushrpt commented Jul 1, 2020

piyushrpt commented Jul 1, 2020

yunjunz commented Jul 1, 2020 • edited Loading

piyushrpt commented Jul 1, 2020

yunjunz commented Jul 1, 2020 • edited Loading

piyushrpt commented Jul 2, 2020

yunjunz commented Jul 2, 2020

yunjunz left a comment

Choose a reason for hiding this comment

piyushrpt commented Jul 2, 2020 • edited Loading

yunjunz commented Jul 2, 2020 • edited Loading

piyushrpt commented Jul 2, 2020

welcome bot commented Jul 2, 2020

yunjunz commented Jul 2, 2020

zhuhaxixiong commented Sep 22, 2021

taliboliver commented Jun 22, 2020 •

edited by yunjunz

Loading

yunjunz commented Jun 22, 2020 •

edited

Loading

yunjunz commented Jun 23, 2020 •

edited

Loading

yunjunz commented Jun 30, 2020 •

edited

Loading

yunjunz commented Jul 1, 2020 •

edited

Loading

yunjunz commented Jul 1, 2020 •

edited

Loading

piyushrpt commented Jul 2, 2020 •

edited

Loading

yunjunz commented Jul 2, 2020 •

edited

Loading