Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multitrack directory structure #1

Open
falkamelung opened this issue Aug 19, 2019 · 13 comments
Open

Multitrack directory structure #1

falkamelung opened this issue Aug 19, 2019 · 13 comments
Assignees
@ranneylxr
Labels
discussion

Comments

@falkamelung
Copy link
Member

Lets agree here about the multitrack directory structure.

Here is Yunjun's directory structure:

[yunjunz:~/insarlab/Kirishima] yunjunz% vi $TE/KirishimaAlosAT424F620_630.template 

yunjunz:~] yunjunz% cd $KIRISHIMA_DIR/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll
drwxr-xr-x@ 14 yunjunz  staff   476B May 13 23:35 ../
drwxr-xr-x   9 yunjunz  staff   306B Jul 18 21:49 Alos2DT23F2970/
drwxr-xr-x  18 yunjunz  staff   612B May 20 22:27 AlosAT424F620_630/
drwxr-xr-x   5 yunjunz  staff   170B Apr 15 18:09 AlosDT72F2980/
drwxr-xr-x  13 yunjunz  staff   442B Jun 18 16:23 AlosDT73F2970_2980/
drwxr-xr-x   6 yunjunz  staff   204B Jul  5 17:35 Asc_desc/
drwxr-xr-x  20 yunjunz  staff   680B Aug 14 17:41 DEM/
drwxr-xr-x   7 yunjunz  staff   238B Apr 14 19:46 Data/
drwxr-xr-x  35 yunjunz  staff   1.2K Jul  4 00:37 Doc/
drwxr-xr-x  17 yunjunz  staff   578B Apr 22 12:50 GPS/
drwxr-xr-x  10 yunjunz  staff   340B Jun 29 17:57 KirishimaAlos2DT23/
drwxr-xr-x  10 yunjunz  staff   340B Jul 20 20:29 Model/
drwxr-xr-x   9 yunjunz  staff   306B Aug 16 22:35 Seismic/
drwxr-xr-x   5 yunjunz  staff   170B Aug  4 18:24 ShinmoeAlosAT424/
drwxr-xr-x   5 yunjunz  staff   170B Aug  4 18:24 ShinmoeAlosDT73/
drwxr-xr-x  12 yunjunz  staff   408B Apr  9 14:50 Tephra/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll Model/
drwxr-xr-x   6 yunjunz  staff   204B Jul  1 21:42 GeodMod/
drwxr-xr-x  11 yunjunz  staff   374B Jul 20 02:22 Shinmoe_2008/
-rw-r--r--@  1 yunjunz  staff   8.5K Jul 20 22:35 Shinmoe_2008.inp
drwxr-xr-x   6 yunjunz  staff   204B Jun 30 20:47 Shinmoe_2016/
-rw-r--r--@  1 yunjunz  staff   6.3K Jul  1 00:53 Shinmoe_2016.inp
drwxrwxrwx  23 yunjunz  staff   782B Jul  9 14:47 dMODELS/
drwxr-xr-x  15 yunjunz  staff   510B Jul 21 18:54 data/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll Model/data/
-rw-r--r--   1 yunjunz  staff   830K Jun 17 13:12 Alos2DT23_20150223_20161031.cor
-rw-r--r--   1 yunjunz  staff   2.0K Jun 17 13:12 Alos2DT23_20150223_20161031.cor.rsc
-rw-r--r--   1 yunjunz  staff   5.2M Jun 30 20:21 Alos2DT23_20150223_20161031.mat
-rw-r--r--   1 yunjunz  staff   936K Jun 30 20:21 Alos2DT23_20150223_20161031.png
-rw-r--r--   1 yunjunz  staff   830K Jun 17 13:11 Alos2DT23_20150223_20161031_msk.unw
-rw-r--r--   1 yunjunz  staff   2.1K Jun 17 13:11 Alos2DT23_20150223_20161031_msk.unw.rsc
-rw-r--r--   1 yunjunz  staff   1.1M Jul 20 16:42 ShinmoeAlosAT424_20080929_20100520.mat
-rw-r--r--   1 yunjunz  staff   413K Jul 20 16:42 ShinmoeAlosAT424_20080929_20100520.png
-rw-r--r--   1 yunjunz  staff   1.0M Jul 20 16:42 ShinmoeAlosDT73_20081012_20100302.mat
-rw-r--r--   1 yunjunz  staff   433K Jul 20 16:42 ShinmoeAlosDT73_20081012_20100302.png
-rw-r--r--   1 yunjunz  staff   397K Jul 21 18:54 ShinmoeHz.h5
-rw-r--r--   1 yunjunz  staff   397K Jul 21 18:54 ShinmoeUp.h5

[yunjunz:~/insarlab/Kirishima] yunjunz% ll ShinmoeAlosAT424/mintpy/
[yunjunz:~/insarlab/Kirishima] yunjunz% ll AlosAT424F620_630/
drwxr-xr-x    8 yunjunz  staff   272B Jul  6 01:04 20101120_20110220/
drwxrws---   18 yunjunz  staff   612B Apr  1 16:59 DEM/
drwxrws---  227 yunjunz  staff   7.5K Apr 15 17:36 Igrams/
drwxrws---   30 yunjunz  staff   1.0K Apr 15 17:36 baselines/
drwxrws---  566 yunjunz  staff    19K May 14 12:36 configs/
drwxr-xr-x   92 yunjunz  staff   3.1K Apr  7 19:14 gacos/
drwxrws---   41 yunjunz  staff   1.4K Apr 15 22:28 geom_master/
-rw-r--r--@   1 yunjunz  staff   1.9K Apr  5 17:36 master.xml
drwxr-s---    6 yunjunz  staff   204B Apr 15 22:28 masterShelve/
drwxr-xr-x   10 yunjunz  staff   340B Aug 10 22:06 mintpy/
-rw-rw----    1 yunjunz  staff    82K Apr 14 23:30 pairs.pdf
drwxrws---   29 yunjunz  staff   986B Apr 15 17:36 run_files/
-rw-rw----    1 yunjunz  staff   5.5K Apr 14 23:27 run_unPackALOS

[yunjunz:~/insarlab/Kirishima] yunjunz% ll AlosAT424F620_630/
@falkamelung
Copy link
Member Author

Current helps:

//login3/nethome/famelung/test/development/rsmas_insar/sources/MimtPy[1013] save_geodmod.py --help
usage: save_geodmod.py [-h] [-t TEMPLATEFILE] [-ds [DATASET]] [-dt [DATATYPE]]
                       [-b S N W E] [-y LATSTEP] [-x LONSTEP] [-s [STARTDATE]]
                       [-e [ENDDATE]] [-outdir OUTDIR]
                       [file]

Prepare data for Geodmod software

positional arguments:
  file                  ascending and descending timeseries files

optional arguments:
  -h, --help            show this help message and exit
  -t TEMPLATEFILE, --template TEMPLATEFILE
                        Template file with geocoding options.
  -ds [DATASET], --dataset [DATASET]
                        name of dataset.Seperating by ','. 
  -dt [DATATYPE], --datatype [DATATYPE]
                        clarify the type of data.[velocity, ifgramStack, timeseries].Only used in template file
  -b S N W E, --bbox S N W E
                        Bounding box of area to be geocoded.
                        Include the uppler left corner of the first pixel    and the lower right corner of the last pixel
  -y LATSTEP, --latstep LATSTEP
                        output pixel size in degree in latitude.
  -x LONSTEP, --lonstep LONSTEP
                        output pixel size in degree in longitude.
  -s [STARTDATE], --startDate [STARTDATE]
                        date1 of timeseires to be converted.The default is the StartDate
  -e [ENDDATE], --endDate [ENDDATE]
                        date2 of timeseries to be converted.The default is the EndDate
  -outdir OUTDIR, --outdir OUTDIR
                        output directory

example:
  for singletrack:
  Note: startDate, endDate and outdir have default values, any of them can be not given:
    startDate default value = atr['START_DATE']
    endDate default value = atr['END_DATE']
    outdir default value = '$MODELDIR/project/Sen**/geodmod_startDate_endDate/'
  save_geodmod.py timeseries_ECMWF_demErr.h5 -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001  
  save_geodmod.py ifgramStack.h5  -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_geodmod.py velocity.h5  -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_geodmod.py S1_IW23_026_0108_0113_20171117_XXXXXXXX.he5 -s 20171128 -e 20181210 
  for multitrack:
  Note: startDate, endDate and DataSet can be not given in template:
  save_geodmod.py -t $MIMTFILES/Darbandikhan.txt

And for save_gbis.py:

./save_gbis.py --help
usage: save_gbis.py [-h] [-t TEMPLATEFILE] [--model-software MODEL_SOFTWARE]
                    [-ds [DATASET]] [-dt [DATATYPE]] [-b S N W E] [-y LATSTEP]
                    [-x LONSTEP] [-s [STARTDATE]] [-e [ENDDATE]]
                    [-m MASK_FILE] [--ref-lalo [REF_LALO]] [-outdir OUTDIR]
                    [file]

Prepare data for GBIS software

positional arguments:
  file                  ascending or descending files

optional arguments:
  -h, --help            show this help message and exit
  -t TEMPLATEFILE, --template TEMPLATEFILE
                        Template file with geocoding options.
  --model-software MODEL_SOFTWARE
                        Goephysical model's name.
  -ds [DATASET], --dataset [DATASET]
                        name of dataset.Seperating by ','. 
  -dt [DATATYPE], --datatype [DATATYPE]
                        clarify the type of data.[velocity, ifgramStack, timeseries, S1*.he5].Only used in template file
  -b S N W E, --bbox S N W E
                        Bounding box of area to be geocoded.
                        Include the uppler left corner of the first pixel    and the lower right corner of the last pixel
  -y LATSTEP, --latstep LATSTEP
                        output pixel size in degree in latitude.
  -x LONSTEP, --lonstep LONSTEP
                        output pixel size in degree in longitude.
  -s [STARTDATE], --startDate [STARTDATE]
                        date1 of timeseires to be converted.The default is the StartDate
  -e [ENDDATE], --endDate [ENDDATE]
                        date2 of timeseries to be converted.The default is the EndDate
  -m MASK_FILE, --mask MASK_FILE
                        mask file.
  --ref-lalo [REF_LALO]
                        custom reference pixel in lat/lon
  -outdir OUTDIR, --outdir OUTDIR
                        output directory

example:
  for singletrack:
  Note: startDate, endDate and outdir have default values, any of them can be not given:
    startDate default value = atr['START_DATE']
    endDate default value = atr['END_DATE']
    outdir default value = '$MODELDIR/project/Sen**/gbis_startDate_endDate/'
  save_gbis.py timeseries_ECMWF_demErr.h5 --model-software gbis -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001  
  save_gbis.py ifgramStack.h5 --model-software gbis -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_gbis.py velocity.h5 --model-software gbis -b 34.2 35.2 45.0 46.3 -y 0.001 -x 0.001 -s 20171117 -e 20171129 -outdir $MODELDIR/Darbandikhan/SenAT73/
  save_gbis.py S1_IW23_026_0108_0113_20171117_XXXXXXXX.he5 --model-software gbis -s 20171128 -e 20181210 
  for multitrack:
  Note: startDate, endDate and DataSet can be not given in template:
  save_gbis.py -t $MIMTFILES/Darbandikhan.txt

@falkamelung
Copy link
Member Author

falkamelung commented Aug 20, 2019 •
edited

The geo_* files that save_geodmod produces have a different posting then what we have in the geo directory. Shall we place them into a tmp_geo directory to avoid confusion?

@ranneylxr
Copy link
Contributor

ranneylxr commented Aug 21, 2019 •
edited

I design the structure of directory as following:
Firstly, the template files for mimtpy are stored in MIMTFILES; the generated results are stored in MODELDIR; Data that mimtpy used(generated by mintpy) are stored in SCRATCHDIR.

//centos7/data2/rsmasdata/lxr[1005] ll
drwxrwxr-x  2 lxr lxr       38 Aug 17 11:18 MIMTFILES
drwxrwsr-x  3 lxr famelung  34 Aug 21 09:17 MODELDIR
drwxrwxr-x 13 lxr lxr      334 Aug 20 15:28 SCRATCHDIR
drwxrwxr-x  4 lxr lxr       61 Jun 21 16:13 TESTDATA_ISCE

Secondly, I want to design the directory structure of MODELDIR like

$MODELDIR/project/SenAT(DT)***/gbis_startdate_enddate
$MODELDIR/project/SenAT(DT)***/geodmod_startdate_enddate
$MODELDIR/project/horzvert/startdate_enddate

Example:

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan[1009] ll
drwxrwsr-x 4 lxr famelung  68 Aug 20 20:46 horzvert
drwxrwsr-x 6 lxr famelung 149 Aug 19 20:42 SenAT72
drwxrwsr-x 9 lxr famelung 254 Aug 19 20:49 SenAT73
drwxrwsr-x 4 lxr famelung  81 Aug 19 17:14 SenDT79

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/SenAT72[1011]
gbis_20171117_20171129  
gbis_20171123_20171223
gbis_20171117_20171223 
geodmod_20171117_20171223

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/SenAT72/gbis_20171117_20171129/
DarbandikhanSenAT72_20171117_20171129.mat
geo_20171117_20171129.unw
geo_20171117_20171129.unw.rsc

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/SenAT72/geodmod_20171117_20171223/
geo_20171117_20171223.cor
geo_20171117_20171223.cor.rsc
geo_20171117_20171223.unw
geo_20171117_20171223.unw.rsc
srtm.dem
srtm.dem.jpeg
srtm.dem.jpeg.rsc
srtm.dem.rsc

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/horzvert/
20171117_20171205
0171117_20190423

//centos7/data2/rsmasdata/lxr/MODELDIR/Darbandikhan/horzvert/20171117_20190423/
geo_20171117_20190423.unw
geo_20171117_20190423.unw.rsc
geo_20171118_20190430.unw
geo_20171118_20190430.unw.rsc
hor.h5
hz.h5
up.h5
vel_horizontal.h5
vel_vertical.h5
ver.h5

@ranneylxr
Copy link
Contributor

For the geo_file in geod , gbis* and horzvert folders, do you mean we put the geo_file into geod/tmp_geo/ folder?

@falkamelung
Copy link
Member Author

No. We don’t need them, right? I mean /mintpy/tmp_geo

@ranneylxr
Copy link
Contributor

Yes, we don't need geo_***.h5 files. How about just delete them?

@falkamelung
Copy link
Member Author

falkamelung commented Aug 21, 2019 via email

@ranneylxr
Copy link
Contributor

OK! I will change the code

@yunjunz yunjunz mentioned this issue Aug 21, 2019
Closed
@falkamelung
Copy link
Member Author

@ranneylxr : If we follow Yunjun's suggestion and have save_gbis.py and save_geodmod.py only do very simple things. then we could have a save_for_modelling.py --modelSoftware gbis that takes care of the multitrack issues. All the functions such as track_date could be in mimtpy/utils/multitrack_utilities.py

By the way, some of the functions don't fave self-explanatory names. find_date --> find_start_end_date.py, track_date.py -->find_nearest_date.py

@falkamelung
Copy link
Member Author

@ranneylxr , I did some minor modifications as you have seen while I was debugging. Here a few more comments on the code:

  • if no --outdir is given it should write right into the directory where it is called (i.e. the mintpy dir
  • The temporarily geocoded data are written in the outdir, I believe. I would write them in the working directory (i.e. /mintpy/tmpgeo), as this is temporarily anyway.

@ranneylxr
Copy link
Contributor

OK. I will try to change them.

@ranneylxr
Copy link
Contributor

The new usage of save_geodmod.py and save_gbis_mimt.py is:

  1. save_for_modelling.py process multitrack information. It support geodmod software and gbis software now.
  2. save_geodmod.py and save_gbis both support four kind of data types(HDFEOS/ifgramStack/timeseries/velocity). Both of them can only be used for single track. And you have to call this command under $SCRATCHDIR/project/mintpy/ directory
  3. The default outdir is $SCRATCHDIR/project/mintpy/geodmod(gbis)_startDate_endDate/. And you can set your favorite outdir
  4. About generate_horzvert.py, I am thinking that it may be unnecessary to use save_for_modelling.py to process it. Because the input of this script must be two tracks. I will modify the environment setting in this script tomorrow.

@ranneylxr
Copy link
Contributor

The structure of directory in $MODELDIR (outdir you set) is changed:

  1. for geodmod and gbis, the output will be:
    DarbandikhanSenAT73/geodmod_startdate_enddate/
    DarbandikhanSenDT80/geodmod_startdate_enddate/
  2. for generate_horzvert, the output will be:
    DarbandikhanSen/horzvert/startdate_enddate/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ranneylxr ranneylxr

Labels
discussion
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@falkamelung @ranneylxr