How to run the script with new datas from other sites

[duyhoangatk]

Hi Prof Yumorishita,
Sorry to ask you again. I'm practicing analyzing and have few questions about it.

1. Is there anyway to add other datas to the result folder without running script again from beginning because some places don't have the data's time that I want (Ex: I want to know the displacement of a place from 2020/01/01 until 2024/05/01 but the latest data is only until 2023/06/03, so I'm thinking that I will add the datas from 2023/07/01 from other sites to folder and run again) ? If there is, can you share which site can I download to match the data downloaded from COMET ?
2. Can you share me the site where explains the meaning of every legend in the result graph (such as 'vel(1)', 'vel(2)', 'mask',... until 'hgt') ?

Maybe these are very basic questions but I'm still new so again sorry to ask.
Waiting for your reply. Thank you very much.

[yumorishita]

1. I don't know any other sites. I generally recommend creating all the unwrapped data yourself if COMET data is not enough.
2. Please see Table 1 in the LiCSBAS paper.
   vel(2): A problem in LiCSBAS_cum2vel.py #117 (comment)
   Others: https://github.com/yumorishita/LiCSBAS/wiki/file_structure
   

[duyhoangatk]

Hi Prof Yumorishita,
Thank you for your kind response. Now I'm having this issue below when running batch script that after finishing download datas, the process is not running anymore just stop at downloading files although I have been waiting for hours but still nothing happen (this error may occur depending on places)

How can I fix this problem.
Waiting for your reply. Thank you very much.

[yumorishita]

Please retry.

[duyhoangatk]

Hi Prof Yumorishita,
I have retried to run batch script again, and it happens new error as following:

How can I fix this ?
Waiting for your reply. Thank you very much.

[yumorishita]

The bug was fixed in LiCSBAS2 (yumorishita/LiCSBAS2#55). Please use LiCSBAS2 or apply the fix in your local LiCSBAS script.

[duyhoangatk]

Hi Prof Yumorishita,
Thank you for your reply, the batch script is now working back normally.
And I continue to learn this software, so I have few more basic questions related to the results:

1. According to the result graph of 「Velocity / Cummulative Displacement」, I think that the areas with blue color are going downward (沈下) due to minus number, areas with red are going upward（隆起）due to plus number. Am I correct ?
2. I found that there are some points that have the cummulative displacement through time different from the time-series graph (check photo as an example). Can you kindly explain why ?
   
3. When I try to export the time series of the displacement at a specified point text file, I face the 「out of range」error, how can I fix this ?
   
4. When display the GeoTIFF file that created by LiCSBAS_flt2geotiff.py, is the value of created parameter same as the result graph ? (because for example, I run batch script for a place which has the following range of masked velocity, but when I convert the masked velocity to GeoTIFF and display on QGIS, the value has the range 4.57 ~ -37.207 )
   

There are many many new knowledge and terminologies that I still don't know yet, so sorry if I asked these questions.
Waiting for your reply. Thank you

[yumorishita]

1. According to the result graph of 「Velocity / Cummulative Displacement」, I think that the areas with blue color are going downward (沈下) due to minus number, areas with red are going upward（隆起）due to plus number. Am I correct ?

Almost yes, but you should note that LOS displacement includes the EW component as well.

2. I found that there are some points that have the cummulative displacement through time different from the time-series graph (check photo as an example). Can you kindly explain why ?

Do you mean that some of the displacement values exceed the max/min color range? It is because the color range is automatically determined by 99% of the data value to ignore outliers. Please try --auto_crange 100 option.

3. When I try to export the time series of the displacement at a specified point text file, I face the 「out of range」error, how can I fix this ?

It is because the location of the point you specified (107/83) is outside of the data (i.e., exceeds the size of the data). Please select the correct point location inside of the data.

4. When display the GeoTIFF file that created by LiCSBAS_flt2geotiff.py, is the value of created parameter same as the result graph ? (because for example, I run batch script for a place which has the following range of masked velocity, but when I convert the masked velocity to GeoTIFF and display on QGIS, the value has the range 4.57 ~ -37.207 )

It should be the same. Please check the answer 2 above.

[duyhoangatk]

Hi Prof Yumorishita,

1. 「Almost yes, but you should note that LOS displacement includes the EW component as well.」
   → Thank you for your information. Actually there is an analysis called 2.5 dimension horizontal analysis (2.5次元水平解析）that can produce EW （東西方向 or 水平方向）displacement and up-down（鉛直方向） displacement results serepately, and I'm doing my job using it.
2. 「Do you mean that some of the displacement values exceed the max/min color range? It is because the color range is automatically determined by 99% of the data value to ignore outliers. Please try --auto_crange 100 option.」
   → Sorry but can you show me how and where to use the --auto_crange 100 option (in Ubuntu LST or in graph directly ? Sorry I'm completely blank)
3. 「It is because the location of the point you specified (107/83) is outside of the data (i.e., exceeds the size of the data). Please select the correct point location inside of the data.」
   → My mistake here. I just copied the code from your sample pdf file but forgot to change the point coordinate. Thank you for your reply.
4. 「It should be the same. Please check the answer 2 above.」
   → I will try again when Prof can show me how to use the option. But for more details, I wil show the photos.
   
5. One more thing, for example, I converted the masked vel(1) to GeoTIFF, so the GeoTIFF will be converted in which time ? And, the max-min values are completely different from the color range values, is it the same as the problem 2 ?
   

Waiting for your reply. Thank you

[yumorishita]

--auto_crange is the one of the options of LiCSBAS_plot_ts.py and LiCSBAS_disp_img.py.

One more thing, for example, I converted the masked vel(1) to GeoTIFF, so the GeoTIFF will be converted in which time ?

What command and arguments did you use to create the GeoTIFF?

[duyhoangatk]

Hi Prof Yumorishita,
1.「--auto_crange is the one of the options of LiCSBAS_plot_ts.py and LiCSBAS_disp_img.py.」
→ For example, usually I display the results by using 「LiCSBAS_plot_ts.py -i TS_GEOCml1clip/cum_filt.h5 &」code, so if want to use options auto_crange as you recommended, will it be added like: 「LiCSBAS_plot_ts.py TS_GEOCml1clip/cum_filt.h5 & --auto_crange」? (Sorry if you think I'm asking silly questions)
2. 「What command and arguments did you use to create the GeoTIFF?」
→ Simply copy the code 「LiCSBAS_flt2geotiff.py -i TS_GEOCml1clip/results/vel.mskd -p TS_GEOCml1clip/info/EQA.dem_par」from your sample pdf file to create GeoTIFF (this one is to convert masked velocity graph to GeoTIFF, I think), but still I don't know which time the result is showing...

Waiting for your reply. Thank you

[yumorishita]

1. Try LiCSBAS_plot_ts.py TS_GEOCml1clip/cum_filt.h5 --auto_crange 100 &.
2. Then it is the unfiltered velocity (vel(2)) and have no particular date. vel(1) corresponds to vel.filt.mskd.

[duyhoangatk]

Hi Prof Yumorishita,

1. 「Try LiCSBAS_plot_ts.py TS_GEOCml1clip/cum_filt.h5 --auto_crange 100 &.」
   → Thank you for your instruction. I will try
2. 「Then it is the unfiltered velocity (vel(2)) and have no particular date. vel(1) corresponds to vel.filt.mskd.」
   → Oh, I didn't notice it. I will try again with vel(1). And if the above code have no particular date, how can I convert it in a particular date ? (I think maybe add the date into the code ? )
3. I have another problem when running batch file, which is Index error:
   
   How can I fix this ?

Waiting for your reply. Thank you very much

[yumorishita]

how can I convert it in a particular date ?

First you should create a float file for a particular date from cum_filt.h5 using LiCSBAS_cum2flt.py.

3. I have another problem when running batch file, which is Index error:

I could not reproduce the error using the latest version. Please see #244 (comment).

[duyhoangatk]

Hi Prof Yumorishita,

1. 「First you should create a float file for a particular date from cum_filt.h5 using LiCSBAS_cum2flt.py.」
   → Thank you very much. I will try
2. Actually I'm using the LiCSBAS2. The batch is now running normally again, maybe the error was temporarily occured.

Thank you for your reply. I will contact again if there is any problem.

[duyhoangatk]

Hi Prof Yumorishita,
I have few questions:

1. About the option --auto_crange 100, the result (take vel(1) as example) with and without the option are different when displaying, but the GeoTIFF's value is taken from the one with the option. So which one is the correct result because this is making me confused. (Please check attached photo)
   
2. Just confirmation but, are vel(1) and vel(2) not in any specific time but shown in a whole analysis time right ? (For example I analyzed a place from 2020/01/01 until today, does that mean vel(1) is the velocity of whole time ?)
3. Is there anyway to export csv file of vel(1) in entire area (not at specific point) ?
4. Can you kindly explain what is different between filtered velocity vel(1) and unfiltered velocity vel(2) ? (because I see in the result graph that both have nearly same values)

Waiting for your reply. Thank you very much

[yumorishita]

1. It is just a visualization (color range) difference and both are identical.
2. Yes.
3. You can export a text file using gdal2xyz from GeoTIFF.
4. A problem in LiCSBAS_cum2vel.py #117 (comment)

[duyhoangatk]

Hi Prof Yumorishita,
3. 「You can export a text file using gdal2xyz from GeoTIFF.」
→ Thank you for your infomation, but I don't see any .py file of gdal2xyz in 'bin' folder. Can you please show me how to do it ?
4. I have seen the comment, but what I want to know is the difference between filtered vel(1) and unfiltered velocity vel(2) (yet I still don't know the difference between filtered and unfiltered) and how they reflect to the result ? (for example, filterd time series vel(1) has better noise handling compared to unfiltered vel(2), which reduce noises in result or something like that. Sorry if my question is not clearly pointed)

Waiting for your reply. Thank you very much

[yumorishita]

gdal2xyz is not a LiCSBAS command but a gdal command: https://gdal.org/programs/gdal2xyz.html

Please read the LiCSBAS paper for the filtering: https://www.mdpi.com/2072-4292/12/3/424#sec2dot4dot6-remotesensing-12-00424

[duyhoangatk]

Hi Prof Yumorishita,
Thank you for your reply. I am now able to use gdal2xyz.
In the sample paper, there is a way to convert the time serires displacement at a specific point ino .txt file. So, is there anyway to convert all points at once because I want to see all but it really takes time to convert every single point for a large area.
Waiting for your reply. Thank you very much

[yumorishita]

You can use gdal2xyz for all epochs after LiCSBAS_cum2flt.py and LiCSBAS_flt2geotiff.py to get all the points in text.

[duyhoangatk]

Hi Prof Yumorishita,
I have tried and successfully exported, but it is only in one specific date not whole in time period. Is there anyway to convert in all period time ? (For example I have a result of cummulative displacement of a place from 2015 to 2020, but in according to your solution it can only export all point at only 1 specific date, such as 2019/01/01 not all dates within period)
And, about the 「IndexError: list index out of range」, even though I'm using LiCSBAS2 but sometimes this bug still occur (Yesterday was not but today has occured). Can you please check it ?
Waiting for your reply. Thank you

[yumorishita]

Just iterate exporting by for-loop.
Please attach the full log for the error.

[duyhoangatk]

Hi Prof Yumorishita,
1.「Just iterate exporting by for-loop.」
→ I think you are meaning to add a for-loop in LiCSBAS_cum2flt.py code, I have tried and be able to export all cummulative displacement datas of dates within period but it export single data for single date not single data for all dates. Is there anyway to export only 1 data but for all dates ?
2. 「Please attach the full log for the error.」
→ The server is now under maintenance so I can't reproduce it, maybe it is for fixing the mentioned error. If there is error again, I will show the full log.

Waiting for your reply. Thank you very much

[yumorishita]

1. I cannot understand what data format you want to export.

[duyhoangatk]

HI Prof Yumorishita,
Sorry if my question is difficult to understand. Let me summarize:
I want to export csv file of cummulative displacement of all points in a specific place, in a period of time (ex: 2019~2022). But right now in your code, there is only able to export at one specified coordinate (using LiCSBAS_cum2tstxt.py), not all.
→ which means basically I have to export each point's cummulative displacement at once, which will take times if I do with large areas.
And then Prof recommend that do the for-loop. I tried to modify the 「LiCSBAS_cum2flt.py」 code and now I can produce all the cum files of cummulative displacement of all points at once, but they are seperated files, not compacted in 1 single cum file.
→ which means, I still only can convert 1 cum file into csv at once, which will take much times if there are hundreds of cum files.
(Basically, I convert a cummulative displacement cum file into csv file following your instruction: LiCSBAS_cum2flt.py → LiCSBAS_flt2geotiff.py → gdal2xyz)

So, what I want to ask is: Is there any method to produce all points' cummulative displacement (from LiCSBAS_cum2flt.py step) compacted in only 1 single cum file, which is convenient to convert into csv file ?
Or is there any method to convert cum files → csv file immediately without converting into GeoTIFF file ?
I hope you can understand with this explanation.

And yet, I encoutered the ListIndex error again even though I'm using LiCSBAS2 (and I already replace the newest LiCSBAS01_get_geotiff.py file). Please check

Waiting for your reply. Thank you very much.

[yumorishita]

I would iterate LiCSBAS_cum2flt.py → LiCSBAS_flt2geotiff.py → gdal2xyz processing for all dates (by for-loop) and combine the output csv into one. It should not take so long time compared to the LiCSBAS processing time even if the number of epochs is more than a hundred.

Regarding the error, please attach the full log.

[duyhoangatk]

Hi Prof Yumorishita,

1. 「I would iterate LiCSBAS_cum2flt.py → LiCSBAS_flt2geotiff.py → gdal2xyz processing for all dates (by for-loop) and combine the output csv into one. It should not take so long time compared to the LiCSBAS processing time even if the number of epochs is more than a hundred.」
   → Thank you for your reply. I will try
2. 「Regarding the error, please attach the full log.」→ You mean this:
   Start step: 01
   End step: 16
   Log file: log/202406260823batch_LiCSBAS_01_16.log

LiCSBAS01_get_geotiff.py ver1.6.4 20230215 Y. Morishita
LiCSBAS01_get_geotiff.py -s 20190101

Frame ID is 055A_06858_131510

055A_06858_131510.geo.E.tif already exist. Skip download.
055A_06858_131510.geo.N.tif already exist. Skip download.
055A_06858_131510.geo.U.tif already exist. Skip download.
055A_06858_131510.geo.hgt.tif already exist. Skip download.

Download baselines
baselines, 6.54KB, 1s, 6.54KB/s
Download network.png
network.png, 206.41KB, 2s, 103.21KB/s
Download metadata.txt
metadata.txt, 249.0B, 1s, 249.0B/s

Searching latest epoch for mli...
120/129
No mli available on https://gws-access.jasmin.ac.uk/public/nceo_geohazards/LiCSAR_products/55/055A_06858_131510/epochs/

Download geotiff of unw and cc
Traceback (most recent call last):
File "/home/atk/LiCSBAS2/bin/LiCSBAS01_get_geotiff.py", line 500, in
sys.exit(main())
^^^^^^
File "/home/atk/LiCSBAS2/bin/LiCSBAS01_get_geotiff.py", line 340, in main
print('{} IFGs available from {} to {}'.format(n_ifg, imdates[0], imdates[-1]), flush=True)
~~~~~~~^^^
IndexError: list index out of range

I think this issue should have been fixed before. But right now even I updated your fixed 「LiCSBAS01_get_geotiff.py」, the error is still occured for some places, not all. Please check.
Waiting for your reply. Thank you very much.

[yumorishita]

You are using the old version: LiCSBAS01_get_geotiff.py ver1.6.4 20230215 Y. Morishita
The latest version is v1.6.5.
https://github.com/yumorishita/LiCSBAS2/blob/f1e2592dfb37cfde243f644fcfe554ab4b67bba7/bin/LiCSBAS01_get_geotiff.py#L99

[duyhoangatk]

Hi Prof Yumorishita,
Thank you for your reply.
I have tried to run again with newest version, but the error still occured. Maybe it occured only in the place I'm trying to. It's in Ha Long, Vietnam with following image: (https://maps.app.goo.gl/rZEB1NXQUvU1xHjW8)

I tried with other places and the code run very smoothly. Maybe there is something wrong with this place only ?
Please check. Waiting for your reply. Thank you very much

[yumorishita]

The error should not depend on the AOI. Please attached the full log again.

[duyhoangatk]

Hi Prof Yumorishita,
I have tried again, and the error log is the same as above comment:

(licsbas) atk@atkttwc0725:/mnt/c/licsbas/Halong/055A_06858_131510$ ./batch_LiCSBAS.sh

Start step: 01
End step: 16
Log file: log/202407020918batch_LiCSBAS_01_16.log

LiCSBAS01_get_geotiff.py ver1.6.5 20240607 Y. Morishita
LiCSBAS01_get_geotiff.py -s 20210101

Frame ID is 055A_06858_131510

Download 055A_06858_131510.geo.E.tif
055A_06858_131510.geo.E.tif, 13.09MB, 16s, 837.87KB/s
Download 055A_06858_131510.geo.N.tif
055A_06858_131510.geo.N.tif, 13.31MB, 14s, 973.87KB/s
Download 055A_06858_131510.geo.U.tif
055A_06858_131510.geo.U.tif, 13.09MB, 10s, 1.31MB/s
Download 055A_06858_131510.geo.hgt.tif
055A_06858_131510.geo.hgt.tif, 13.09MB, 26s, 515.61KB/s

Download baselines
baselines, 6.54KB, 1s, 6.54KB/s
Download network.png
network.png, 206.41KB, 2s, 103.21KB/s
Download metadata.txt
metadata.txt, 249.0B, 1s, 249.0B/s

Searching latest epoch for mli...
60/ 67
No mli available on https://gws-access.jasmin.ac.uk/public/nceo_geohazards/LiCSAR_products/55/055A_06858_131510/epochs/

Download geotiff of unw and cc
Traceback (most recent call last):
File "/home/atk/LiCSBAS2/bin/LiCSBAS01_get_geotiff.py", line 502, in
sys.exit(main())
^^^^^^
File "/home/atk/LiCSBAS2/bin/LiCSBAS01_get_geotiff.py", line 342, in main
print('{} IFGs available from {} to {}'.format(n_ifg, imdates[0], imdates[-1]), flush=True)
~~~~~~~^^^
IndexError: list index out of range

I used the frame ID: 055A_06858_131510, do05op_clip="y", p05_clip_range_geo="107.03/107.05/20.96/20.98"
Date is not very inportant.
Am I doing something wrong with these parameters ?

Waiting for your reply. Thank you very much.

[yumorishita]

The error is because there are no ifgs after 20210101.
https://gws-access.jasmin.ac.uk/public/nceo_geohazards/LiCSAR_products/55/055A_06858_131510/interferograms/

You need to change the start date.

[duyhoangatk]

Hi Prof Yumorishita,
Thank you for your reply. I will change the date and try again

[duyhoangatk]

Hi Prof. Yumorishita,
Is the COMET site now in maintenance ? I tried to run the batch script but then failed at the 1st data downloading step from ysterday.
Sorry for asking question not related to the LiCSBAS but still hope for your reply. Thank you

[yumorishita]

I think so.
https://x.com/COMET_database/status/1826570211601838537