Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clip error #25

Closed
atangmao opened this issue Jul 1, 2020 · 16 comments
Closed

Clip error #25

atangmao opened this issue Jul 1, 2020 · 16 comments

Comments

@atangmao
Copy link

atangmao commented Jul 1, 2020

Hi,
great work, but I meet a problem when I trying to apply on area 005D_04801_131313, in the clip step it says:
Traceback (most recent call last):
File "/home/weiyi/Desktop/LiCSBAS/bin/LiCSBAS05op_clip_unw.py", line 308, in
sys.exit(main())
File "/home/weiyi/Desktop/LiCSBAS/bin/LiCSBAS05op_clip_unw.py", line 235, in main
hgt = io_lib.read_img(os.path.join(out_dir, 'hgt'), length_c, width_c)
File "/home/weiyi/Desktop/LiCSBAS/LiCSBAS_lib/LiCSBAS_io_lib.py", line 146, in read_img
data = np.fromfile(file, dtype=dtype).reshape((length, width))
ValueError: cannot reshape array of size 9844835 into shape (952,1052)
What kind of input clip input data should I input?
I used "72.25/73.30/41.15/42.10" here.

Best reagrads.
@atangmao
Copy link
Author

atangmao commented Jul 1, 2020

And also, if I don't use the clip step, it will end with this step:
Process 1345/2690th line (1/2th patch)...
Reading 66 ifg's unw data...
1327736/4188330 points removed due to not enough ifg data...

Identifing gaps and counting n_gap...
Counting n_ifg_noloop...

Small Baseline inversion by LS...

Solving 1341331/2860594th points with full unw at a time...
Next, solve 1519263 points including nan point-by-point...
Running 1519000/1519263th point...
Elapsed time for 1th patch: 00h 06m 54s

Process 2690/2690th line (2/2th patch)...
Reading 66 ifg's unw data...
1633923/4188330 points removed due to not enough ifg data...

Identifing gaps and counting n_gap...
Counting n_ifg_noloop...
(base) weiyi@weiyi-virtual-machine:~/Desktop/LiCSBAS/114A_05790_131313$

@yumorishita
Copy link
Owner

Can you upload full log messages for the first issue?

The second issue might be caused by memory over. This step could use larger amount of memory than expected. Please retry the step 1-3 with, e.g., p14_mem_size="2000" in batch_LiCSBAS.sh.

@yumorishita
Copy link
Owner

Have you solved this issue? @tanmeimei

@atangmao
Copy link
Author

atangmao commented Jul 8, 2020

I haven't solved this issue and here is the log file
202007081414batch_LiCSBAS_01_16.log

@yumorishita
Copy link
Owner

I investigated the files and realised that the file size of [ENU].tif and hgt.tif is inconsistent with the unw.tif. This problem is in the LiCSAR processing and the files need to be recreated. I have asked the person in charge. I hope this problem will be fixed soon.

@espiritocz
Copy link

hi tanmeimei,

thank you for reporting this issue.
there was an error in the frame files. we are working on fixing it and should generate actually much more interferograms soon!
hope it is ok if you can wait for the data a week or two (please let us know if this is urgent)

@atangmao
Copy link
Author

Thanks, this is not very urgent.

@yumorishita
Copy link
Owner

It took a long time but now the data in 005D_04801_131313 are available. Please try again.

@fengnian-chang
Copy link

I meet the same problem in area 063D_04886_161920. An error occurred (cannot reshape array of size 2337741 into shape***) in step05 or step13. It seems that the file size of [ENU].tif and hgt.tif is inconsistent with the unw.tif.

@espiritocz
Copy link

espiritocz commented Jul 25, 2023 via email

@yumorishita yumorishita mentioned this issue Jul 25, 2023
Closed
@fengnian-chang
Copy link

thank you for reporting this issue with the LiCSAR data, frame 063D_04886_161920. I see there were 14 ifg pairs of the frame with this issue. We fixed this by removing those pairs and we will investigate the source of the error. Please rerun your processing chain, or just delete following directories from the GEOC folder: 20200505_20200517 20200529_20200622 20200622_20200716 20200704_20200728 20200728_20200809 20200728_20200902 20200809_20200821 20200809_20200902 20200809_20200914 20200821_20200902 20200821_20200914 20200902_20200914 20200902_20210217 20200914_20210217 út 25. 7. 2023 v 4:53 odesílatel Feng-Nian Chang @.> napsal:
…
I meet the same problem in area 063D_04886_161920. An error occurred (cannot reshape array of size 2337741 into shape) in step05 or step13. It seems that the file size of [ENU].tif and hgt.tif is inconsistent with the unw.tif. — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYYKQ65BQN3HHOAV4Y5IDXR47NFANCNFSM4ONPU24Q . You are receiving this because you commented.Message ID: @.***>

Thank you for your prompt reply! However, the error still occurred after deleting the problematic directory. In the step5 (Clip) it says:
1

And also, if I don't use the clip step, it will end with step13 (NSBAS inversion):
2

The problem has not been solved. In fact, I have removed these unwrapped data with inconsistent sizes in the last processing. So it seems that the problem is not because of this, I guess it may have something to do with [ENU].tif and hgt.tif?

Best regards!
Fengnian Chang

@espiritocz
Copy link

espiritocz commented Jul 27, 2023 via email

@fengnian-chang
Copy link

fengnian-chang commented Jul 27, 2023 •
edited
Loading

hi, indeed, the problem remained in the latest mli.tif files on LiCSAR side. fixed now. M. st 26. 7. 2023 v 3:14 odesílatel Feng-Nian Chang @.> napsal:
…
thank you for reporting this issue with the LiCSAR data, frame 063D_04886_161920. I see there were 14 ifg pairs of the frame with this issue. We fixed this by removing those pairs and we will investigate the source of the error. Please rerun your processing chain, or just delete following directories from the GEOC folder: 20200505_20200517 20200529_20200622 20200622_20200716 20200704_20200728 20200728_20200809 20200728_20200902 20200809_20200821 20200809_20200902 20200809_20200914 20200821_20200902 20200821_20200914 20200902_20200914 20200902_20210217 20200914_20210217 út 25. 7. 2023 v 4:53 odesílatel Feng-Nian Chang @. > napsal: … <#m_-1295850161274870515_> I meet the same problem in area 063D_04886_161920. An error occurred (cannot reshape array of size 2337741 into shape) in step05 or step13. It seems that the file size of [ENU].tif and hgt.tif is inconsistent with the unw.tif. — Reply to this email directly, view it on GitHub <#25 (comment) <#25 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYYKQ65BQN3HHOAV4Y5IDXR47NFANCNFSM4ONPU24Q . You are receiving this because you commented.Message ID: @.> Thank you for your prompt reply! However, the error still occurred after deleting the problematic directory. In the step5 (Clip) it says: [image: 1] https://user-images.githubusercontent.com/62586311/256085134-330f92d4-869e-4fd3-b36d-abcf7cef0074.png And also, if I don't use the clip step, it will end with step13 (NSBAS inversion): [image: 2] https://user-images.githubusercontent.com/62586311/256085408-20cabba2-b663-43e7-b235-5cc2df40baa5.png The problem has not been solved. In fact, I have removed these unwrapped data with inconsistent sizes in the last processing. So it seems that the problem is not because of this, I guess it may have something to do with [ENU].tif and hgt.tif? Best regards! Fengnian Chang — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYYKSUUDLXBM6I2XMTTE3XSB4P5ANCNFSM4ONPU24Q . You are receiving this because you commented.Message ID: @.***>

Thank you and your team for your efforts to provide such a good InSAR product, looking forward to your early reply.

@fengnian-chang
Copy link

fengnian-chang commented Sep 18, 2023 •
edited
Loading

hi, indeed, the problem remained in the latest mli.tif files on LiCSAR side. fixed now. M. st 26. 7. 2023 v 3:14 odesílatel Feng-Nian Chang @.> napsal:
…
thank you for reporting this issue with the LiCSAR data, frame 063D_04886_161920. I see there were 14 ifg pairs of the frame with this issue. We fixed this by removing those pairs and we will investigate the source of the error. Please rerun your processing chain, or just delete following directories from the GEOC folder: 20200505_20200517 20200529_20200622 20200622_20200716 20200704_20200728 20200728_20200809 20200728_20200902 20200809_20200821 20200809_20200902 20200809_20200914 20200821_20200902 20200821_20200914 20200902_20200914 20200902_20210217 20200914_20210217 út 25. 7. 2023 v 4:53 odesílatel Feng-Nian Chang @. > napsal: … <#m_-1295850161274870515_> I meet the same problem in area 063D_04886_161920. An error occurred (cannot reshape array of size 2337741 into shape) in step05 or step13. It seems that the file size of [ENU].tif and hgt.tif is inconsistent with the unw.tif. — Reply to this email directly, view it on GitHub <#25 (comment) <#25 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYYKQ65BQN3HHOAV4Y5IDXR47NFANCNFSM4ONPU24Q . You are receiving this because you commented.Message ID: @.> Thank you for your prompt reply! However, the error still occurred after deleting the problematic directory. In the step5 (Clip) it says: [image: 1] https://user-images.githubusercontent.com/62586311/256085134-330f92d4-869e-4fd3-b36d-abcf7cef0074.png And also, if I don't use the clip step, it will end with step13 (NSBAS inversion): [image: 2] https://user-images.githubusercontent.com/62586311/256085408-20cabba2-b663-43e7-b235-5cc2df40baa5.png The problem has not been solved. In fact, I have removed these unwrapped data with inconsistent sizes in the last processing. So it seems that the problem is not because of this, I guess it may have something to do with [ENU].tif and hgt.tif? Best regards! Fengnian Chang — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYYKSUUDLXBM6I2XMTTE3XSB4P5ANCNFSM4ONPU24Q . You are receiving this because you commented.Message ID: @.***>

image
image

I had the same problem again in frame 056A_04947_282019 and 136D_04941_172131. Maybe there is a problem with the [ENU].tif, hgt.tif, or mli.tif?
@yumorishita @espiritocz
202309170824batch_LiCSBAS_01_16.log
202309180358batch_LiCSBAS_02_16.log

BTW, frame 136D_04941_172131 does not exist in the LiCSAR visualization map (https://comet.nerc.ac.uk/COMET-LiCS-portal/).

@espiritocz
Copy link

espiritocz commented Sep 19, 2023 via email

@fengnian-chang
Copy link

fengnian-chang commented Sep 19, 2023 •
edited
Loading

@espiritocz
Thank you for your reply and suggestion! This error may not be caused by the coherence threshold.
mask
I mask in step 04 to preserve more ifgs in step 12. otherwise many ifgs would be discarded due to large loop closure errors.
The COMET version you recommend seems to improve this by nullification of pixels rather than dropping ifgs. This is a big improvement and will be worth a try.
However, I ran into problems installing the COMET version in the same way. Is there any installation instructions or manuals?

Additionally, frame 056A_04947_282019 has the same error in step 03.
202309180358batch_LiCSBAS_02_16.log

Best regards!
Fengnian Chang

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@espiritocz @atangmao @yumorishita @fengnian-chang