Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [OPS] S2 S2-L1B execution failed "Error during task L1B - Exit code: 1" due to missing DEM data file #1087

Open
12 tasks
suberti-ads opened this issue Feb 6, 2024 · 0 comments
Open
12 tasks

[BUG] [OPS] S2 S2-L1B execution failed "Error during task L1B - Exit code: 1" due to missing DEM data file #1087

suberti-ads opened this issue Feb 6, 2024 · 0 comments
Labels
bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team

Comments

@suberti-ads
Copy link

suberti-ads commented Feb 6, 2024 •
edited
Loading

Environment:

  • Delivery tag:
  • Platform: OPS Orange Cloud
  • Configuration:
    S2-L1 1.8.0

Traceability:

Current Behavior:
S2 L1B execution failed with following error "Error during task L1B - Exit code: 1"

Expected Behavior:
Execution should be successfully done.

Steps To Reproduce:
Sample processing S2, error on following execution:

[
 {
  "filename": "S2A_OPER_MSI_L1A_DS_REFS_20240129T181737_S20240129T113720_N04.00.L1SA"
 },
 {
  "filename": "S2A_OPER_MSI_L1B_DS_REFS_20240129T181737_S20240129T113720_N04.00.L1SA"
 }
]

Test execution artefacts (i.e. logs, screenshots…)
pod execution logs:
Explore-logs-2024-01-30 08_18_31.txt
IPF logs Errors:
logs_FAILURE_L1B_T000_20240130085218.tar.gz

Whenever possible, first analysis of the root cause

There was 2 weird behavior.
first one:
Orchestration launcher last execution percent seen at 27% after 489 seconds:

2024-01-29T18:50:31+00:00	{"header":{"type":"LOG","timestamp":"2024-01-29T18:50:31.942363Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"2024-01-29 18:50:31 [INFO] OrchestratorLauncher : #########################"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
2024-01-29T18:50:31+00:00	{"header":{"type":"LOG","timestamp":"2024-01-29T18:50:31.942080Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"9"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
2024-01-29T18:50:26+00:00	{"header":{"type":"LOG","timestamp":"2024-01-29T18:50:26.927364Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"2024-01-29 18:50:26 [INFO] OrchestratorLauncher : Processing L1B  : 27% in 489 seconds : 1 / 1 : 14 / 14 : 12 / 24 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : "},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}

Many step RESAMPLE_TO_VS execution failed:


{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227610Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808082         13  08:59:15          99    280  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227655Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808084         14  08:59:15         100    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227677Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808088         15  08:59:15          99    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227701Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808090         16  08:59:15         100    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227724Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815809         17  09:00:55          15    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227748Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815811         18  09:00:55          16    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227771Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815859         19  09:00:55          15    273  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227802Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815905         20  09:00:56          16    270  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227827Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816705         21  09:01:11          16    255  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227859Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816706         22  09:01:11          15    255  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227890Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816755         23  09:01:11          16    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227917Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816801         24  09:01:12          16    276  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227950Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     4  TP_COLLECT                   0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227981Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     5  TP_FILTER                    0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228010Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     6  SPATIO                       0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228042Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     7  GEO1B_FINALIZE               0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228072Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     8  FORMAT_METADATA_DS_L1B       0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228098Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     9  GET_TILE_LIST                0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}

After investigation on IPF log We saw an warning on DEM missing file


2024-01-30T09:00:50.098000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/functional/cpp/Process.C:process:115: RESAMPLE_TO_VS Process:: Resampling
2024-01-30T09:00:52.549000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n10
2024-01-30T09:00:52.674000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w018/n09
2024-01-30T09:00:52.750000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n09
2024-01-30T09:00:52.757000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w016/n09
2024-01-30T09:00:52.891000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n08
2024-01-30T09:00:52.893000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w016/n08
2024-01-30T09:00:52.952000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w015/n08
2024-01-30T09:00:53.687000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/functional/cpp/Process.C:generateVATFGRI:340: Loading GRI_ORBIT: /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/input/GRI_ORBIT/S2A_OPER_MSI_L1B_DS_EPA__20160615T072312_S20160609T113637_N02.02
2024-01-30T09:00:54.916000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/libcommonGMV/functional/filehandler/cpp/GMVFileUtils.C:callProductReportWriter:111: The file /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/REPORT_808082/report.xml REPORT has been created
2024-01-30T09:00:54.923000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/functional/algorithms/cpp/fileUtils.C:validateSchema:1245: XML file: /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/REPORT_808082/report.xml validated against the schema successfully

Missing file were not found on shared directory.

heraftter error log in ipf
logs RESAMPLE_TO_VS_808082.err


2024-01-30T09:00:54.913000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[ALARM] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/cpp/main.C:start:248: Standard exception caught: AUX_DATA folder does not contain a valid GIP_VIEDIR.
2024-01-30T09:00:54.913000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[ALARM] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/cpp/main.C:start:249: Abort processing

Bug Generic Definition of Ready (DoR)

  • The affect version in which the bug has been found is mentioned
  • The context and environment of the bug is detailed
  • The description of the bug is clear and unambiguous
  • The procedure (steps) to reproduce the bug is clearly detailed
  • The tested User Story / features is linked to the bug if available
  • Logs are attached if available
  • A data set attached if available

Bug Generic Definition of Done (DoD)

  • the modification implemented (the solution to fix the bug) is described in the bug.
  • Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
  • Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
  • Code is compliant with coding rules (SONAR Report as evidence)
  • Acceptance criteria of the related User story are checked and Passed
@suberti-ads suberti-ads added bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team labels Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@suberti-ads