Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [OPS] S3-PUG-STC execution failed "[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]" due to "Illegal Value: OLCI missing lines global info" #1080

Open
12 tasks
suberti-ads opened this issue Aug 21, 2023 · 6 comments
Open
12 tasks

[BUG] [OPS] S3-PUG-STC execution failed "[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]" due to "Illegal Value: OLCI missing lines global info" #1080

suberti-ads opened this issue Aug 21, 2023 · 6 comments
Assignees
@w-fsi
Labels
bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team priority:blocking Set the priority to blocking because the production is blocked WERUM dev Ticket dedicated to WERUM development

Comments

@suberti-ads
Copy link

suberti-ads commented Aug 21, 2023 •
edited by Woljtek
Loading

Environment:

  • Delivery tag:
  • Platform: OPS Orange Cloud
  • Configuration:
    PUG-STC 1.14.0-rc1

Traceability:

Current Behavior:
PUG-STC execution for product SY_2_SYN fall in error

[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]

Expected Behavior:
Production successfully done with nominal production input.

Steps To Reproduce:
Production 3%

Test execution artefacts (i.e. logs, screenshots…)
Execution logs:
pug-stc-post-core-JobOrder.131506.xml.log

Whenever possible, first analysis of the root cause
All issue were found on production with input SY_2_SYN
sample for joborder JobOrder.131506.xml
Product input :

"S3B_AX___FRO_AX_20230727T000000_20230806T000000_20230730T065449___________________EUM_O_AL_001.SEN3"
"S3B_SY_2_SYN____20230729T203841_20230729T204041_20230730T091811_0119_082_171______LN3_D_ST_002.SEN3"
"S3B_SY_2_SYN____20230729T203641_20230729T203841_20230730T093239_0119_082_171______LN3_D_ST_002.SEN3"
"JobOrder.131506.xml"

On log there was following issue during production

2023-07-30T15:21:22+00:00	{"header":{"type":"LOG","timestamp":"2023-07-30T15:21:22.400898Z","level":"INFO","line":129,"file":"TaskCallable.java","thread":"pool-35-thread-1"},"message":{"content":"Ending task /usr/local/components/PUG-3.48/bin/PUGPostProcessor with exit code 255"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ipf.execution.worker.job.process.TaskCallable"}}
2023-07-30T15:21:22+00:00	2023-07-30T15:21:22.392721 s3-pug-stc-part1-execution-worker-v1-6d7567f956-ld27p PUG_SY_2_SYN 03.48 [0000000140]: [I] PUGPostProcessor: Exiting with EXIT CODE: 255
2023-07-30T15:21:22+00:00		FATAL: All the product data unit generations exited in error!
2023-07-30T15:21:22+00:00			 must have size 1.2023-07-30T15:21:22.392605 s3-pug-stc-part1-execution-worker-v1-6d7567f956-ld27p PUG_SY_2_SYN 03.48 [0000000140]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140399911340224]
2023-07-30T15:21:22+00:00		acs::ManifestBuilderS3SynL1L2::ManifestBuilderS3SynL1L2Exception in ManifestBuilderS3SynL1L2.C(370) from virtual void acs::ManifestBuilderS3SynL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140399393896192]
2023-07-30T15:21:22+00:00		caused by:
2023-07-30T15:21:22+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2023-07-30T15:21:22+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140399911340224]
2023-07-30T15:21:22+00:00	2023-07-30T15:21:22.332761 s3-pug-stc-part1-execution-worker-v1-6d7567f956-ld27p PUG_SY_2_SYN 03.48 [0000000140]: [I] PUGPostProcessor: writeProduct DONE.

It seems to be an issue with input product.
i have 2 hypothesys:

  • issue on preparation worker which not select good input
  • Issue on SY_2_SYN production

Bug Generic Definition of Ready (DoR)

  • The affect version in which the bug has been found is mentioned
  • The context and environment of the bug is detailed
  • The description of the bug is clear and unambiguous
  • The procedure (steps) to reproduce the bug is clearly detailed
  • The tested User Story / features is linked to the bug if available
  • Logs are attached if available
  • A data set attached if available

Bug Generic Definition of Done (DoD)

  • the modification implemented (the solution to fix the bug) is described in the bug.
  • Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
  • Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
  • Code is compliant with coding rules (SONAR Report as evidence)
  • Acceptance criteria of the related User story are checked and Passed
@suberti-ads suberti-ads added bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team priority:blocking Set the priority to blocking because the production is blocked labels Aug 21, 2023
@Woljtek
Copy link

Woljtek commented Aug 25, 2023

@w-jka
On OPS side, we don't understand why the IPF crashes.
Could you provide to us your feedback ?

@Woljtek Woljtek added the WERUM dev Ticket dedicated to WERUM development label Aug 25, 2023
@Woljtek
Copy link

Woljtek commented Sep 26, 2023

A clarification has been requested from ESA: PSC-68 PUG_SY_2_SYN ununderstood exPDUGeneratorException
We are waiting for feedback to go further.

@suberti-ads
Copy link
Author

suberti-ads commented Jan 19, 2024 •
edited
Loading

We had new occurrence on sample processing report on PUG-NRT
New Product type impacted: OL_1_ERR, OL_2_LRR, OL_2_LRR, OL_2_LRR, SL_1_RBT

Same error seen

2024-01-17T23:20:19+00:00	{"header":{"type":"REPORT","timestamp":"2024-01-17T23:20:19.055000Z","level":"ERROR","mission":"S3","workflow":"NOMINAL","rs_chain_name":"S3-PUG-NRT","rs_chain_version":"1.14.0"},"message":{"content":"[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]"},"task":{"uid":"1b73463c-7860-4955-a769-2d084e0c748a","name":"JobProcessing","event":"END","status":"NOK","output":{},"input":{"job_order_id_string":"JobOrder.146541.xml","ipf_release_string":"03.48","filename_strings":["JobOrder.146541.xml","S3B_OL_1_ERR____20240117T144238_20240117T144438_20240117T174035_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145038_20240117T145238_20240117T172440_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150238_20240117T150438_20240117T172440_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T152038_20240117T152232_20240117T174008_0114_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144438_20240117T144638_20240117T172440_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151438_20240117T151638_20240117T171638_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151638_20240117T151838_20240117T171639_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144038_20240117T144238_20240117T173224_0119_088_310______LN3_D_NR_002.SEN3","S3B_AX___FRO_AX_20240114T000000_20240124T000000_20240117T064525___________________EUM_O_AL_001.SEN3","S3B_OL_1_ERR____20240117T150838_20240117T151038_20240117T171637_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T143840_20240117T144038_20240117T170840_0118_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144638_20240117T144838_20240117T173237_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151038_20240117T151238_20240117T173226_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150638_20240117T150838_20240117T173229_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144838_20240117T145038_20240117T172441_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150438_20240117T150638_20240117T173226_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145638_20240117T145838_20240117T172441_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145438_20240117T145638_20240117T174016_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150038_20240117T150238_20240117T171637_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151838_20240117T152038_20240117T175633_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145838_20240117T150038_20240117T174834_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145238_20240117T145438_20240117T171637_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151238_20240117T151438_20240117T170840_0119_088_310______LN3_D_NR_002.SEN3"]},"quality":{},"error_code":1,"duration_in_seconds":187.733,"missing_output":[{"product_metadata_custom_object":{"product_type_string":"OL_1_ERR___","platform_serial_identifier_string":"B"},"end_to_end_product_boolean":true,"estimated_count_integer":1}]}}

Previous IPF logs error which:

2024-01-17T23:18:54+00:00	{"header":{"type":"LOG","timestamp":"2024-01-17T23:18:54.962501Z","level":"INFO","line":129,"file":"TaskCallable.java","thread":"pool-1025-thread-1"},"message":{"content":"Ending task /usr/local/components/PUG-3.48/bin/PUGPostProcessor with exit code 255"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ipf.execution.worker.job.process.TaskCallable"}}
2024-01-17T23:18:54+00:00	2024-01-17T23:18:54.954289 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [I] PUGPostProcessor: Exiting with EXIT CODE: 255
2024-01-17T23:18:54+00:00		FATAL: All the product data unit generations exited in error!
2024-01-17T23:18:54+00:00		Illegal Value: OLCI missing lines global info must have size 1.2024-01-17T23:18:54.954146 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140025336170688]
2024-01-17T23:18:54+00:00		acs::ManifestBuilderS3OlciL1L2::ManifestBuilderS3OlciL1L2Exception in ManifestBuilderS3OlciL1L2.C(306) from virtual void acs::ManifestBuilderS3OlciL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140024886081280]
2024-01-17T23:18:54+00:00		caused by:
2024-01-17T23:18:54+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-01-17T23:18:54+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140025336170688]

First Error seem link to this issue:

XMLException caugth: "{140024886081280} @ acs::XMLHandler::hpTrimmedString(...):[200] ": TranscodingException, invalid multi-byte sequence
2024-01-17T23:18:02+00:00	2024-01-17T23:18:02.105971 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [E] PUGPostProcessor: XMLException caugth: "{140024886081280} @ acs::XMLHandler::hpTrimmedString(...):[200] ": TranscodingException, invalid multi-byte sequence
2024-01-17T23:18:02+00:00	2024-01-17T23:18:02.105692 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [E] PUGPostProcessor: XMLException caugth: "{140024886081280} @ acs::XMLHandler::hpTrimmedString(...):[200] ": TranscodingException, invalid multi-byte sequence
2024-01-17T23:18:02+00:00	2024-01-17T23:18:02.098524 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [W] PUGPostProcessor: MetadataFormatter::updatedOutputFilename : using hardwar ID got from the input filename.

Last part was not seen on another error logs.

sample log for OL_1_ERR:
Explore-logs-2024-01-19_13_19_42.gz
sample log for SL_1_RBT :
s3-pug-nrt-part1-execution-worker-v51-747c777695-cfcbm.log.gz

@suberti-ads
Copy link
Author

suberti-ads commented Apr 12, 2024 •
edited
Loading

New Occurrence for SL_1_RBT on PUG-NRT
Product impacted (one execution)

S3B_SL_1_RBT____20240411T140345_20240411T140845_20240411T204550_0299_091_367______LN3_O_NR_002.SEN3
S3B_SL_1_RBT____20240411T135845_20240411T140345_20240411T183010_0299_091_366______LN3_O_NR_002.SEN3

Error

2024-04-11T21:06:55+00:00	2024-04-11T21:06:55.641690 s3-pug-nrt-part1-execution-worker-v52-cf7758c9d-l7sbw PUG_SL_1_RBT 03.48 [0000001425]: [I] PUGPostProcessor: Exiting with EXIT CODE: 255
2024-04-11T21:06:55+00:00		FATAL: All the product data unit generations exited in error!
2024-04-11T21:06:55+00:00		Illegal Value: SLSTR missing lines global info must have size < 8.2024-04-11T21:06:55.641565 s3-pug-nrt-part1-execution-worker-v52-cf7758c9d-l7sbw PUG_SL_1_RBT 03.48 [0000001425]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [139955093186752]
2024-04-11T21:06:55+00:00		acs::ManifestBuilderS3SlstrL1L2::ManifestBuilderS3SlstrL1L2Exception in ManifestBuilderS3SlstrL1L2.C(287) from virtual void acs::ManifestBuilderS3SlstrL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [139954643080960]
2024-04-11T21:06:55+00:00		caused by:
2024-04-11T21:06:55+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-04-11T21:06:55+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [139955093186752]

@suberti-ads
Copy link
Author

New Occurrence for SY_2_SYN on PUG-STC
Product impacted (one execution)

S3B_SY_2_SYN____20240411T131622_20240411T131822_20240411T213708_0119_091_366______LN3_D_ST_002.SEN3
S3B_SY_2_SYN____20240411T131845_20240411T132022_20240411T173823_0096_091_366______LN3_D_ST_002.SEN3

Error:

2024-04-11T22:10:40+00:00		FATAL: All the product data unit generations exited in error!
2024-04-11T22:10:40+00:00		Illegal Value: OLCI missing lines global info must have size 1.2024-04-11T22:10:40.652555 s3-pug-stc-part1-execution-worker-v12-d7546576c-zbdtd PUG_SY_2_SYN 03.48 [0000000089]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140160865097920]
2024-04-11T22:10:40+00:00		acs::ManifestBuilderS3SynL1L2::ManifestBuilderS3SynL1L2Exception in ManifestBuilderS3SynL1L2.C(370) from virtual void acs::ManifestBuilderS3SynL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140160415008512]
2024-04-11T22:10:40+00:00		caused by:
2024-04-11T22:10:40+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-04-11T22:10:40+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140160865097920]

@suberti-ads
Copy link
Author

No CAMS still created on this subject ==> To be created

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@w-fsi w-fsi

Labels
bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team priority:blocking Set the priority to blocking because the production is blocked WERUM dev Ticket dedicated to WERUM development
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Woljtek @w-fsi @suberti-ads