[BUG] Metadata extraction failed for some products S3B_SR_1_LAN_RD_ and S3B_SR_1_SRA_A .

[pcuq-ads]

Environment:

• Platform: OPS Orange Cloud
• Configuration: OPS
• Infrastructure v1.5.0
• RS core metadata 1.13.2-rc1
• RS add-on S3-SR1-NRT 1.13.1-rc1
• RS add-on S3-SR1-NTC 1.13.1-rc1
• RS add-on S3-SR1-STC 1.13.1-rc1

Traceability:

• Test(s):
• Requirements:

Current Behavior:
Some products S3B_SR_1_LAN_RD_ and S3B_SR_1_SRA_A failed during the JobProcessing task of the RS core metadata extraction.

Expected Behavior:
Metadata Extraction service shall extract all product metadata without failing.

Steps To Reproduce:
Check trace status for JobProcessing for the service Metadata Extraction.

Test execution artefacts (i.e. logs, screenshots…)
Here are the errors raised by RS core Metadata Extraction:

https://app.zenhub.com/files/398313496/e396b1b0-816b-4914-bae1-564759e40984/download

On the trace we see this error.

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [utcTime] of type [date] in document with id 'S3B_SR_1_LAN_RD_20230428T000006_20230428T005036_20230630T193816_3029_079_002______LN3_O_NT_002.SEN3'. Preview of field's value: ''"}],"type":"mapper_parsing_exception","reason":"failed to parse field [utcTime] of type [date] in document with id 'S3B_SR_1_LAN_RD_20230428T000006_20230428T005036_20230630T193816_3029_079_002______LN3_O_NT_002.SEN3'. Preview of field's value: ''","caused_by":{"type":"illegal_argument_exception","reason":"cannot parse empty date"}},"status":400}

Here are the logs.
https://app.zenhub.com/files/398313496/24f3793e-1ad2-43dd-ada2-6c54ff94db13/download

Whenever possible, first analysis of the root cause
Hypothesis :

Bug Generic Definition of Ready (DoR)

• The affect version in which the bug has been found is mentioned
• The context and environment of the bug is detailed
• The description of the bug is clear and unambiguous
• The procedure (steps) to reproduce the bug is clearly detailed
• The tested User Story / features is linked to the bug if available
• Logs are attached if available
• A data set attached if available

Bug Generic Definition of Done (DoD)

• the modification implemented (the solution to fix the bug) is described in the bug.
• Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
• Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
• Code is compliant with coding rules (SONAR Report as evidence)
• Acceptance criteria of the related User story are checked and Passed

[SYTHIER-ADS]

This occurs also yesterday for SY 2 data (between 10:00 and 10:30 11th July 2023)

[w-jka]

I checked the xfdumanifest.xml of the file
S3B_SR_1_LAN_RD_20230428T000006_20230428T005036_20230630T193816_3029_079_002______LN3_O_NT_002.SEN3 which is the first mentioned by the provided logs.
The extraction works fine, however the product does not contain the necessary orbit information for the system. Other products of the same day and product type do contain this information.

In this case the orbit start times (ANX and ANX1) are not present.

[suberti-ads]

New occurrence for following product type:

• SY_2_VGK_ (26/202)
• SY_2_VGP (26/202)
• SY_2_SYN (29/222)
• SY_1_MISR (29/222)

Currently we had 110 new occurences

Hereafter sample of product impacted:
S3B_SY_1_MISR___20230428T002719_20230428T002919_20230707T014536_0119_079_002______LN3_D_ST_002.SEN3

suberti@refsys-client:~/Documents/investigation/SYN_MDC$ s3cmd ls s3://ops-rs-s3-l1-stc/ | grep S3B_SY_1_MISR___20230428T002719_20230428T002919_20230707T014536_0119_079_002______LN3_D_ST_002
                       DIR   s3://ops-rs-s3-l1-stc/S3B_SY_1_MISR___20230428T002719_20230428T002919_20230707T014536_0119_079_002______LN3_D_ST_002.SEN3/
2023-07-10 09:55      5585   s3://ops-rs-s3-l1-stc/S3B_SY_1_MISR___20230428T002719_20230428T002919_20230707T014536_0119_079_002______LN3_D_ST_002.SEN3.md5sum

Hereafter xfdumanifest.xml for this product:
xfdumanifestfailed.xml.gz

hereafter product with execution OK:
S3A_SY_1_MISR___20230708T202626_20230708T202826_20230712T020958_0119_101_014______LN3_D_ST_002.SEN3
Hereafter xfdumanifest.xml for this product:
xfdumanifest.xml.gz

By comparison we have additional field found on working manifest:
ephemeris

[w-jka]

@suberti-ads
As mentioned by my last comment, these missing fields are the reason, why the metadata extraction is failing, as these are containing necessary orbit information. I am however unsure why the IPF is producing these xml-files without the orbit information.

[LAQU156]

IVV_CCB_2023_w28 : Moved into "Accepted Werum" for discussions. Open question : Is it a mandatory field on ICD metadata from ESA ? Do we need to create an anomaly on ESA side ? Priority blocking, to be fixed phase 1

[LAQU156]

Werum_CCB_2023_w28 : Moved into "Product Backlog" for further analysis

[w-jka]

@Woljtek @pcuq-ads
As Fabien stated in the last CCB that the fields containing the orbit information might not be mandatory: While the fields are not to be included on the PRIP the fields are indeed mandatory by the Sentinel-3 Product Attributes Mapping.v1.6.xlsx. The cardinality for the fields

is not 0..3 but 3, meaning they are not to be left out.

[pcuq-ads]

@w-jka
We only have the version 1.3 of the ICD on our side. Regardless, the definition is the same on version 1.3 and 1.6 for this field.

What is no clear for me is perhaps the way to read the document. I understand on the applicability section, column "PR", that this field is not to be provided

Is that right ? Otherwise, an anomaly must be created on the ESA side.

[w-jka]

@pcuq-ads
The document contains different kinds of information. The parts on the left are listing the specifics of the Sentinel-3 SAFE format. I would assume that there should also be some kind of xsd for this, however I was unable to find one.

The columns on the right are for the metadata information that have to be exposed to end users of the different systems. As the orbit information is an information that is only used internally, it is never exposed to the end user directly, and all the columns are empty.

From our point of view the missing metadata is an anomaly of the IPF and should be communicated to ESA.

[pcuq-ads]

@w-jka ,
With this missing information, the chain is broken on the Rs Core Metadata Extraction. Is it possible to published the catalog-event and let the compression, the distribution and all other processing running well ?
We will communicate the lack of the information from the IPF.
Regards

[w-jka]

@pcuq-ads
I checked the implementation. The orbit information is used at the PUG step for almost all products, as well as for the SLSTR VISCAL product. So the processing will not run well without major changes that would have to be handled as a change.

[pcuq-ads]

SYS_CCB_w29 : action for @pcuq-ads : create issue on ESA side.

[Woljtek]

New occurrences with the 4 following products:

- S3B_SY_2_VGK____20230428T000321_20230428T004740_20230722T174302_2659_079_002______LN3_D_NT_002.SEN3
- S3B_SY_2_VGP____20230428T000321_20230428T004740_20230722T174259_2659_079_002______LN3_D_NT_002.SEN3
- S3B_SY_2_SYN____20230428T000321_20230428T004740_20230722T174247_2659_079_002______LN3_D_NT_002.SEN3
- S3B_SY_1_MISR___20230428T000321_20230428T004740_20230722T155923_2659_079_002______LN3_D_NT_002.SEN3

[suberti-ads]

New occurrence this week for following products:

S3B_SY_2_VGK____20230428T000321_20230428T004740_20230722T174302_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_VGK____20230428T000321_20230428T004740_20230722T174302_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_VGK____20230428T000321_20230428T004740_20230722T174302_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_VGP____20230428T000321_20230428T004740_20230722T174259_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_VGP____20230428T000321_20230428T004740_20230722T174259_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_VGP____20230428T000321_20230428T004740_20230722T174259_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_SYN____20230428T000321_20230428T004740_20230722T174247_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_SYN____20230428T000321_20230428T004740_20230722T174247_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_2_SYN____20230428T000321_20230428T004740_20230722T174247_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_1_MISR___20230428T000321_20230428T004740_20230722T155923_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_1_MISR___20230428T000321_20230428T004740_20230722T155923_2659_079_002______LN3_D_NT_002.SEN3
S3B_SY_1_MISR___20230428T000321_20230428T004740_20230722T155923_2659_079_002______LN3_D_NT_002.SEN3

[pcuq-ads]

Werum_CCB_2023_w30 : remaining action is on ADS side.

[LAQU156]

System_CCB_2023_w31 : Moved into "On hold", Action on ADS side to open a PSC ticket, will be corrected once IPF will be delivered. Limitation.

[Woljtek]

The issue PSC-66 has been created on ESA side.
We are waiting for feedback to go further.