[OPS] S1 L0ASP Job stuck in preparation worker due to true Ghost #1089
Labels
bug
Something isn't working
CCB
Issue for CCB
ops
Ticket from ADS operation team
workaround
Workaround activated
Environment:
rs-addon s1-l0asp v1.15
Traceability:
Current Behavior:
L0ASP doesn't publish job despite of reception of FULL segment.
Expected Behavior:
When all data available preparation worker should generate job to be processed.
Steps To Reproduce:
This issue has been seen on S1 Sample production.
To reproduced it :
Test execution artefacts (i.e. logs, screenshots…)
Preparation log (job 152500 impacted)
s1-l0asp-part1-preparation-worker-v31-5b75f68487-86wmp.log
Whenever possible, first analysis of the root cause
DT 06A5D5 has been downloaded on 2 Stations (SGS and MTI)
2 Segment generated :
Due to ingestion issue first Segment generated was the end of this DT(S1A_IW_RAW__0SSH_20240704T150234_20240704T150234_054612_06A5D5_8834.SAFE)
Preparation worker job stuck in INITIAL State (other test with job 152757)
Following information in this job, It only missing:
ORBSCT and MPL_ORBPRE.
These 2 AUX were available on catalog.
When we search l0asp preparation logs it seems that segment was not detected has complete
Hypthesys "True Ghost" prevent the prepation worker to detect FULL segment.
Test to delete Ghost from catalog and regenerate job ==> Production OK
Impact: Timeliness on DT.
Workaround ( delete Job, scale down preparation worker, delete "Ghost Segment", scale up preparation worker, published "FULL" segment)
Bug Generic Definition of Ready (DoR)
Bug Generic Definition of Done (DoD)
The text was updated successfully, but these errors were encountered: