[BUG] PEDC-trigger failed to poll EDRS station since 2023-07-05T15:05:27.657212

[Woljtek]

Environment:

• Delivery tag: 2.0.0
• Platform: OPS Orange Cloud

Current Behavior:
Since the date of 2023-07-05T15:05:27.657212, the RS CORE ingestion-pedc has stopped to detect data from EDRS station.

Impacts
=> So the ingestion of the session L20230525124932598000075 (07/07) was missed.
=> All the future PEDC ingestion will be lost while issue wont be fixed.

Expected Behavior:
The RS CORE ingestion-pedc shell be able to connect to EDRS.

Whenever possible, first analysis of the root cause
The root cause of the incident is an issue on certificate. Indeed, the logs are fully filled with the following stack trace: java.security.cert.CertPathValidatorException: validity check failed
Full error:

{
  "header": {
    "type": "LOG",
    "timestamp": "2023-07-10T16:03:56.540504Z",
    "level": "ERROR",
    "line": 128,
    "file": "Inbox.java",
    "thread": "scheduling-1"
  },
  "message": {
    "content": "Error on polling Inbox at ftps://154.14.100.130:21/NOMINAL for productFamily EDRS_SESSION java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connectedClient(AbstractApacheFtpClient.java:67) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.RobustFtpClient.list(RobustFtpClient.java:59) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ingestion.trigger.edip.EdipInboxAdapter.list(EdipInboxAdapter.java:36) ~[classes!/:?]\n\tat esa.s1pdgs.cpoc.ingestion.trigger.inbox.AbstractInboxAdapter.read(AbstractInboxAdapter.java:65) ~[classes!/:?]\n\tat esa.s1pdgs.cpoc.ingestion.trigger.inbox.Inbox.poll(Inbox.java:92) ~[classes!/:?]\n\tat esa.s1pdgs.cpoc.ingestion.trigger.service.IngestionTriggerService.get(IngestionTriggerService.java:30) ~[classes!/:?]\n\tat esa.s1pdgs.cpoc.ingestion.trigger.service.IngestionTriggerService.get(IngestionTriggerService.java:13) ~[classes!/:?]\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:702) ~[spring-cloud-function-context-3.2.3.jar!/:3.2.3]\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551) ~[spring-cloud-function-context-3.2.3.jar!/:3.2.3]\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.get(SimpleFunctionRegistry.java:562) ~[spring-cloud-function-context-3.2.3.jar!/:3.2.3]\n\tat org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.get(PartitionAwareFunctionWrapper.java:95) ~[spring-cloud-stream-3.2.2.jar!/:3.2.2]\n\tat org.springframework.integration.dsl.IntegrationFlows$1.doReceive(IntegrationFlows.java:174) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:142) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:212) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:444) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:413) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:348) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-5.3.18.jar!/:5.3.18]\n\tat org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$5(AbstractPollingEndpoint.java:341) ~[spring-integration-core-5.5.10.jar!/:5.5.10]\n\tat org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.3.18.jar!/:5.3.18]\n\tat org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95) [spring-context-5.3.18.jar!/:5.3.18]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]\n\tat java.lang.Thread.run(Thread.java:829) [?:?]\nCaused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:360) ~[?:?]\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:298) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]\n\tat sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]\n\tat sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]\n\tat sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]\n\tat org.apache.commons.net.ftp.FTPSClient.sslNegotiation(FTPSClient.java:283) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.ftp.FTPSClient._connectAction_(FTPSClient.java:225) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient._connect(SocketClient.java:254) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient.connect(SocketClient.java:212) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connect(AbstractApacheFtpClient.java:190) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.newClient(AbstractApacheFtpClient.java:161) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connectedClient(AbstractApacheFtpClient.java:48) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\t... 28 more\nCaused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed\n\tat sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]\n\tat sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263) ~[?:?]\n\tat sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]\n\tat sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]\n\tat sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]\n\tat sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]\n\tat org.apache.commons.net.ftp.FTPSClient.sslNegotiation(FTPSClient.java:283) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.ftp.FTPSClient._connectAction_(FTPSClient.java:225) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient._connect(SocketClient.java:254) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient.connect(SocketClient.java:212) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connect(AbstractApacheFtpClient.java:190) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.newClient(AbstractApacheFtpClient.java:161) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connectedClient(AbstractApacheFtpClient.java:48) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\t... 28 more\nCaused by: java.security.cert.CertPathValidatorException: validity check failed\n\tat sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) ~[?:?]\n\tat sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224) ~[?:?]\n\tat sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144) ~[?:?]\n\tat sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]\n\tat java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]\n\tat sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]\n\tat sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263) ~[?:?]\n\tat sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]\n\tat sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]\n\tat sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]\n\tat sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]\n\tat org.apache.commons.net.ftp.FTPSClient.sslNegotiation(FTPSClient.java:283) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.ftp.FTPSClient._connectAction_(FTPSClient.java:225) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient._connect(SocketClient.java:254) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient.connect(SocketClient.java:212) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connect(AbstractApacheFtpClient.java:190) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.newClient(AbstractApacheFtpClient.java:161) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connectedClient(AbstractApacheFtpClient.java:48) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\t... 28 more\nCaused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Jul 05 16:07:03 UTC 2023\n\tat sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277) ~[?:?]\n\tat sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:675) ~[?:?]\n\tat sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) ~[?:?]\n\tat sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) ~[?:?]\n\tat sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:?]\n\tat sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224) ~[?:?]\n\tat sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144) ~[?:?]\n\tat sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]\n\tat java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]\n\tat sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]\n\tat sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263) ~[?:?]\n\tat sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) ~[?:?]\n\tat sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]\n\tat sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]\n\tat sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[?:?]\n\tat sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) ~[?:?]\n\tat sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) ~[?:?]\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) ~[?:?]\n\tat org.apache.commons.net.ftp.FTPSClient.sslNegotiation(FTPSClient.java:283) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.ftp.FTPSClient._connectAction_(FTPSClient.java:225) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient._connect(SocketClient.java:254) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat org.apache.commons.net.SocketClient.connect(SocketClient.java:212) ~[commons-net-3.8.0.jar!/:3.8.0]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connect(AbstractApacheFtpClient.java:190) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.newClient(AbstractApacheFtpClient.java:161) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\tat esa.s1pdgs.cpoc.ebip.client.apacheftp.AbstractApacheFtpClient.connectedClient(AbstractApacheFtpClient.java:48) ~[s1pro-core-edip-client-3.32.1.jar!/:?]\n\t... 28 more\n"
  },
  "custom": {
    "logger_string": "esa.s1pdgs.cpoc.ingestion.trigger.inbox.Inbox (WILE) for EDRS_SESSION"
  }
}

[suberti-ads]

Hereafter truststore validity date

Valide du : Fri Jun 25 19:49:59 GMT 2021 au : Wed Jun 24 19:49:59 GMT 2026

[Woljtek]

A CAMS issue is opened => https://cams.esa.int/browse/PDGSANOM-13012

[Woljtek]

PS S1A confirmed it is a EDRS incident:

Dear EDRS team,
just to inform you that PEDC/BEDC server certificate is expired. For this reason we have temporary set "skipping certificate" option as to not block nominal workflow or further recovery.
Regards
S1A Processing Farm EXprivia (PS S1A SERCO)

[pcuq-ads]

System_CCB_2023-w30: The incident is still opened.