Octorpki taking longer time to process the ROAs because of the parent certificate issue

[PrajaktaYPatil]

We are facing the issue with the octorpki as it is taking longer time to process ROAs. We are getting the expiration issue for certificate and parent issue for certificate error.

Please find the below errors for certificate from the octorpki -

[root@rpki01.syd ~]# grep f57670700d60 /var/log/messages
Oct 27 04:30:57 rpki01 f57670700d60[1246]: time="2021-10-26T17:30:57Z" level=info msg="Stable state. Revalidating in 20m0s"
Oct 27 04:50:57 rpki01 f57670700d60[1246]: time="2021-10-26T17:50:57Z" level=info msg="RRDP sync https://rrdp.arin.net/notification.xml"
Oct 27 04:50:57 rpki01 f57670700d60[1246]: time="2021-10-26T17:50:57Z" level=info msg="RRDP: Downloading root notification https://rrdp.arin.net/notification.xml"
Oct 27 04:50:58 rpki01 f57670700d60[1246]: time="2021-10-26T17:50:58Z" level=info msg="RRDP: https://rrdp.arin.net/notification.xml has 0 deltas to parse (cur: 49882, last: 49882)"
Oct 27 04:50:58 rpki01 f57670700d60[1246]: time="2021-10-26T17:50:58Z" level=info msg="RRDP: finished processing notifications (0). Last serial 49882"
Oct 27 04:50:58 rpki01 f57670700d60[1246]: time="2021-10-26T17:50:58Z" level=info msg="RRDP sync https://rpki1.terratransit.de/rrdp/notification.xml"
Oct 27 04:50:58 rpki01 f57670700d60[1246]: time="2021-10-26T17:50:58Z" level=info msg="RRDP: Downloading root notification https://rpki1.terratransit.de/rrdp/notification.xml"
…..
…..
Oct 27 04:54:38 rpki01 f57670700d60[1246]: time="2021-10-26T17:54:38Z" level=error msg="expiration issue for certificate ski:a83ab2c6ed0667e3b1d12693ae280b538292338c aki:b97dc46099cd50fcf299da5e82b005646984c4f3: Certificate beginning of validity: 2021-11-07 06:51:50 +0000 UTC is after: 2021-10-26 17:54:01.517023988 +0000 UTC"
Oct 27 04:54:40 rpki01 f57670700d60[1246]: time="2021-10-26T17:54:40Z" level=error msg="expiration issue for certificate ski:be9ebd64aa77c69dc3653c62d9405896fe546ab2 aki:797d88d813e20fff982cc7419e969baea6bfd69b: Certificate end of validity: 2021-08-26 07:58:47 +0000 UTC is before: 2021-10-26 17:54:01.517023988 +0000 UTC"
Oct 27 04:54:41 rpki01 f57670700d60[1246]: time="2021-10-26T17:54:41Z" level=error msg="expiration issue for certificate ski:f54618c9a2a7b6e0dcdcd1d6329206ddab19dde1 aki:797d88d813e20fff982cc7419e969baea6bfd69b: Certificate end of validity: 2021-08-26 07:58:12 +0000 UTC is before: 2021-10-26 17:54:01.517023988 +0000 UTC"
Oct 27 04:57:33 rpki01 f57670700d60[1246]: time="2021-10-26T17:57:33Z" level=error msg="revocation due to manifest issue for certificate ski:5d35939557110cc43429ae301f7cef0e5889942b aki:0e65a4f5fd36b5bd68eb3c923408978c907aa79f"
Oct 27 04:57:33 rpki01 f57670700d60[1246]: time="2021-10-26T17:57:33Z" level=error msg="parent issue for certificate ski:408d4ce1b008c186db6933dbf9ee9e175e07030c aki:5d35939557110cc43429ae301f7cef0e5889942b: missing parent"
Oct 27 04:57:33 rpki01 f57670700d60[1246]: time="2021-10-26T17:57:33Z" level=error msg="revocation due to manifest issue for certificate ski:5d35939557110cc43429ae301f7cef0e5889942b aki:0e65a4f5fd36b5bd68eb3c923408978c907aa79f"
Oct 27 04:57:33 rpki01 f57670700d60[1246]: time="2021-10-26T17:57:33Z" level=error msg=""
Oct 27 05:07:21 rpki01 f57670700d60[1246]: time="2021-10-26T18:07:21Z" level=error msg="expiration issue for certificate ski:5b2576145345854e2fa3c5456230e6d834186699 aki:47ca36b68f7ebb87a8e744a7072f9e3d860263a1: Certificate end of validity: 2021-06-18 16:21:00 +0000 UTC is before: 2021-10-26 18:01:50.898144585 +0000 UTC"
Oct 27 05:07:21 rpki01 f57670700d60[1246]: time="2021-10-26T18:07:21Z" level=error msg="revocation due to manifest issue for certificate ski:fa048af3ea62e5575b4698c5cc2403982893f0c5 aki:721e5b49438ec0c95d40798ba788778fa954cb3a"
……
……
Oct 27 05:18:43 rpki01 f57670700d60[1246]: time="2021-10-26T18:18:43Z" level=error msg="revocation due to manifest issue for certificate ski:7ea5316dbdac01cd05d0eaa05a89c04da3e7398f aki:2a94a8dd554ae701072099c70b6407555ddde669"
Oct 27 05:18:43 rpki01 f57670700d60[1246]: time="2021-10-26T18:18:43Z" level=error msg="parent issue for certificate ski:d885844510c0c353cbdf685184804d1b970632e8 aki:7ea5316dbdac01cd05d0eaa05a89c04da3e7398f: missing parent"
Oct 27 05:18:43 rpki01 f57670700d60[1246]: time="2021-10-26T18:18:43Z" level=error msg="revocation due to manifest issue for certificate ski:7ea5316dbdac01cd05d0eaa05a89c04da3e7398f aki:2a94a8dd554ae701072099c70b6407555ddde669"
Oct 27 05:18:43 rpki01 f57670700d60[1246]: time="2021-10-26T18:18:43Z" level=error msg="parent issue for certificate ski:9efc117bc3c7ebb263a2031b543cf3bf4e09a224 aki:7ea5316dbdac01cd05d0eaa05a89c04da3e7398f: missing parent"
……
…..
Oct 27 05:20:48 rpki01 f57670700d60[1246]: time="2021-10-26T18:20:48Z" level=info msg="Stable state. Revalidating in 20m0s"

Below mentioned line shows that the validator was in stable state after stable state it starts the validation -
Oct 27 04:30:57 rpki01 f57670700d60[1246]: time="2021-10-26T17:30:57Z" level=info msg="Stable state. Revalidating in 20m0s"
During the process of validation it starts getting the error parent issue for certificate ski:a737a17cf23ac890b4b67ca41eed6ee8cdaaeb1d aki:2bbfef195f7b4083df377b2ffbc77b50dd100ae3: missing parent. It took around 15 -20 mins to come out of it and then again it went in to validation state. So this error is basically increasing the processing time.
Oct 27 05:20:48 rpki01 f57670700d60[1246]: time="2021-10-26T18:20:48Z" level=info msg="Stable state. Revalidating in 20m0s"
So if we see the total time it’s around 50 mins.
I also ran the below command to check the stable state for validator(Cloudflare)

root@rpki01.syd prod]# grep -i 'Stable state' /var/log/messages
Oct 27 04:30:57 rpki01 f57670700d60[1246]: time="2021-10-26T17:30:57Z" level=info msg="Stable state. Revalidating in 20m0s"
Oct 27 05:20:48 rpki01 f57670700d60[1246]: time="2021-10-26T18:20:48Z" level=info msg="Stable state. Revalidating in 20m0s”

Please find the attached log file (detailed) from the ocrtorpki -
Cloudflare_log.txt