SB 5.1 - OCI - Deploy/UCI/Orchestrator pod going into crashloopbackoff

[subhashchandrab]

Following the documentation https://ed.sunbird.org/use/installing-ed-platform/release-5.1.0/build

After Build and Deploy job execution, we noticed that the orchestrator pod is entering the CrashLoopBackoff state as it's startup is failing due to the following error.

2023-02-24 10:56:19.056  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2023-02-24 10:56:19.127  INFO 1 --- [           main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-02-24 10:56:19.206 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'reactiveConsumer': Unsatisfied dependency expressed through field 'xMessageRepository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'XMessageRepository' defined in com.uci.dao.repository.XMessageRepository defined in @EnableReactiveCassandraRepositories declared on CassandraConfig: Cannot resolve reference to bean 'reactiveCassandraTemplate' while setting bean property 'reactiveCassandraOperations'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'reactiveCassandraTemplate' defined in class path resource [com/uci/dao/config/CassandraConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.core.ReactiveCassandraTemplate]: Factory method 'reactiveCassandraTemplate' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'reactiveCassandraSessionFactory' defined in class path resource [com/uci/dao/config/CassandraConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.ReactiveSessionFactory]: Factory method 'reactiveCassandraSessionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'reactiveCassandraSession' defined in class path resource [com/uci/dao/config/CassandraConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.ReactiveSession]: Factory method 'reactiveCassandraSession' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/uci/dao/config/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: Invalid column name sessionid because it conflicts with an existing column
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[message-rosa-5.0.0.jar!/:5.0.0]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[message-rosa-5.0.0.jar!/:5.0.0]

As we can see the from the logs, the orchestrator pod is trying to alter the cassandra table xmessage to add some additional columns which already exists. Looks like the DAO definition(https://github.com/samagra-comms/dao/blob/release-5.1.0/src/main/java/com/uci/dao/models/XMessageDAO.java) is already updated in the 5.1.0 release to have the columns sessionId, ownerOrgId, ownerId, botUuid. However, the application startup script(getMigrationScripts() method in https://github.com/samagra-comms/dao/blob/release-5.1.0/src/main/java/com/uci/dao/config/CassandraConfig.java) is still trying to add these columns resulting in the failure.

I tried to manually edit the cassandra table to remove these additional columns and started the pod. This resolved the crashloopbackoff problem and the pod is running fine now. This may need to be fixed in the samagra-comms dao codebase.(release-5.1.0).

[subhashchandrab]

As per my understanding, the ALTER queries in the below lines(line numbers 124-127) need to be modified to include the clause IF NOT EXISTS so that column name conflicting error can be avoided.

https://github.com/samagra-comms/dao/blob/a450907889eb68a8046fb50f9aef6311fb595cf0/src/main/java/com/uci/dao/config/CassandraConfig.java#L124

[Krishnaj20]

@vinukumar-vs @rajeevsathish - FYA please.

[chitranshu-keshav]

@subhashchandrab A thread has been raised in the UCI Building Blocks discussion forum (samagra-comms/community#33). Please refer the same for any update on this issue

cc: @vinukumar-vs @Krishnaj20 @anandvarada @ChakshuGautam @abhay-samagra