Update to cBioPortal cache persistence using REDIS

[n1zea144]

• get the connection to the redis server to work and to store cached values
• each portal needs to save to a different cache names (see https://stackoverflow.com/questions/13189814/how-can-i-make-the-cache-name-in-spring-cache-configurable and https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/cache/interceptor/CacheResolver.html) -- Manda will start this
• configure the redis caches (client side)
• add configuration properties into deployment / portal properties
• allow configuration of either ehcache or redis as the storage for caching
• tuning the helm settings / properties ... and considering the limits on memory footprint / disk use
• we should test on dashi/dashi-dev and see whether the packaged redisson client (v3.13.2) clashes with the jar version of redisson in the tomcat deployment directory (in use for session storage)
• reconnect the cacheStatistics service so that (at least for Ehcache) the statistics are available (maybe Mbeans is enough)
• Update documentation: e.g. ./docs/Caching.md:@Cacheable(cacheNames = "ClinicalDataCache", condition = "@cacheEnabledConfig.getEnabled()")

Re-configure so that REDIS is not built into the JVM process, but is instead run as an external process. The cBioPortal persistence layer annotations can be updated to refer to an external service.

Consult with Hongxin ... oncoKB is using external caching and we can use the approach as a template. (Helm chart and linkage) (OncoKb CacheConfiguration)

One or more help deployments of Redis are running in the kubernetes cluster and being used for various websites to cache persistence layer return values.

Use a separate pool of Redis services for the distinct cohort databases
Start with Genie database .. but plan for having a separate pool of servers for public.

The code base should allow continued use of embedded Ehcache .. but allow reconfiguration for using external redis services.

[sheridancbio]

Consult with Hongxin ... onkoKB is using external caching and we can use the approach as a template. (Helm chart and linkage)

see notes here: https://docs.google.com/document/d/1bu7NVOavH_ekK1O_DKHZAp1HAFwPVlUkp1c0gE5tDYo

[sheridancbio]

A new cluster node was added, in consideration of the expected high memory footprint.

helm and tiller executables were copied from dashi-dev to pipelines and needed environment variables were added to the cbioportal_importer shell startup script to allow use of helm on pipelines. Also, file access mode on relevant startup scripts were tightened.

Adding new node:
kops get ig
kops create ig redis-persistence-cache --subnet=us-east-1a --node-count=1 --node-size=r5.xlarge
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install bitnami/redis --name test-redis-rob --set master.nodeSelector."kops\.k8s\.io/instancegroup"='redis-persistence-cache'

It was noticed that only the master pod was deployed to the specified instance group. Other pods were deployed to general compute nodes on the cluster.

[sheridancbio]

We had some trouble finding the redis server ip address and port to use in the configuration.

You can find these with commands like this:
kubectl get service --all-namespaces | grep

Output looks like:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault test-redis-rob-master ClusterIP XXX.XXX.XXX.XXX 6379/TCP 4d

[sheridancbio]

We were having trouble due to failing to comment out the beans for Ehcache in applicationContext-ehcache.xml
once that was corrected, we were having a failure to wire bean problem for the EhCacheStatistics.

Also we tried to redirect log4j away from the application log (/srv/www/schultz-tomcat/logs...) and stream it out to the console (which we hoped would appear in the catalina.out log (accessible by "kubectl logs")

Helpful commands:
(from cbioportal code base root)
docker build -t "sheridancbio/cbioportal:cbioportal-redis-persistence" -f docker/web/Dockerfile .
docker push sheridancbio/cbioportal:cbioportal-redis-persistence

(from knowlegesystems-k8s-configs?)
kubectl delete -f cbioportal_redis_cache.yaml
kubectl apply -f cbioportal_redis_cache.yaml

kubectl get pods
kubectl exec --stdin --tty cbioportal-redis-cache-5d9cbbb466-h9ftd -- /bin/bash
apt-get update
apt-get install curl
apt-get install redis-tools
curl -X GET "http://localhost:8888/api/cancer-types/GBM" -H "accept: application/json"
redis-cli -u redis://:PASSWORD@100.96.35.3:6379/0
dbsize
keys *

[sheridancbio]

Latest failure report during application startup:
2020-07-08 20:57:29 [localhost-startStop-1] WARN org.springframework.web.context.support.XmlWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ehCacheStatistics' defined in URL [jar:file:/target/tomcat.8888/webapps/expanded/WEB-INF/lib/persistence-api-3.2.5-357-gae63ab3d8c-dirty-SNAPSHOT.jar!/org/cbioportal/persistence/util/EhCacheStatistics.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'javax.cache.CacheManager' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}

2020-07-08 20:57:29 [localhost-startStop-1] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ehCacheStatistics' defined in URL [jar:file:/target/tomcat.8888/webapps/expanded/WEB-INF/lib/persistence-api-3.2.5-357-gae63ab3d8c-dirty-SNAPSHOT.jar!/org/cbioportal/persistence/util/EhCacheStatistics.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'javax.cache.CacheManager' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:228)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)

[mandawilson]

How to actually find stuff in the cache using redis-cli: https://scalegrid.io/blog/redis-iterating-over-keys/

E.g. HGETALL GeneralRepositoryCache

[mandawilson]

Configuration information:

• https://github.com/redisson/redisson/wiki/14.-Integration%20with%20frameworks
• https://www.javadoc.io/doc/org.redisson/redisson/3.2.0/org/redisson/config/Config.html
• https://www.javadoc.io/doc/org.redisson/redisson/3.5.6/org/redisson/spring/cache/CacheConfig.html

Decide on:

ttl - - time to live for key\value entry in milliseconds. If 0 then time to live doesn't affect entry expiration.
maxIdleTime - - max idle time for key\value entry in milliseconds.
if maxIdleTime and ttl params are equal to 0 then entry stores infinitely.

https://redis.io/topics/faq:

• Redis is an in-memory but persistent on disk database, so it represents a different trade off where very high write and read speed is achieved with the limitation of data sets that can't be larger than memory.
• you can configure it to evict keys when the max memory limit is reached in the case where you are using Redis for caching. See "What happens when Redis runs out of memory?"

[sheridancbio]

The initial round of coding has reached a point where we could see persistence layer calls being cached in the external redis server deployment.

Code has now been committed to a feature branch in the cbioportal code base here:
https://github.com/cBioPortal/cbioportal/tree/redis-cache-dev

redis-cache-dev is the branch name and for reference, here is the PR with the code changes:
cBioPortal/cbioportal#7680

[mandawilson]

Status of "each portal needs to save to a different cache names":

I wanted to do something like:

    @Value("#{generalRepositoryCacheName}")
    private String generalRepositoryCacheName;

    @Cacheable(cacheNames = generalRepositoryCacheName, condition = "@cacheEnabledConfig.getEnabled()")
    List<Geneset> getAllGenesets(String projection, Integer pageSize, Integer pageNumber);

But I can't because all of our @Cacheable annotations are in interfaces, and variables in interfaces are static final by default and must be set equal to something when declared. So we cannot use @Value. Also the cacheNames parameter of @Cacheable does not accept SpEL like condition does, so we have to pass it a String or a variable.

Plan for Monday:

• Define two CacheResolver classes to return either the GeneralRepositoryCache or the StaticRespositoryCacheOne. See https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/cache/interceptor/CacheResolver.html. Pass those to @Cacheable. See https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/cache/annotation/Cacheable.html.

It worked!!!

100.96.35.3:6379> keys *
 1) "redisson__timeout__set:{public-portalGeneralRepositoryCache}"
 2) "redisson__execute_task_once_latch:{public-portalGeneralRepositoryCache}"
 3) "public-portalGeneralRepositoryCache"
 4) "redisson__idle__set:{public-portalGeneralRepositoryCache}"
 5) "redisson__timeout__set:{StaticRepositoryCacheOne}"
 6) "GeneralRepositoryCache"
 7) "{GeneralRepositoryCache}:redisson_options"
 8) "{public-portalGeneralRepositoryCache}:redisson_options"
 9) "public-portalStaticRepositoryCacheOneResolver"
10) "testBucket"
11) "{StaticRepositoryCacheOne}:redisson_options"
12) "redisson__idle__set:{GeneralRepositoryCache}"
13) "redisson__timeout__set:{GeneralRepositoryCache}"
14) "testkey"
15) "redisson__idle__set:{StaticRepositoryCacheOne}"
16) "StaticRepositoryCacheOne"

Except "public-portalStaticRepositoryCacheOneResolver" needs to be renamed "public-portalStaticRepositoryCacheOne" (doing that now).

[mandawilson]

To get information about the helm chart deployed:

helm list
NAME                             	REVISION	UPDATED                 	STATUS  	CHART                       	APP VERSION	NAMESPACE   
cbioportal-redis                 	1       	Wed Sep  4 12:28:35 2019	DEPLOYED	redis-6.1.4                 	4.0.13     	default     
...
test-redis-rob                   	1       	Thu Jul  2 16:59:22 2020	DEPLOYED	redis-10.7.9                	6.0.5      	default      

Options are defined here: https://github.com/bitnami/charts/tree/master/bitnami/redis/#installing-the-chart and it looks like major version 10 is the most recent.

Rob deployed a r5.xlarge node, which has 32 GiB (probably 64 bit, Redis says use 32 bit when possible).

• Use PVC to persist data to disk? By default data is stored in /data on node. Default size is 8Gi. What is disk size of r5.xlarge?

Rob ran:

helm install bitnami/redis --name test-redis-rob --set master.nodeSelector."kops\.k8s\.io/instancegroup"='redis-persistence-cache'

To configure the server add:

--set master.extraFlags[0]="--maxmemory 16gb"  --set master.extraFlags[1]="--maxmemory-policy allkeys-lru" --set slave.extraFlags[0]="--maxmemory 16gb"  --set slave.extraFlags[1]="--maxmemory-policy allkeys-lru"

Currently set to use unlimited memory with no eviction policy:

100.96.35.3:6379> config get maxmemory
1) "maxmemory"
2) "0"
100.96.35.3:6379> config get maxmemory-policy
1) "maxmemory-policy"
2) "noeviction"

Current memory usage around 76mb, given by https://redis.io/commands/memory-stats:

MEMORY STATS
...
 3) "total.allocated"
 4) (integer) 76789816

Set Maxmemory to 50mb and see if keys are removed from cache (before and after we add eviction policy):

dbsize
(integer) 13
100.96.35.3:6379> set mytestkey mytestvalue
(error) OOM command not allowed when used memory > 'maxmemory'.
100.96.35.3:6379> config set maxmemory-policy allkeys-lru
OK
100.96.35.3:6379> set mytestkey mytestvalue
OK
100.96.35.3:6379> keys *
 1) "{public-portalGeneralRepositoryCache}:redisson_options"
 2) "testBucket"
 3) "public-portalStaticRepositoryCacheOneResolver"
 4) "{public-portalStaticRepositoryCacheOne}:redisson_options"
 5) "redisson__idle__set:{GeneralRepositoryCache}"
 6) "redisson__idle__set:{StaticRepositoryCacheOne}"
 7) "mytestkey"
 8) "redisson__timeout__set:{StaticRepositoryCacheOne}"
 9) "redisson__timeout__set:{GeneralRepositoryCache}"
10) "StaticRepositoryCacheOne"

• How much memory does the slave (follower) node have?

Lots of details about pod given with kubectl describe pod test-redis-rob-slave-0

Get pods running on a specific node with kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-172-20-53-190.ec2.internal

helm delete test-redis-rob --dry-run
helm delete test-redis-rob

helm install bitnami/redis --name test-redis-manda --set master.nodeSelector."kops\.k8s\.io/instancegroup"='redis-persistence-cache' --set master.extraFlags={"--maxmemory 50mb", "--maxmemory-policy allkeys-lru"} --set slave.extraFlags={"--maxmemory 50mb", "--maxmemory-policy allkeys-lru"}
Error: unknown flag: --maxmemory-policy allkeys-lru}

helm install bitnami/redis --name test-redis-manda --set master.nodeSelector."kops\.k8s\.io/instancegroup"='redis-persistence-cache' --set master.extraFlags[0]="--maxmemory 50mb"  --set master.extraFlags[1]="--maxmemory-policy allkeys-lru" --set slave.extraFlags[0]="--maxmemory 50mb"  --set slave.extraFlags[1]="--maxmemory-policy allkeys-lru"

New Redis Cluster has maxmemory and the Maxmemory-policy set correctly:

# redis-cli -u redis://:XXXXXX@100.96.35.44:6379/0
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
100.96.35.44:6379> keys *
(empty list or set)
100.96.35.44:6379> config get maxmemory-policy
1) "maxmemory-policy"
2) "allkeys-lru"
100.96.35.44:6379> config get maxmemory
1) "maxmemory"
2) "52428800"

• Decide what we really want maxmemory to be.

Looks like peak allocated memory can be over what maxmemory is set to but I haven't seen the
total allocated go over.

100.96.35.44:6379> config get maxmemory
1) "maxmemory"
2) "52428800"

 1) "peak.allocated"
 2) (integer) 115659872
 3) "total.allocated"
 4) (integer) 36021128

I checked and /data is created and it is size 8G (really 7.9G) which is the default size.

• Try allkeys-lfu, and with more memory. Least frequently used not least recently used.

I think not enough memory for the cache caused a new instance of the portal to not be able to start up.

100.96.35.44:6379> config set maxmemory 16gb
OK
100.96.35.44:6379> config get maxmemory
1) "maxmemory"
2) "17179869184"
100.96.35.44:6379> config set maxmemory-policy allkeys-lfu
OK
100.96.35.44:6379> config get maxmemory-policy
1) "maxmemory-policy"
2) "allkeys-lfu"

Seems like setting the ttl and maxIdleTime to 0 made the portal unable to start up, even though it is supposed to be OK with a max-memory policy of allkeys-lfu or allkeys-lru. Not sure why it is a problem.

• Set disk size to RAM x 6? https://docs.redislabs.com/latest/rs/administering/designing-production/hardware-requirements/

[mandawilson]

Tomorrow: Try to get ehcache + Redis working. Ideally one property has: ehcache-heap, ehcache-disk, ehcache-hybrid, redis, none options. Use that to figure out which beans to set up.

[mandawilson]

Really separate ehcache and Redis: cBioPortal/cbioportal#7696

Ideally we would not create the ehcache beans when we are using redis but right now we need to because the EhCacheStatistics is a @Compontent with a constructor that expects a javax.cache.CacheManager which we do not have when running redis.

Maybe we can change the Redis setup to also use javax.cache.CacheManager:
Details: https://www.javadoc.io/doc/org.redisson/redisson/3.2.0/org/redisson/jcache/JCacheManager.html
Example: https://dzone.com/articles/jcache-api-jsr-107-implementation-for-redis

Some of the code in EhCacheStatistics is specific to ehcache but some would work with any javax.cache.CacheManager. Maybe split the code out?

[n1zea144]

@mandawilson @sheridancbio I may have misheard/understood something during the scrum today about having to add the REDIS IP into portal.properties in order for the cbioportal backend to find the REDIS host. Did I hear right? If so, why can't we use DNS?

• Use DNS

[mandawilson]

Commit b61e6b8de5e2f11beed285ab1e97a1114c2c24a3 works without any statistics for Redis (poorly described exception is thrown if API endpoints are called with Redis profile).

Trying to get a javax.cache.Cache (JCache API) for Redis so that we can get two of the cache statistics endpoints working with redis.

We need to somehow get a Redisson CachingProvider/CacheManager in CustomRedisCachingProvider line CacheManager manager = Caching.getCachingProvider().getCacheManager();:

Cannot resolve reference to bean 'redisCacheManager' while setting bean property 'cacheManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisCacheManager' defined in class path resource [applicationContext-rediscache.xml]: Cannot resolve reference to bean 'cacheManager' while setting bean property 'cacheManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheManager' defined in class path resource [applicationContext-rediscache.xml]: Invocation of init method failed; nested exception is javax.cache.CacheException: Multiple CachingProviders have been configured when only a single CachingProvider is expected

JCache API/Redisson: https://github.com/redisson/redisson/wiki/14.-Integration-with-frameworks#144-jcache-api-jsr-107-implementation

redis-ehcache is my branch.

Try: https://www.javadoc.io/static/javax.cache/cache-api/1.0.0/javax/cache/Caching.html#getCachingProviders()

[mandawilson]

I was able to loop through the available CachingProviders and got this:

cbioportal_importer@pipelines:/data/portal-cron/git-repos/knowledgesystems-k8s-deployment/cbioportal> kubectl logs cbioportal-redis-cache-6d9ccbd885-k5s5c| grep ERROR
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - Created Redisson Client: org.redisson.Redisson@204e4aa4
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - CachingProvider: org.ehcache.jsr107.EhcacheCachingProvider@708039b5
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - CacheManager: Eh107CacheManager[urn:X-ehcache:jsr107-default-config]
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - ClassLoader: org.ehcache.core.util.ClassLoading$ChainedClassLoader@2895d662
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - CachingProvider: org.redisson.jcache.JCachingProvider@5b76a0e0
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - CacheManager: org.redisson.jcache.JCacheManager@76125018
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.cbioportal.persistence.util.CustomRedisCachingProvider - ClassLoader: ParallelWebappClassLoader
2020-07-21 15:14:33 [localhost-startStop-1] ERROR org.springframework.web.context.ContextLoader - Context initialization failed

So I added code to check for instanceof JCachingProvider, but since then Rob and I have been unable to get anything to work. Even though the changes were tiny.

Tomorrow maybe focus on commit b61e6b8de5e2f11beed285ab1e97a1114c2c24a3 which should be tested with both Redis and Ehcache. I had it working with redis. Change the cache_type in your deployment yaml file from redis to one of the ehcache ones.

[sheridancbio]

• synchronize the redisson library version

[mandawilson]

• At end make sure unit tests still work

[mandawilson]

• Hongxin asked this, check on it, "But since all methods are using the generalRepositoryCacheResolver with the same name (app name + cache name), how does the redisson differentiate the caches for methods with same type of the signatures? For instance getMetaStudies and getTags in the StudyRepository.java They all accept the type string, when the keyword and the studyId are the same, wouldn’t the cache overlap?"

"So the cache names are different for each portal instance in Redis (as you know) and then in both Redis and EhCache the key stored in the cache is the class name + method name + parameters (e.g. "StudyMyBatisRepository_getAllStudies_null_SUMMARY_10000000_0_null_ASC"). Does that answer your question?"

[mandawilson]

Config Map:
kubectl get configmaps cbioportal-redis-cache -o yaml
cd /data/portal-cron/cbio-portal-data/portal-configuration/k8s-config-vars/public

kubectl delete configmap cbioportal-redis-cache

kubectl create -f config_map_redis_cache.yaml

[mandawilson]

• Rebase and make sure there aren't any unchanged @Cacheable references.

[sheridancbio]

Resolving redission version clashes with libraries installed in the tomcat server is now delegated to another card.

[mandawilson]

Rob and I tried to make the Kubernetes redis services available to dashi-dev (and the world) with ingress, but ingress does not support port 6379 (or anything besides http and https) by default. You can modify ingress to do so, but we didn't feel confident enough to do that. See: https://stackoverflow.com/questions/62939846/exposing-redis-with-ingress-nginx-controller

We see that Ben installed a redis for testing on dashi-dev in 2018 and my plan is to use that for testing. schultz-tomcat on dashi-dev is configured to use a redis running on pipelines, database 3, but that database doesn't have any keys stored in it.

To test on dashi-dev:

• Revert Ingress changes
• Modify code to connect to one server
• Connect to 127.0.0.1 port 9999 on dashi-dev, database 8
• Confirm 127.0.0.1:9999/8 has caches
• Is redis cache still on when no-cache is set?
• Confirm schultz-tomcat is using redis on pipelines
• Are these two redis servers playing nicely together?

[mandawilson]

Redis database usage:

127.0.0.1:9999> CONFIG GET databases
1) "databases"
2) "16"
127.0.0.1:9999> INFO keyspace
# Keyspace

[averyniceday]

Determining Redis/Redisson/Tomcat Version

For Redis: kubectl get service (redis) - look for image

For Tomcat: based on webapp-runner jar we are packaging in
Check inside portal/pom.xml -- using webapp-runner 8.5.51.0

For Redisson: For kubernetes instance look inside webapp-runner pom.xml to determine which Redisson version is used. For dashidev instance just look inside tomcat directory.

[mandawilson]

dashi-dev redisson conflicts:

javax.naming.NamingException: Unexpected exception resolving reference [Root exception is java.lang.IllegalArgumentException: The local resource link [redisson] that refers to global resource [bean/redisson] was expected to return an instance of [org.redisson.api.RedissonClient] but returned an instance of [org.redisson.Redisson]]

[mandawilson]

Both the app and tomcat are using the same version of redisson but we are getting this error:

Error creating bean with name 'cacheManager' defined in class path resource [applicationContext-rediscache.xml]: Invocation of init method failed; nested exception is java.lang.LinkageError: loader constraint violation: when resolving method "org.redisson.jcache.configuration.RedissonConfiguration.fromInstance(Lorg/redisson/api/RedissonClient;Ljavax/cache/configuration/Configuration;)Ljavax/cache/configuration/Configuration;" the class loader (instance of org/apache/catalina/loader/ParallelWebappClassLoader) of the current class, org/cbioportal/persistence/util/CustomRedisCachingProvider, and the class loader (instance of java/net/URLClassLoader) for the method's defining class, org/redisson/jcache/configuration/RedissonConfiguration, have different Class objects for the type javax/cache/configuration/Configuration used in the signature

Look at this: redisson/redisson#1668
Also this might be useful: https://stackoverflow.com/questions/29504180/slf4j-error-class-loader-have-different-class-objects-for-the-type

So we added the redisson jar back to the war file and now are getting the same error we got when the redisson versions did not match:

25-Sep-2020 11:40:42.889 WARNING [localhost-startStop-5] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
java.lang.IllegalArgumentException: The local resource link [redisson] that refers to global resource [bean/redisson] was expected to return an instance of [org.redisson.api.RedissonClient] but returned an instance of [org.redisson.Redisson]

Look at this file because it returns the RedissonClient ./persistence/persistence-api/src/main/java/org/cbioportal/persistence/util/CustomRedisCachingProvider.java

[mandawilson]

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheManager' defined in class path resource [applicationContext-rediscache.xml]: Invocation of init method failed; nested exception is java.util.ServiceConfigurationError: javax.cache.spi.CachingProvider: Provider org.redisson.jcache.JCachingProvider not a subtype

[mandawilson]

Check that Tomcat sessions are being stored by my app in Kubernetes:

kubectl exec --stdin --tty cbioportal-redis-cache-5b9b9bdb4-6gsv5 -- /bin/bash
curl --dump-header headers_and_cookies -X GET "http://localhost:8888/"
File contains: Set-Cookie: JSESSIONID=F6B5E639F2DF5D85A43D7E02C9E5F98E; Path=/; Secure; HttpOnly

cbioportal-redis-master:6379> keys *F6B5E639F2DF5D85A43D7E02C9E5F98E

1. "redisson:tomcat_session:F6B5E639F2DF5D85A43D7E02C9E5F98E"

It is there!

[mandawilson]

I renamed the JNDI resource in Tomcat's context.xml and server.xml but still have the same error:

The local resource link [redisson-tomcat] that refers to global resource [session/redisson-tomcat] was expected to return an instance of [org.redisson.api.RedissonClient] but returned an instance of [org.redisson.Redisson]

[mandawilson]

When I exclude the jar from the war, I get this again:

Invocation of init method failed; nested exception is java.util.ServiceConfigurationError: javax.cache.spi.CachingProvider: Provider org.redisson.jcache.JCachingProvider not a subtype

[mandawilson]

Passing this jcacheConfig, instead of just using the RedissonConfiguration.fromInstance(redissonClient):

    MutableConfiguration<String, String> jcacheConfig = new MutableConfiguration<>();
    Configuration<String, String> config = RedissonConfiguration.fromInstance(redissonClient, jcacheConfig);

Causes this:

text-rediscache.xml]: Invocation of init method failed; nested exception is java.lang.LinkageError: loader constraint violation: when resolving method "org.redisson.jcache.configuration.RedissonConfiguration.fromInstance(Lorg/redisson/api/RedissonClient;Ljavax/cache/configuration/Configuration;)Ljavax/cache/configuration/Configuration;" the class loader (instance of org/apache/catalina/loader/ParallelWebappClassLoader) of the current class, org/cbioportal/persistence/util/CustomRedisCachingProvider, and the class loader (instance of java/net/URLClassLoader) for the method's defining class, org/redisson/jcache/configuration/RedissonConfiguration, have different Class objects for the type javax/cache/configuration/Configuration used in the signature

But leaving it and removing the jar from the war give you this again:
org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
java.lang.IllegalArgumentException: The local resource link [redisson-tomcat] that refers to global resource [session/redisson-tomcat] was expected to return an instance of [org.redisson.api.RedissonClient] but returned an instance of [org.redisson.Redisson]

[mandawilson]

In this thread it says All works properly if you won't include redisson.jar into war-archive and move jar containing entity classes into apache-tomcat\lib folder.: redisson/redisson#1668

Perhaps it works in Kubernetes because there is only the web runner jar running our app, so one shared "space" or thread or whatever the class loader needs.

Do we have to take our CustomRedisCachingProvider.java class and put it in a jar, and exclude it from the war and add it to Tomcat?

In portal.pom:
<packagingExcludes>**/CustomRedisCachingProvider.class</packagingExcludes>

Also exclude redisson*.jar.
Make a jar containing the class https://docs.oracle.com/javase/tutorial/security/toolsign/step2.html:

jar cvf Count.jar Count.class

[averyniceday]

added loader option to context.xml
this is the new error across all portal web apps

include a too low setting for -Xss and illegal cyclic inheritance dependencies. The class hierarchy being processed was [org.bouncycastle.asn1.ASN1Boolean->org.bouncycastle.asn1.DERBoolean->org.bouncycastle.asn1.ASN1Boolean]

[sheridancbio]

After contemplating additional efforts towards resolving the problem loading appropriate classes (choosing the container "/lib" jar classes for redisson or choosing the warfile package dependencies) during tomcat deployment, product owner has accepted that the unresolved conflict during tomcat deployments will be allowed to stand. Our local tomcat deployments will not use redis caching for the persistence layer - and will instead continue with the ehcache persistence caching.

A note will be added to the build/deploy documentation that installers should expect difficulties when deploying .war files to a tomcat which has been configured to use redis caching for user sessions.

[averyniceday]

When switching to profiles - remember you'll have to configure this for unit tests!! or use @MockBean