Soft-index file store worse performance than Single File store

[milos-pajic-doodle]

Hey all,

We are upgrading our authentication system from Keycloak-15 (Wildfly) where we used Infinispan-11 as a remote cache store, to Keycloak-18 (Quarkus) and Infinispan-13 as a remote cache store. With the IS-11 we used the Single File store (as it was the default file store at that time) and when we switched to the IS-13 default store (which is the Soft-Index file store) we experienced massive performance degradation which was caused by abnormal CPU usage as the store gets filled up with sessions.
In theory, Soft-Index file stores should be even more performant than Single File stores, so are we doing anything wrong?

P.S when we switched back to the Single File store in IS-13 everything was smooth again.

Here is our configuration of the IS server:

    infinispan:
      cacheContainer:
        name: default
        security:
          authorization: {}
        jmx:
          enabled: "true"
        statistics: "true"
        metrics:
          gauges: "true"
          histograms: "true"
          accurate-size: "false"
        transport:
          stack: kubernetes
        global-state:
          persistent-location:
            path: "/opt/infinispan/custom-data"
        caches:
          replicated-cache-template:
            replicatedCacheConfiguration:
              mode: "SYNC"
              statistics: "true"
              locking:
                concurrency-level: "1000"
                acquire-timeout: "15000"
              persistence:
                file-store:
                  preload: "true"
              state-transfer:
                enabled: "true"
                timeout: 900000
                await-initial-transfer: "false"
          work:
            replicatedCache:
              configuration: replicated-cache-template
          sessions:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "2000000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
          clientSessions:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "2000000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
          offlineSessions:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "1000000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
          offlineClientSessions:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "1000000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
          actionTokens:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "100000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
          loginFailures:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "10000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
          authenticationSessions:
            replicatedCache:
              configuration: "replicated-cache-template"
              memory:
                max-count: "100000"
                when-full: "REMOVE"
              expiration:
                lifespan: "1213200000"
                interval: "900000"
      server:
        interfaces:
          - name: "public"
            inetAddress:
              value: "${env.POD_IP}"
        endpoints:
          endpoint:
            socketBinding: "default"
            securityRealm: "default"
            hotrodConnector:
              authentication:
                sasl:
                  serverName: "infinispan"
                  mechanisms:
                    - "PLAIN"
                    - "DIGEST-MD5"
                    - "GSSAPI"
                    - "EXTERNAL"
                  qop:
                    - "auth"
            restConnector:
              authentication:
                mechanisms:
                  - "DIGEST"
                  - "BASIC"
                securityRealm: "default"
        security:
          credentialStores:
            - clearTextCredential:
                clearText: secret
              name: credentials
              path: credentials.pfx
          securityRealms:
            - name: default
              propertiesRealm:
                groupProperties:
                  path: groups.properties
                groupsAttribute: Roles
                userProperties:
                  path: users.properties
            - name: metrics
              propertiesRealm:
                groupProperties:
                  path: metrics-groups.properties
                  relativeTo: infinispan.server.config.path
                groupsAttribute: Roles
                userProperties:
                  path: metrics-users.properties
                  relativeTo: infinispan.server.config.path
        socketBindings:
          defaultInterface: public
          portOffset: 0
          socketBinding:
            - name: default
              port: 11222
            - name: metrics
              port: 11223

Configuraiton on the KC side:

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:13.0 http://www.infinispan.org/schemas/infinispan-config-13.0.xsd"
        xmlns="urn:infinispan:config:13.0">

    <cache-container name="keycloak" statistics="true">
        <local-cache-configuration name="local-cache-template" statistics="true">
            <expiration lifespan="30000" max-idle="60000"/>
            <encoding media-type="application/x-java-object"/>
            <memory max-count="10000"/>
            <locking concurrency-level="1000"/>
        </local-cache-configuration>

        <distributed-cache-configuration name="distributed-cache-template" owners="${env.CACHE_OWNERS:2}" 
                                         statistics="true">
            <memory max-count="2000000"/>
            <expiration lifespan="${env.CACHE_EXPIRATION:1213200000}"/>
        </distributed-cache-configuration>
        
        <transport lock-timeout="60000"/>
        <local-cache name="realms" configuration="local-cache-template"/>
        <local-cache name="users" configuration="local-cache-template"/>
        <local-cache name="authorization" configuration="local-cache-template"/>
        <local-cache name="keys" configuration="local-cache-template">
            <expiration max-idle="3600000"/>
        </local-cache>

        <distributed-cache name="sessions" configuration="distributed-cache-template">
            <remote-store cache="sessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </distributed-cache>

        <distributed-cache name="authenticationSessions" configuration="distributed-cache-template">
            <memory max-count="1000000"/>
            <remote-store cache="offlineSessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
            </remote-store>
        </distributed-cache>

        <distributed-cache name="offlineSessions" configuration="distributed-cache-template">
            <memory max-count="1000000"/>
            <remote-store cache="offlineSessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </distributed-cache>

        <distributed-cache name="clientSessions" configuration="distributed-cache-template">
            <remote-store cache="clientSessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </distributed-cache>

        <distributed-cache name="offlineClientSessions" configuration="distributed-cache-template">
            <memory max-count="1000000"/>
            <remote-store cache="offlineClientSessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </distributed-cache>

        <distributed-cache name="loginFailures" configuration="distributed-cache-template">
            <remote-store cache="loginFailures" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </distributed-cache>

        <replicated-cache name="work" statistics="true">
            <expiration lifespan="${env.CACHE_EXPIRATION:1213200000}"/>
            <remote-store cache="work" xmlns="urn:infinispan:config:store:remote:13.0"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory"
                          fetch-state="false"
                          purge="false"
                          preload="false"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </replicated-cache>

        <distributed-cache name="actionTokens" configuration="distributed-cache-template">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="${env.CACHE_EXPIRATION:1213200000}" interval="300000"/>
            <memory max-count="-1"/>
            <remote-store cache="actionTokens" xmlns="urn:infinispan:config:store:remote:13.0"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${env.INFINISPAN_PORT:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.INFINISPAN_USERNAME:keycloak}"
                                password="${env.INFINISPAN_PASSWORD:password}"
                                realm="default"/>
                    </authentication>
                </security>
              </remote-store>
        </distributed-cache>
    </cache-container>
</infinispan>

We would like to continue using whatever Infinispan is defaulting to but as per the results of the performance tests per shown configuration, it's just not viable. Any help is appreciated.
Regards

[milos-pajic-doodle]

Maybe more additional information:

• Our remote IS server is replicated cache cluster consisting of 3 pods in k8s.
• When reducing a number of segments from 256 -> 60 (as per documentation we should not go lower than 20*cluster_size) we did see some improvements in terms of that CPU usage was spiking at ~2.5 mil caches in entry, whereas with 256 segments(default) it spiked around 800k.

[wburns]

Not sure what you mean by "spiking at ~2.5 mil caches in entry" vs "spiked around 800k". Is this the number of entries in the cache? Are these being removed via expiration or user interaction?

[milos-pajic-doodle]

Yes, the CPU spikes are correlated with number of entries in the cache. We do have expiration enabled as you can see in the configuration (1-week idle, 2-week max), but we have a use case where we expect 20 mil. of cache entries to be present at the time.

[wburns]

Are you able to collect thread dumps when this occurs? Previous testing SIFS should be faster than SFS, especially when segmented stores are enabled. Can you try maybe with disabling segmentation on the store config if that changes performance or not?

[milos-pajic-doodle]

When we try to disable segmentation on the file store we get this exception:
Invalid cache loader configuration for 'NonBlockingSoftIndexFileStore'. This implementation only supports being segmented!"
Not sure if there is an implementation of SIFS where we can disable segmentations?

[milos-pajic-doodle]

We did some profiling with the granulate profiler 👇🏻
https://drive.google.com/file/d/1mq0LsvadRphZCbDKarC-3PCUCYn3XjjL/view?usp=sharing