Fuseki sessions are immortal and fill up heap

[SamuelBoerlin]

Version

6.1.0

What happened?

Hello

We observed Fuseki's heap memory usage climbing slowly, though consistently, while doing a long chain of sequential SPARQL UPDATE requests over Fuseki's HTTP API, and it looked like a memory leak and did eventually cause us to run into OutOfMemoryError repeatedly.

This prompted us to investigate and we found that the root cause to be that sessions seem to be immortal.
The reason this is especially problematic for us is because we use Shiro's authcBasic module and our backend authenticates on each request. This then causes unlimited sessions to pile up quite quickly in Fuseki.

I must preface that most of the analysis was done mostly by Claude Code, but it does look correct to me and the session "memory leak" was experimentally confirmed to be fixed by disabling session storage altogether in the shiro config with securityManager.subjectDAO.sessionStorageEvaluator.sessionStorageEnabled = false. Since then the heap usage remains low and constant, we're no longer running into OutOfMemoryError, and the heap now fits very comfortably in the same Xmx limit as before.

The analysis mainly consisted of running our workload for a while and then creating a heap dump with jmap. A significant portion of the heap, ~1.6GB, was attributed to ~1.23 M ManagedSession objects and the corresponding container ConcurrentHashMap. It was then concluded that sessions are immortal with the reason being that session lifetime is apparently never configured on the SessionHandler in FMod_Shiro. SessionHandler defaults to immortal sessions as inherited from AbstractSessionManager.

Is it correct that sessions are really immortal? If so, is that intentional?
Should this perhaps be configurable?

For us simply disabling session storage with securityManager.subjectDAO.sessionStorageEvaluator.sessionStorageEnabled = false might work because our backend authenticates on each request anyways, and this feature is documented at https://shiro.apache.org/session-management.html.
But is this really fine or could we run into problems by doing this? Just disabling all session storage seems like a heavy hammer, and I'm not yet sure how this will interact with Fuseki's web UI.

Best regards
Samuel

PS: perhaps this was also the root cause of #3432 because as far as I can tell FMod_Shiro was introduced between 5.2.0 and 5.5.0.

Relevant output and stacktrace

Class Name                                                           |    Objects |  Shallow Heap |    Retained Heap
---------------------------------------------------------------------------------------------------------------------
                                                                     |            |               |                 
byte[]                                                               |  4’384’892 |   289’031’584 |   >= 289’031’584
java.util.concurrent.ConcurrentHashMap$Node                          |  5’725’473 |   274’822’704 | >= 1’622’296’112
java.util.concurrent.ConcurrentHashMap$Node[]                        |  1’227’488 |   216’579’552 | >= 1’652’637’328
org.eclipse.jetty.session.SessionData                                |  1’227’124 |   147’254’880 |   >= 726’457’664
java.lang.String                                                     |  4’344’885 |   139’036’320 |   >= 413’159’864
java.util.concurrent.ConcurrentHashMap                               |  1’228’438 |   127’757’552 | >= 1’652’769’320
org.eclipse.jetty.session.ManagedSession                             |  1’227’124 |   127’620’896 | >= 1’403’459’344
java.util.concurrent.locks.ReentrantLock$NonfairSync                 |  1’227’422 |    58’916’256 |    >= 58’916’992
org.eclipse.jetty.session.SessionInactivityTimer                     |  1’227’124 |    58’901’952 |   >= 137’438’336
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject|  1’227’254 |    49’090’160 |    >= 49’090’176
org.eclipse.jetty.session.SessionInactivityTimer$1                   |  1’227’124 |    49’084’960 |    >= 78’535’984
com.github.benmanes.caffeine.cache.PSMS                              |    781’639 |    43’771’784 |   >= 116’675’280
java.util.concurrent.CopyOnWriteArrayList                            |  1’227’332 |    39’274’624 |    >= 88’377’424
java.lang.Object[]                                                   |  1’248’661 |    31’863’416 |    >= 43’891’880
java.util.concurrent.atomic.AtomicReference                          |  1’239’536 |    29’748’864 |    >= 30’172’384
java.util.concurrent.locks.ReentrantLock                             |  1’227’422 |    29’458’128 |    >= 29’466’504
org.eclipse.jetty.util.thread.AutoLock                               |  1’227’232 |    29’453’568 |    >= 58’912’232
org.eclipse.jetty.ee11.servlet.SessionHandler$ServletSessionApi      |  1’227’124 |    29’450’976 |    >= 29’450’976
org.apache.jena.graph.impl.LiteralLabel                              |    346’091 |    24’918’552 |    >= 66’942’712
org.apache.jena.tdb2.store.NodeId                                    |    582’074 |    23’282’960 |    >= 23’283’688
java.lang.Object                                                     |  1’246’253 |    19’940’048 |    >= 19’940’104
long[]                                                               |        184 |    10’533’144 |    >= 10’533’144
org.apache.jena.graph.Node_Literal                                   |    346’091 |     8’306’184 |    >= 75’245’616
org.apache.jena.graph.Node_URI                                       |    236’551 |     5’677’224 |    >= 35’411’984
java.util.LinkedHashMap$Entry                                        |     37’615 |     2’407’360 |     >= 2’673’384
jdk.nio.zipfs.ZipFileSystem$IndexNode                                |     33’213 |     1’859’928 |     >= 4’646’736
char[]                                                               |      7’117 |     1’273’952 |     >= 1’273’952
java.util.HashMap$Node[]                                             |      2’420 |     1’162’352 |     >= 4’553’760
java.util.HashMap$Node                                               |     23’282 |     1’117’536 |     >= 3’469’776
int[]                                                                |      5’964 |     1’076’056 |     >= 1’076’056
java.nio.DirectByteBuffer                                            |      7’280 |       698’880 |       >= 698’952
java.util.ArrayList                                                  |     13’982 |       559’280 |    >= 11’684’440
Total: 32 of 16’742 entries; 16’710 more                             | 35’587’914 | 1’885’973’400 |                 
---------------------------------------------------------------------------------------------------------------------

Are you interested in making a pull request?

None

[afs]

Is it correct that sessions are really immortal? If so, is that intentional?
Should this perhaps be configurable?

The analysis looks plausible. The only time I can think of where sessions might be significant is for login to the server for admin operations. Fuseki operations themselves aren't making use of session state.

We ought to set the admin UI to timeout, and set a safety default idle time regardless of configuration.

My reading of the Shiro session documentation is that there is a default of 30 mins. That maybe too much for you and everyone else - we are talking machines contacting Fuseki, not people. Have you tried setting int shiro.ini?

I'm not quite clear why each update causes a new session. Are the clients aren't sending the known auth information and doing a challenge/response on each operation because the effective user may have changed?

[SamuelBoerlin]

The only time I can think of where sessions might be significant is for login to the server for admin operations. Fuseki operations themselves aren't making use of session state.

Thanks, that's good to know.

My reading of the Shiro session documentation is that there is a default of 30 mins. That maybe too much for you and everyone else - we are talking machines contacting Fuseki, not people. Have you tried setting int shiro.ini?

Shiro supposedly uses ServletContainerSessionManager when running in Jetty and does not manage the sessions itself, so it's up to SessionHandler to manage the sessions, including expiration, i.e., Shiro's own expiration does not apply.

I'm not quite clear why each update causes a new session. Are the clients aren't sending the known auth information and doing a challenge/response on each operation because the effective user may have changed?

Our backend sends the basic auth header on each request, which apparently results in a new session being created each time.
According to Claude it should be possible to reuse sessions by using a JSESSIONID cookie, but our backend doesn't do that currently.
Maybe the proper fix is for us to handle this cookie, but there should probably be a reasonable default expiration time either way.

[afs]

Our backend sends the basic auth header on each request

I've recreated that and can see the same thing happening.

I have found two ways to remove the need for sessions - both mean that there does not need to a Jetty/jakarta session management at all. In both cases, authentication is happening in the same way as before. Shiro sessions don't come into play until after authetnication has happened.

My testing setup is Fuseki "plain", the core Fuseki engine, with FMod_Shiro, ping and prometheus; no UI. The UI checks server liveness every few seconds which makes breakpoints in shiro code "interesting".

(There are internally four packagings of Fuseki - only "full/UI" is properly exposed. "plain" is running Fuseki as a headless triple store. Longer term work-in-progress in the code - see FusekiRunner , entry point FusekiServerPlainCmd).

This works and should work in 6.1.0:

[main]
securityManager.subjectDAO.sessionStorageEvaluator.sessionStorageEnabled = false

(in 6.1.0 a Jetty SessionManager is unnecessarily created once in setup but never used by Shiro or Fuseki)

The downside isee in this approach is that it goes in the shiro.ini file.

To many users, it looks like gobbledygook.

Fuseki does not use much of shiro-web - servlet filter interception, authentication and URL pattern filtering. Using Shiro gives the potential to fit in schemes other than hardwired basic and digest.

Shiro session creation can be disabled per-request in the Fuseki subclass of ShiroFilter:

        protected ServletRequest prepareServletRequest(ServletRequest request, ServletResponse response, FilterChain chain) {
            request.setAttribute(DefaultSubjectContext.SESSION_CREATION_ENABLED, Boolean.FALSE);
            return super.prepareServletRequest(request, response, chain);
        }

That line was taken from Shiro's NoSessionCreationFilter. I haven't managed to get that to work by using NoSessionCreationFilter - it needs to be inside Shiro processing because prepareServletRequest is Shiro extension point operation.

I haven't managed to find a way to configure the Shiro environment during server initialization to do that or apply the subjectDAO approach in java code.

 

Fuseki is using Shiro-2. There is a Shiro-3 in the pipeline.
(PS It does not look like a major functionality change from the POV of Fuseki's usage)

[afs]

PR #4038 removes sessions. Jetty runs without an HTTP session manager and Shiro does not attempt to create a Shiro session.