Fuseki 6.0.0: canceled federated SERVICE queries can wedge the target dataset until restart

[plvsadi]

Version

6.0.0 Also reproduced on 5.5.0.

What happened?

Version

6.0.0

Also reproduced on 5.5.0.

What happened?

We can reproduce a failure mode where repeated canceled federated SERVICE queries leave
the target dataset effectively wedged until Fuseki is restarted.

The pattern is:

1. A direct query to dataset target succeeds.
2. A federated query from dataset source to dataset target using SERVICE <http://127.0.0.1:3030/target/sparql> succeeds.
3. We then issue a burst of heavy federated queries from source to target, with the
   client canceling/timing out the outer HTTP request almost immediately.
4. After that, even a simple direct query to dataset target times out.
5. Restarting Fuseki clears the problem.

This is reproducible for us on Jena/Fuseki 6.0.0 and also on 5.5.0.

Why this looks distinct from ordinary timeout behavior

This does not look like only the outer client timing out.

After the cancellation storm:

• a direct query to the target dataset also times out
• the store recovers only after restart

So the target dataset/server appears to be left in a bad runtime state.

Reproducing it

We reproduced this against an isolated standalone Fuseki 6.0.0 container built from the
official Apache release tarball.

Our production datasets are private, but the failure can be described with this structure:

• source dataset: source
• target dataset: target

Baseline direct probe against target:

SELECT * WHERE {
  <urn:probe-subject> ?p ?o
}
LIMIT 5

Baseline federated probe from source to target:

  SELECT * WHERE {
    SERVICE <http://127.0.0.1:3030/target/sparql> {
      <urn:probe-subject> ?p ?o
    }
  }
  LIMIT 5

Cancellation-storm query:

  SELECT * WHERE {
    SERVICE <http://127.0.0.1:3030/target/sparql> {
      ?s ?p ?o
    }
  }

We then repeatedly send that last query to the source dataset and cancel the outer HTTP
request almost immediately, for example:

  for i in $(seq 1 40); do
    curl -sS --max-time 0.05 -G \
      --data-urlencode 'query=SELECT * WHERE { SERVICE <http://127.0.0.1:3030/target/sparql> { ?s ?p ?o } }' \
      http://127.0.0.1:3030/source/sparql >/dev/null || true
  done 

Actual result

Before stress:

• direct query succeeds
• federated query succeeds

After the canceled federated-query burst:

• direct query to target times out
• federated query also fails/times out
• Fuseki restart is required to recover

Expected result

Canceled outer federated queries should not leave the target dataset/server wedged.
After the canceled requests, normal direct queries to the target dataset should still work.

Relevant logs

From the Jena 6.0.0 Fuseki log, after the stress starts we see many inner requests like:

  GET http://127.0.0.1:3030/target/sparql?query=SELECT++%2A%0AWHERE%0A++%7B+?s++?p++?o+%7D%0A

The outer requests are being canceled by the client, but the inner SERVICE subqueries
continue to run. After enough of these, the target dataset stops responding to even direct
queries.

Notes

• This was reproduced using loopback 127.0.0.1, so it does not appear to require Docker DNS/
  container-name routing.
• We specifically tested 6.0.0 because the changelog mentions query-cancellation
  improvements, but we still reproduce this failure.

[afs]

Hi @plvsadi

How many triples are in the dataset?
What does the log say? (in particular, it will show queries that have returned so it is then possible to estimate the number of outstanding requests).

Is the database TDB2?

How long did you wait after the storm to see if the server recovered befor restarting?

Does it happen using a repeated requests from the same process? (that curl example is new process each time and using a new connection each time; this can lead to networking exhaustion.)

Could you provide a standalone setup that demonstrates the issue (i.e. with junk data)? There are various details that making trying to reproduce the situation exactly quite difficult.

[plvsadi]

Hi @afs Yes the dataset is TDB2. I have managed to do it with a base test levels of triples. Few 100s i think. As for the storm, I had somethign runnign last night that crashed adn then this morning i started to face this issue. Upon investigation came across the bug. So The server at least had 4-5 hours after the initial error to recover. YOu should be able to use this script

fuseki_federation_cancel_wedge_repro.sh

This repro script is standalone and public-safe:

How to use it:

• run it against a Fuseki instance with admin API enabled
• it creates two temporary TDB2 datasets, loads synthetic triples, runs the stress case, and
  then probes the target dataset again
• it deletes the temp datasets on exit unless --keep-datasets is set

Typical same-host usage:

./fuseki_federation_cancel_wedge_repro.sh \
  --base-url http://127.0.0.1:3030 \
  --service-base-url http://127.0.0.1:3030 \
  --admin-user admin \
  --admin-password admin

If Fuseki is in Docker and exposed on a host port:

- --base-url should be the host-reachable URL
- --service-base-url should be the URL that Fuseki itself will use inside the SERVICE clause
- for same-store self-federation that is often still http://127.0.0.1:3030

Expected failure signature:

1. direct probe before stress succeeds
2. federated probe before stress succeeds
3. canceled federated-query stress runs
4. direct probe after stress times out or fails
5. federated probe after stress also fails or times out
6. Fuseki restart is needed to recover

[Aklakan]

The outer requests are being canceled by the client, but the inner SERVICE subqueries
continue to run. After enough of these, the target dataset stops responding to even direct
queries.

If I understand the problem correctly, then the issue is, that cancellation is not forwarded to a SERVICE execution, and the requests just keep piling up.
Service execution materializes the result sets before returning - cancellation only kicks in after materialization completes. Especially when doing ?s ?p ?o, executions will likely pile up.
However, setting arq:httpQueryTimeout in the Fuseki endpoint context should have an effect for HTTP SERVICE requests (based on the code).

Should work also work with the CLI tooling: (not tested)

# Whichever tool you are using
./rsparql --set 'arq:httpQueryTimeout=1000' ...
./tdb2-query --set `arq:httpQueryTimeout=1000` --set 'arq:queryTimeout=2000'
# Note: queryTimeout should affect the outer query, httpQueryTimeout the inner.

Relevant code in Service.java

    public static QueryIterator exec(OpService op, Context context) {
/...
        long timeoutMillis = timeoutFromContext(context);

        // RegistryServiceModifier is applied by QueryExecHTTP
        Params serviceParams = getServiceParamsFromContext(serviceURL, context);

        // Build the execution
        try (QueryExecHTTP qExec = QueryExecHTTP.newBuilder()
                .endpoint(serviceURL)
                .timeout(timeoutMillis, TimeUnit.MILLISECONDS)
                .httpHeader(HttpNames.hUserAgent, HttpEnv.UserAgent)
                .query(query)
                .params(serviceParams)
                .context(context)
                .httpClient(httpClient)
                .sendMode(querySendMode)
                .build()) {

            // Detach from the network stream.
            RowSet rowSet = qExec.select().materialize();
            QueryIterator qIter = QueryIterPlainWrapper.create(rowSet);
            if (requiresRemapping)
                qIter = QueryIter.map(qIter, varMapping);
            return qIter;
        } catch (HttpException ex) {
            throw QueryExceptionHTTP.rewrap(ex);
        }
// ...
}
// ---------------
    /*package*/ static long timeoutFromContext(Context context) {
        return parseTimeout(context.get(httpQueryTimeout));
    }

Simple solution: Get cancel signal from context, replace qExec.select().materialize(); with a materialization loop that checks the cancel signal.

Complex solution: Move QueryExec into a "deferred iterator" which has the ongoing QueryExec as a field, such that requestCancel() can call qExec.abort() - might be better with nested federations, where a SERVICE inside a SERVICE hangs on materialization.

[afs]

If I understand the problem correctly, then the issue is, that cancellation is not forwarded to a SERVICE execution, and the requests just keep piling up.

Yes, and there are several consequences of that!

Curl is creating a connection every time (even for HTTP/2). Even if it closes connections, the server does not see that if it is itself waiting for the SERVICE.

When running on the same host, HTTP/1.1, it can then run out of ports. Ports aren't immediately released when a connection is locally closed - the kernel has to clear up async later. We've seen connection exhaustion before; when the server is busy, the kernel does not clear up TCP connections fast enough. If open connections are still around, ports may be be released and there aren't that many ports.

Coupled with interaction with services materialization, as above, ther could be a compounding effect.

If the server is negotiating up to HTTP/2, the connection problem becomes head-of-line blocking and again the system can lock up.

And Java does not have HTTP/3 (it's first in Java 26).

Service execution materializes the result sets before returning

We could stream but that has it's own problems (broken connections).

Whatever the lower level improvements are, I think we need a defensive request control in Fuseki to give the user some controls and have back pressure on high-frequency clients. e.g. Max N concurrent requests, hold up requests above N, and if above a higher limit M, start rejecting with HTTP 503. Just queuing does not work because of resource exhaustion.

[afs]

fuseki_federation_cancel_wedge_repro.sh

Thank you!

[plvsadi]

Thanks guys. Glad I could be of some help!