Wrong evaluation of protected SERVICE expressions with subdomain in URL

[redskate]

Using eclipse-rdf4j-3.7.1-onejar.jar

and the code:

TupleQuery tupleQuery = con.prepareTupleQuery(QueryLanguage.SPARQL,query);
TupleQueryResult result = tupleQuery.evaluate();

with as query

select ?s ?p ?o 
{ 
 SERVICE <https://demo2:demo2@semwebs.com/S2Mapi/thesaurus/ASSEMBLY/sparql/>
 {  select ?s ?p ?o  {?s ?p ?o} limit 1 }
} 

Everything is working fine and the code is delivering the first selected triple.

PROBLEMS arise when another URI "it.semweb.ch" instead of the uri "semwebs.com" is used in the query:

select ?s ?p ?o 
{ 
 SERVICE <https://demo2:demo2@it.semweb.ch/S2Mapi/thesaurus/ASSEMBLY/sparql/>
 {  select ?s ?p ?o  {?s ?p ?o} limit 1 }
} 

Both services (under it.semweb.ch and semwebs.com) deliver correct SPARQL results in case the query be executed e.g. inside some other SPARQL engine like GraphDB.

Unfortunately the second expression delivers - when executed by the above java code belonging to rdf4j - an exception because the java evaluate() code does not properly evaluate the SERVICE expression having this URI (it.semweb.ch with one subdomain AND login:password)

I suppose some regex inside fails to parse properly the whole URI https://demo2:demo2@it.semweb.ch/S2Mapi/thesaurus/ASSEMBLY/sparql/ and the evaluation fails - since null results are delivered.

Please would you fix this?

The URI inside the SERVICE expression needs to include user:password credentials - since the data are protected.

Here some lines of the thrown exception:

org.eclipse.rdf4j.query.QueryEvaluationException: org.eclipse.rdf4j.query.QueryEvaluationException: Server responded with an unsupported file format: null
	at org.eclipse.rdf4j.repository.sail.SailTupleQuery.evaluate(SailTupleQuery.java:58)
	at com.semweb.sesame.SESAME_inmemory_repository.fetch_tuples(SESAME_inmemory_repository.java:155)
        ...

Caused by: org.eclipse.rdf4j.sail.SailException: org.eclipse.rdf4j.query.QueryEvaluationException: Server responded with an unsupported file format: null
	at org.eclipse.rdf4j.sail.base.SailSourceConnection.evaluateInternal(SailSourceConnection.java:253)
	at org.eclipse.rdf4j.sail.helpers.AbstractSailConnection.evaluate(AbstractSailConnection.java:249)
	at org.eclipse.rdf4j.repository.sail.SailTupleQuery.evaluate(SailTupleQuery.java:50)
	... 31 more
Caused by: org.eclipse.rdf4j.query.QueryEvaluationException: Server responded with an unsupported file format: null
	at org.eclipse.rdf4j.repository.sparql.query.SPARQLTupleQuery.evaluate(SPARQLTupleQuery.java:46)
	at org.eclipse.rdf4j.repository.sparql.federation.RepositoryFederatedService.select(RepositoryFederatedService.java:227)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:438)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:704)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:286)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:784)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:698)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:286)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:721)
	at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:286)
	at org.eclipse.rdf4j.sail.base.SailSourceConnection.evaluateInternal(SailSourceConnection.java:248)
	... 33 more
Caused by: org.eclipse.rdf4j.repository.RepositoryException: Server responded with an unsupported file format: null
	at org.eclipse.rdf4j.http.client.SPARQLProtocolSession.lambda$getBackgroundTupleQueryResult$0(SPARQLProtocolSession.java:656)
	at java.base/java.util.Optional.orElseThrow(Optional.java:401)
	at org.eclipse.rdf4j.http.client.SPARQLProtocolSession.getBackgroundTupleQueryResult(SPARQLProtocolSession.java:656)
	at org.eclipse.rdf4j.http.client.SPARQLProtocolSession.sendTupleQuery(SPARQLProtocolSession.java:363)
	at org.eclipse.rdf4j.repository.sparql.query.SPARQLTupleQuery.evaluate(SPARQLTupleQuery.java:43)
	... 43 more

[abrokenjester]

This is not a bug in RDF4J, but a problem in the SPARQL endpoint that you're querying. And it does not appear to be related to the presence or absence of username/password in the url, either.

If we do a direct curl request with the same url and a simple sparql query, we see the following:

❯ curl -v https://demo2:demo2@it.semweb.ch/S2Mapi/thesaurus/ASSEMBLY/sparql/\?query\=select%20%2A%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D%20limit%2010
*   Trying 31.27.148.198:443...
* TCP_NODELAY set
* Connected to it.semweb.ch (31.27.148.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=it.semweb.ch
*  start date: Feb 17 00:00:00 2021 GMT
*  expire date: Mar 20 23:59:59 2022 GMT
*  subjectAltName: host "it.semweb.ch" matched cert's "it.semweb.ch"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.
* Server auth using Basic with user 'demo2'
> GET /S2Mapi/thesaurus/ASSEMBLY/sparql/?query=select%20%2A%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D%20limit%2010 HTTP/1.1
> Host: it.semweb.ch
> Authorization: Basic ZGVtbzI6ZGVtbzI=
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Date: Sat, 17 Jul 2021 04:09:52 GMT
< Server: sks
< Strict-Transport-Security: max-age=0
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=675D738F6A90AD887EE8624538C1F732; Path=/S2Mapi; Secure; HttpOnly
< Transfer-Encoding: chunked
< 
{
  "errors_notification": {
    "error": [
      "No SKOS Shuttle API service found for s=thesaurus&p=ASSEMBLY&x=sparql&"
    ]
  }
* Connection #0 to host it.semweb.ch left intact
}

Note that the HTTP status response is 200 (meaning "Ok"), but that the body of the response indicates an error. Also note that the response does not contain a Content-Type header. This is why RDF4J gets a little confused processing the response: the status code says all is well, but then there is no content-type header so we can't parse the response (and even if we could, the response is not actually anything that we can do anything with).

I have not been able to determine why the SPARQL endpoint responds with this error. The request as RDF4J (and indeed curl) sends it is syntactically correct. However, I did note something curious, which is that if I change the request URL by one character, we suddenly do get a correct response:

❯ curl -v https://demo2:demo2@it.semweb.ch/S2Mapi/thesaurus/ASSEMBLY/sparql/\&query\=select%20%2A%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D%20limit%2010
*   Trying 31.27.148.198:443...
* TCP_NODELAY set
* Connected to it.semweb.ch (31.27.148.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=it.semweb.ch
*  start date: Feb 17 00:00:00 2021 GMT
*  expire date: Mar 20 23:59:59 2022 GMT
*  subjectAltName: host "it.semweb.ch" matched cert's "it.semweb.ch"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.
* Server auth using Basic with user 'demo2'
> GET /S2Mapi/thesaurus/ASSEMBLY/sparql/&query=select%20%2A%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D%20limit%2010 HTTP/1.1
> Host: it.semweb.ch
> Authorization: Basic ZGVtbzI6ZGVtbzI=
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Date: Sat, 17 Jul 2021 04:14:02 GMT
< Server: sks
< Strict-Transport-Security: max-age=0
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Disposition: attachment; filename="sks-2021-07-17-06:14:02.423684.json"
< Content-Type: application/sparql-results+json
< Set-Cookie: JSESSIONID=77B9B6447A8718BE93E121D1A0FD8CF3; Path=/S2Mapi; Secure; HttpOnly
< Transfer-Encoding: chunked
< 
{
  "head" : {
    "vars" : [ "s", "p", "o" ]
  },
  "results" : {
    "bindings" : [ {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/0"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_prefix"
      },
      "o" : {
        "type" : "literal",
        "value" : "ASSEMBLY"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/0"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_uri"
      },
      "o" : {
        "type" : "literal",
        "value" : "https://semweb.ch/assembly/"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/1"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_prefix"
      },
      "o" : {
        "type" : "literal",
        "value" : "dcterms"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/1"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_uri"
      },
      "o" : {
        "type" : "literal",
        "value" : "http://purl.org/dc/terms/"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/10"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_prefix"
      },
      "o" : {
        "type" : "literal",
        "value" : "umbel-ac"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/10"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_uri"
      },
      "o" : {
        "type" : "literal",
        "value" : "http://umbel.org/umbel/ac/"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/100"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_prefix"
      },
      "o" : {
        "type" : "literal",
        "value" : "dbpedia"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/100"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_uri"
      },
      "o" : {
        "type" : "literal",
        "value" : "http://dbpedia.org/resource/"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/101"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_prefix"
      },
      "o" : {
        "type" : "literal",
        "value" : "fn"
      }
    }, {
      "s" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns/101"
      },
      "p" : {
        "type" : "uri",
        "value" : "http://semweb.ch/onto/ns_uri"
      },
      "o" : {
        "type" : "literal",
        "value" : "http://www.w3.org/2005/xpath-functions/"
      }
    } ]
  }
}

The only thing different here is that I've replaced the character in front of the query parameter: in the original URL it was a '?', in the replacement it is an '&'. It should be noted that the first URL is factually the correct one: the '?' character indicates the start of the list of parameters in a url, the '&' is only a separator between different params. It looks as if the endpoint software is not parsing the request parameters correctly.

[abrokenjester]

As a workaround: I've noticed that if you use a POST request instead of a GET request to evaluate the query, the endpoint does respond correctly.

RDF4J uses HTTP GET by default for SPARQL queries unless the request url gets too long, but you can force it to always use HTTP POST by setting the environment variable rdf4j.sparql.url.maxlength to a value of '0' (you can do this from code by using System.setProperty("rdf4j.sparql.url.maxlength", "0"); or else when starting up your Java program by supplying a -Drdf4j.sparql.url.maxlength=0 flag).

[redskate]

Thank you so much for trying that with curl and GET. Strange difference because both endpoints are made with absolutely the same software, just deployed to different machines. This query evaluation should be in both cases always be performed by a POST. The fact that using curl with GET the characters "?" and "&" are interpreted differently is not an issue, since the software does some URL rewriting for user convenience. [But the behaviour of the same software must be the same independently of the server executing it.]

Wontfix?

Is in your good right.

BUT

The fact that RDF4j gets confused "could" result at least in a more solid RDF4j reaction "I am confused". At least a check to one missing header like you think - could help (others) seeing what is missing.

Thanks.

[abrokenjester]

@redskate I have converted your issue to a discussion topic on our forum, so we can talk about it further here.

Wontfix?

Is in your good right.

Please don't misunderstand: I marked your issue as "won'tfix" because I believe there is nothing to fix in RDF4J here. The problem is purely at the end of the server that you're querying. That doesn't mean I am not interested in helping by providing what advice I can and discussing it further. And if it turns out that I was wrong in my assessment we can re-open an issue for a fix in RDF4J, where necessary.

The fact that RDF4j gets confused "could" result at least in a more solid RDF4j reaction "I am confused". At least a check to one missing header like you think - could help (others) seeing what is missing.

But this is exactly what RDF4J is doing. It is signalling an error with the following message: "Server responded with an unsupported file format: null" - and that is exactly what is wrong, the server responded with an unsupported file format (or rather, no file format at all, hence "null"). I'll happily hear from you how we can make this clearer but it seems pretty specific to me.

Now, back to this:

Thank you so much for trying that with curl and GET. Strange difference because both endpoints are made with absolutely the same software, just deployed to different machines.

Thanks for highlighting this, I had previously overlooked that you were in fact working with two separate endpoints. Yet the problem is clearly in how the server at it.semweb.ch is set up, because when I do a curl request to the semwebs.com machine I can not reproduce the same error that I saw when doing curl requests to the it.semweb.ch machine.

This query evaluation should be in both cases always be performed by a POST.

I am not sure what you mean by this, but: the SPARQL protocol allows the client to choose whether to use GET or POST for sending a SPARQL query. RDF4J defaults to GET unless the query (and therefore the resulting URL) becomes too long (default limit is 4083 characters), then it falls back to using POST requests. But any standards-compliant SPARQL endpoint is expected to honour both GET and POST requests.

The fact that using curl with GET the characters "?" and "&" are interpreted differently is not an issue, since the software does some URL rewriting for user convenience. [But the behaviour of the same software must be the same independently of the server executing it.]

Then that rewriting bit might be where things are going wrong. Is that something you have configured yourself and/or have any control over? I suspect that rather than this "not being an issue", it is actually the heart of the problem. The variant I tried with an '&' instead of a '?' is actually not syntactically legal, I just happened to try that because I made a copy-paste error when testing using curl. So the endpoint accepts a syntactically incorrect request (with the '&'), but not the syntactically correct version (with a '?').

[redskate]

Hi Jeen - thank you for that.

The API of that piece of software was designed (by myself) so that the API uses an expression like /token/token/token in order to identify the desired objects. Thank you for your hint, but I do not think my self made (sounds vorbidden ;) ) URL rewriting be the heart of the problem, since the sesame access works to semwebs.com ! So no problem actually. If you force using of GET for a SPARQL query there you will not get any good result since I predisposed that a SPARQL delivery be made using a POST (this is compliant with "law" ;) and allows even looong queries to be submitted and processed).

Using the other server (it.semweb.ch) - which is phisically btw, in my intranet but uses an external domain URI for the moment - this is the only "electrical" difference between the two servers I am considering at the moment. One (semwebs.com) is a virtual Ubuntu 18 server (hosted outside) and it.semweb.ch is a metal linux machine 3 meters phisically from my studio.

Using for a moment the SERVICE directive inside the SPARQL engine in the Workbench of one of the current RDF graph producers, GraphDB, access to BOTH servers delivers triples in that GraphDB SPARQL engine panel. Without having reverse engineered GraphDB (prohibited or not my target at the moment) I suppose even that engine is using some SESAME code to access the SPARQL servers on my machines. If this is not so, then again something in RDF4J could be revisited - since - it should run at least as well for both servers - as GraphDB does using the SERVICE directive (see above).

Thanks.

[abrokenjester]

The API of that piece of software was designed (by myself) so that the API uses an expression like /token/token/token in order to identify the desired objects. Thank you for your hint, but I do not think my self made (sounds vorbidden ;) ) URL rewriting be the heart of the problem, since the sesame access works to semwebs.com ! So no problem actually.

Well, you know your own code best - I can only comment on the behaviour I see from the outside, and what I see is that the endpoint at it.semweb.ch is not responding correctly to a valid SPARQL query request, while the server at semwebs.com is responding correctly to that same request. This has nothing to do with RDF4J, since we can see that same incorrect response when doing a curl request.

If you force using of GET for a SPARQL query there you will not get any good result since I predisposed that a SPARQL delivery be made using a POST (this is compliant with "law" ;) and allows even looong queries to be submitted and processed).

It may not be against the law precisely, but it certainly limits you in the SPARQL-compliant clients the endpoint implementation can interact with. It also strikes me as odd to say that this is a design choice made in the endpoint software, given that the exact same software on a different domain does support using a GET request.

Finally, even if we assume that not supporting GET is by design, the it.semweb.ch endpoint has a bug, because it responds to a GET request with a HTTP 200 response, but that then contains an error message in json format, and omits a content-type. To quote SPARQL 1.1 Protocol section 2.1.6:

The response body of a successful query operation with a 2XX response is either:

• a SPARQL Results Document in XML, JSON, or CSV/TSV format (for SPARQL Query forms SELECT and ASK); or,
• an RDF graph [RDF-CONCEPTS] serialized, for example, in the RDF/XML syntax [RDF-XML], or an equivalent RDF graph serialization, for SPARQL Query forms DESCRIBE and CONSTRUCT).

The response the it.semweb.ch endpoint sends is neither of these things. If it wants to indicate that it doesn't support GET requests, it should at the very least use an error status code (HTTP 500 "Internal Server Error" or perhaps 501 "not implemented").

Having said all that, if you want to stick by the choice to only allow POST requests for queries, you can (as indicated in an earlier reply) still configure your RDF4J client to force it to always use POSTs, which will work around the problem.

I also think it's fair to say that we should make this config choice a little more obvious and easy to tweak from code - an undocumented system property is hardly the most obvious thing :) - I've logged an enhancement ticket for this: #3209.

Using for a moment the SERVICE directive inside the SPARQL engine in the Workbench of one of the current RDF graph producers, GraphDB, access to BOTH servers delivers triples in that GraphDB SPARQL engine panel. Without having reverse engineered GraphDB (prohibited or not my target at the moment) I suppose even that engine is using some SESAME code to access the SPARQL servers on my machines.

GraphDB uses RDF4J internally, not Sesame (Sesame is no longer actively developed, and hasn't been since RDF4J got started about 5 years ago). As for why it successfully queries both endpoints: an educated guess is that GraphDB adds a long list of namespace declarations to the query before sending it, which pushes the length over the preconfigured threshold, thereby switching to POST requests.

[redskate]

Hi Jeen

I accept all your points and argumentation (this exchange has a bit of a thriller) and will dive into my code adding a content type information in case of "GET received while POST expected" (yes, the API is designed to work in some cases using GET, in others osing POST - "mea culpa" ;) ) - and appreciate your enhancement tickets (#3209)

thank you!