sparql: counts don't seem to be reliable

[joernhees]

I'm trying to get a top type count for DBpedia (Virtuoso version 07.00.3207 on Linux (x86_64-redhat-linux-gnu), Single Server Edition):

select ?type count(distinct ?s) as ?c where {
  ?s a ?type.
}
group by ?type
order by desc(?c)
limit 50

returns (apart from other rows) this row 1:
http://dbpedia.org/ontology/Place 89498

Out of curiosity i checked this again with the query below:

select count(distinct ?s) where { ?s a <http://dbpedia.org/ontology/Place> }

tells me it's 754450 2

There's an order of magnitude difference in these 2 counts. Please tell me I'm doing it wrong.

PS: i tried the first query without the group by, order by and limit clause, doesn't make a difference.

[kidehen]

Extending the timeout parameter increases the time allotted to producing
the query solution. This feature is critical to enabling the whole world
use DBpedia rather than specific queries monopolizing processing time.

See the different results produced when I doubled up the processing time:
http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbpedia.org&query=select+%3Ftype+count%28distinct+%3Fs%29+as+%3Fc+where+%7B%0D%0A++%3Fs+a+%3Ftype.%0D%0A%7D%0D%0Agroup+by+%3Ftype%0D%0Aorder+by+desc%28%3Fc%29%0D%0Alimit+50&format=text%2Fhtml&timeout=60000&debug=on

Note, there are hard limits also configured on the server that override
what may come in from an HTTP client.

[indeyets]

@kidehen timeouts are understandable. But giving wrong result because of timeout is the whole different story.

shouldn't it report failure instead? that's what fuseki does, for example.

both outcomes are not helpful, but fuseki doesn't provide false results

[kidehen]

This isn't a false result.

This is a solution to the query within the constraints of a timeout.
The server should indicate via HTTP response metadata the nature of the
solution i.e., partial or complete.

This is a feature of Virtuoso.

[joernhees]

Thanks, the time limit explains a bit, but this "feature" is highly confusing if not dangerous because the user (in this case me and i'm not exactly a novice) might not be aware that all the counts might be terribly wrong.

Is there any way to distinguish a "cut-off" result from one which is accurate?

I had assumed that a query which hits a timeout limit would return with an error (something like a 408, even though i'm not sure it's actually the right one) instead of silently returning wrong results.

[kidehen]

On 12/3/13 9:18 AM, Jörn Hees wrote:

Thanks, the time limit explains a bit, but this "feature" is highly
confusing if not dangerous because the user (in this case me and i'm
not exactly a novice) might not be aware that all the counts might be
terribly wrong.

There is a DBpedia fair-use document [1],[2] about this matter. You won't
have this issue if you are running your own Virtuoso instance with the
DBpedia dataset. Please remember, on the World Wide Web we have to cater
for everyone. The Web presents unique challenges to DBMS technology that
we address in Virtuoso, specifically.

[1] http://dbpedia.org/OnlineAccess -- search on "Fair Use Policy"
[2] http://lists.w3.org/Archives/Public/public-lod/2011Aug/0028.html

Is there any way to distinguish a "cut-off" result from one which is
accurate?

This should be part of the response headers. Note:
X-SQL-State: S1TAT

I had assumed that a query which hits a timeout limit would return
with an error (something like a 408, even though i'm not sure it's
actually the right one) instead of silently returning wrong results.

Yes, there has to be HTTP response metadata indicating the state of
affairs, to the degree possible. The problem right now is that we don't
have any standardization here. 408 doesn't cut it because it implies the
request couldn't be completed. In our case, we are completing a task
within a set time that has been reached.

The closest analogy here is a quiz contest where you have X seconds to
answer a question, this is the very model to which Virtuoso's query
engine has been developed.

[joernhees]

@kidehen neither of the links you provide describe/warn of the reported problem: that counts can be wrong if a timeout is hit.

I don't seem to have gotten my point across, let me try again:

I'm not arguing with fair use, timeouts or limits to be able to satisfy more users. I'm a fan!

I'm arguing with the way you're treating a timeout.
If a query takes too long there are two ways of dealing with this:

1. return an error, not a result
2. return a result with a BIG WARNING
3. makes a developer, user or scientist (with a quick one-off sparql query in your web interface) look into it again. They will definitely not run in danger of using a wrong results as there is none!
4. probably leads to the warning being lost somewhere in the process, never be shown to the user and the numbers taken for granted in the end. This is what happened here. Not even your own HTML result page of your SPARQL Web Interface shows the tiniest hint to the user that he should be careful. Even if you're arguing that this is not the prime "end user"... can you name a widely used sparql client / lib which handles this correctly?

I can see your point of view trying to answer the query as good as you can in the given time, but as this report demonstrates it is more dangerous than just returning an error.

[joernhees]

one addenum, sorry:
it should be optional to get partial results, not an implicit default that you then have to check for

[kidehen]

On 12/4/13 6:50 AM, Jörn Hees wrote:

one addenum, sorry:
it should be optional to get partial results, not an implicit default
that you then have to check for

No for the public DBpedia instance. We are deliberately not giving
anyone the ability to hog the instance. The instance has to be
accessible to the whole world, that's the basic requirement. Again, for
those that want to service specific use of DBpedia there are a range of
options from making your own instances across:

1. local setup
2. cloud setup -- e.g., Amazon AMI.

There are also other instances of DBpedia data across:

1. http://lod.openlinksw.com/sparql -- LOD Cloud cache (this setup has
   more computing power behind it)
2. http://dbpedia-live.openlinksw.com/sparql
3. http://live.openlinksw.com/sparql.

[kidehen]

On 12/4/13 6:49 AM, Jörn Hees wrote:

@kidehen https://github.com/kidehen neither of the links you provide
describe/warn of the reported problem: that counts can be wrong if a
timeout is hit.

I don't seem to have gotten my point across, let me try again:

I'm /not/ arguing with fair use, timeouts or limits to be able to
satisfy more users. I'm a fan!

I'm arguing with the way you're treating a timeout.
If a query takes too long there are two way of dealing with this:

What you are not getting from my comment is the fact that there isn't a
notion of "query taking too long" the notion is "what solution can be
produced in X amount of seconds, for a given query". There's a world of
difference here. The technical challenge is old, SQL DBMS engines never
even got to tackling this issue since their usage context (closed world)
doesn't expose the problem.

With DBpedia and the Web, everything is unpredicatable. Data is
fundamentally time variant.

1. return an error, not a result
2. return a result with a BIG WARNING

We return an indicator via HTTP response (which you can test for) re.
partial results.

3. makes a developer, user or scientist (with a quick one-off sparql
   query in your web interface) look into it again. They will
   definitely not run in danger of using a wrong results as there is
   none!

DBpedia isn't a gospel of any kind. That isn't the purpose here. Please
think about your request a little. You want a full Transitive Closure
intermingled with entailments for all the entity relationship semantics
in the data space? I am sure (as you digest that last sentence and its
implications) you get the point re., the nature of the pursuit and its
fundamental impracticalities, at Web-scale.

4. probably leads to the warning being lost somewhere in the process,
   never be shown to the user and the numbers taken for granted in
   the end. This is what happened here. Not even your own HTML result
   page of your SPARQL Web Interface shows the tiniest hint /to the
   user/ that he should be careful. Even if you're arguing that this
   is not the prime "end user"... can you name a widely used sparql
   client / lib which handles this correctly?

I can see your point of view trying to answer the query as good as you
can in the given time, but as this report demonstrates it is more
dangerous than just returning an error.

It is dangerous to attempt the opposite i.e., have no restrictions and
let clients deliberately or inadvertently deprive others of use. As I
said, there are other options for special use of DBpedia. It isn't right
to assume DBpedia is there to produce complete solutions for any kind
of query, issued by any kind of client, at any given point in time.

We have made a choice to make DBpedia available to the world, backed up
with usage restrictions that defend the goal :-)

[joernhees]

On 4 Dec 2013, at 14:09, Kingsley Idehen notifications@github.com wrote:

It isn't right
to assume DBpedia is there to produce complete solutions for any kind
of query, issued by any kind of client, at any given point in time.

Are you sure that this is your statement?
It's a marketing disaster.
And it's not what i'm asking for / reporting here as a problem.

I just wanted the correct counts for types used on the DBpedia endpoint.
There is no open world assumption in my query:
i'm neither asking the SPARQL endpoint to resolve redirects, nor is this a federated query.

All i'm asking for are its counts at the current point in time.
Nothing fancy and i could happily live with an error due to time exceeded.

Partial results are cool when you ask for them (explicitly), i didn't and most people don't.

If not explicitly asked for a partial result, it's more dangerous to report them in a very similar fashion to a complete result than reporting an error.

Ask a couple of developers what they would expect to happen…
Rather get an error or a result that looks quite right but isn't?

Cheers,
Jörn

[kidehen]

On 12/9/13 12:37 PM, Jörn Hees wrote:

On 4 Dec 2013, at 14:09, Kingsley Idehen notifications@github.com
wrote:

It isn't right
to assume DBpedia is there to produce complete solutions for any kind
of query, issued by any kind of client, at any given point in time.

Are you sure that this is your statement?

My statement is this:

DBpedia is going to produce solutions to SPARQL queries subject to
timeout limits and other constraints that have been deliberately
configured to ensure global access, in line with its fair use policy.
This is how DBpedia's SPARQL endpoint has been configured to run since
inception.

It's a marketing disaster.

DBpedia isn't about marketing. I am making a statement about the
technical infrastructure behind the DBpedia SPARQL endpoint.

And it's not what i'm asking for / reporting here as a problem.

You are reporting the fact that you are executing a specific query that
(in the form you are seeking) exceeds some of the fair use constraints.
There are other instances of the DBpedia dataset, associated with
different infrastructure, that will give you more computing power per
timeout restrictions etc.. Examples include:

[1] http://lod.openlinksw.com/sparql -- all you have to do is simply
change the host part of your SPARQL Protocol URL to see what I mean re.
this cluster edition of Virtuoso which also has more computing power
behind it.

[2] http://dbpedia-live.openlinksw.com -- which doesn't match LOD for
capacity but can have less concurrent traffic than the main dbpedia.org
SPARQL endpoint.

I just wanted the correct counts for types used on the DBpedia endpoint.

The DBpedia Endpoint is one point of access for the DBpedia dataset. An
Endpoint != a Dataset. It is a service that provides access to a
dataset. There are other services providing access to the same dataset
that are configured for more intensive use of the data. The main
endpoint is for the whole world, and that's the focus of its
configuration i.e., every one (human or machine) has fair use of the
endpoint.

There is no open world assumption in my query:
i'm neither asking the SPARQL endpoint to resolve redirects, nor is
this a federated query.

That doesn't eradicate entailment, transitive closures, and related
matters. Even if you aren't actually de-referencing HTTP URIs, does that
apply to all other agents (human or machine) ?

All i'm asking for are its counts at the current point in time.

You are one of many.

Nothing fancy and i could happily live with an error due to time
exceeded.

There are other endpoints, change the hostname part of the URL, as I've
already told you.

Partial results are cool when you ask for them (explicitly), i didn't
and most people don't.

I guess Google give you complete results?

If not explicitly asked for a partial result it's more dangerous to
report them in a very similar fashion to a complete result than
reporting an error.

For you and your use case. You are but one agent.

Ask a couple of developers what they would expect to happen…
Rather get an error or a result that looks quite right but isn't?

It been so since 2007, so I don't understand what you are making a fuss
about, especially when the lod.openlinksw.com/sparql instance will more
than likely get you a complete answer based on the nature of its
configuration.

Cheers,
Jörn

—
Reply to this email directly or view it on GitHub
#112 (comment).

Regards,

Kingsley Idehen
Founder & CEO
OpenLink Software
Company Web: http://www.openlinksw.com
Personal Weblog: http://www.openlinksw.com/blog/~kidehen
Twitter Profile: https://twitter.com/kidehen
Google+ Profile: https://plus.google.com/+KingsleyIdehen/about
LinkedIn Profile: http://www.linkedin.com/in/kidehen

[joernhees]

On 9 Dec 2013, at 20:18, Kingsley Idehen notifications@github.com wrote:

You are reporting the fact that you are executing a specific query that
(in the form you are seeking) exceeds some of the fair use constraints.

wrong, read again

i'm fine with that and always have been

i report that how you treat a timeout is bad, not that there is a timeout

end of my feedback

thanks for your time

j

[kidehen]

On 12/9/13 2:56 PM, Jörn Hees wrote:

On 9 Dec 2013, at 20:18, Kingsley Idehen notifications@github.com wrote:

You are reporting the fact that you are executing a specific query that
(in the form you are seeking) exceeds some of the fair use constraints.

wrong, read again

i'm fine with that and always have been

i report that how you treat a timeout is bad, not that there is a
timeout

end of my feedback

thanks for your time

j

—
Reply to this email directly or view it on GitHub
#112 (comment).

I am not claiming that the timeout treatment is perfect. I've told you
repeatedly that we are using a custom HTTP header due to the lack of a
standard header for this situation.

This isn't a 408 or 500 condition.

[joernhees]

@iv-an-ru any update on this?