Problem with Subquery

[hannahbast]

UPDATE: I edited this post a bit. There was a small mistake in the query and it's not true that the GROUP BY in the subquery is irrelevant.

Consider the following query on Wikidata, which asks for the political party (P102) of politicians, but only the one joined most recently (P580 = start time). This requires a subquery, which is exactly what seems to be causing problems here.

Without the last triple (... rdfs:label ...) the query seems to work fine. But with that triple, it returns an empty result. In fact, it returns an empty result also if other triples are added (for example triples to get the number of sitelinks).

I have tried to find a smaller example, but did not find one within reasonable time.

PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?person ?party WHERE {
  ?person p:P102 ?p102 .
  ?p102 ps:P102 ?party .
  ?p102 pq:P580 ?party_date .
  {
    SELECT ?person (MAX(?date) AS ?party_date) WHERE {
      ?person p:P102 ?tmp .
      ?tmp pq:P580 ?date .
    }
    GROUP BY ?person
  }
  ?person rdfs:label ?person_name .
}

[niklas88]

So I first thought that this is a problem with TwoColumnJoin having wrong information about which rows are sorted but this does not seem to be the case. I made getResultSortedOn() return an empty vector thus forcing a resort but still the Join upstream of this ends up empty. Here is the query runtime information with the additional Sort on ASC(?person):

│
├─ Join on ?person
│  result_size: 0 x 5
│  columns: ?person, ?person_name, ?party_date, ?p102, ?party
│  total_time: 0.00 ms
│  operation_time: 0.00 ms
│  cached: true
│    original_operation_time: 1.00
│    original_total_time: 6,387.00
│  ┬
│  │
│  ├─ IndexScan ?person <http://www.w3.org/2000/01/rdf-schema#label> ?person_name
│  │  result_size: 285,386,229 x 2
│  │  columns: ?person, ?person_name
│  │  total_time: 6,252.00 ms
│  │  operation_time: 6,252.00 ms
│  │  cached: false
│  │
│  ├─ Sort on ASC(?person)
│  │  result_size: 7,492 x 4
│  │  columns: ?person, ?party_date, ?p102, ?party
│  │  total_time: 134.00 ms
│  │  operation_time: 0.00 ms
│  │  cached: false
│  │  ┬
│  │  │
│  │  ├─ TwoColumnJoin on ?person ?party_date
│  │  │  result_size: 7,492 x 4
│  │  │  columns: ?person, ?party_date, ?p102, ?party
│  │  │  total_time: 134.00 ms
│  │  │  operation_time: 1.00 ms
│  │  │  cached: false
│  │  │  ┬
│  │  │  │
│  │  │  ├─ GroupBy on ?person
│  │  │  │  result_size: 7,528 x 2
│  │  │  │  columns: ?person, ?party_date
│  │  │  │  total_time: 96.00 ms
│  │  │  │  operation_time: 0.00 ms
│  │  │  │  cached: false
│  │  │  │  ┬
│  │  │  │  │
│  │  │  │  ├─ OrderBy on ASC(?date) ASC(?person)
│  │  │  │  │  result_size: 9,962 x 3
│  │  │  │  │  columns: ?tmp, ?person, ?date
│  │  │  │  │  total_time: 96.00 ms
│  │  │  │  │  operation_time: 1.00 ms
│  │  │  │  │  cached: false
│  │  │  │  │  ┬
│  │  │  │  │  │
│  │  │  │  │  ├─ Join on ?tmp
│  │  │  │  │  │  result_size: 9,962 x 3
│  │  │  │  │  │  columns: ?tmp, ?person, ?date
│  │  │  │  │  │  total_time: 95.00 ms
│  │  │  │  │  │  operation_time: 7.00 ms
│  │  │  │  │  │  cached: false
│  │  │  │  │  │  ┬
│  │  │  │  │  │  │
│  │  │  │  │  │  ├─ IndexScan ?person <http://www.wikidata.org/prop/P102> ?tmp
│  │  │  │  │  │  │  result_size: 348,968 x 2
│  │  │  │  │  │  │  columns: ?person, ?tmp
│  │  │  │  │  │  │  total_time: 24.00 ms
│  │  │  │  │  │  │  operation_time: 24.00 ms
│  │  │  │  │  │  │  cached: false
│  │  │  │  │  │  │
│  │  │  │  │  │  ├─ IndexScan ?tmp <http://www.wikidata.org/prop/qualifier/P580> ?date
│  │  │  │  │  │  │  result_size: 2,382,479 x 2
│  │  │  │  │  │  │  columns: ?tmp, ?date
│  │  │  │  │  │  │  total_time: 64.00 ms
│  │  │  │  │  │  │  operation_time: 64.00 ms
│  │  │  │  │  │  │  cached: false
│  │  │  │
│  │  │  ├─ OrderBy on ASC(?person) ASC(?party_date)
│  │  │  │  result_size: 9,879 x 4
│  │  │  │  columns: ?p102, ?party_date, ?party, ?person
│  │  │  │  total_time: 37.00 ms
│  │  │  │  operation_time: 0.00 ms
│  │  │  │  cached: false
│  │  │  │  ┬
│  │  │  │  │
│  │  │  │  ├─ Join on ?p102
│  │  │  │  │  result_size: 9,879 x 4
│  │  │  │  │  columns: ?p102, ?party_date, ?party, ?person
│  │  │  │  │  total_time: 37.00 ms
│  │  │  │  │  operation_time: 1.00 ms
│  │  │  │  │  cached: false
│  │  │  │  │  ┬
│  │  │  │  │  │
│  │  │  │  │  ├─ Join on ?p102
│  │  │  │  │  │  result_size: 9,879 x 3
│  │  │  │  │  │  columns: ?p102, ?party_date, ?party
│  │  │  │  │  │  total_time: 36.00 ms
│  │  │  │  │  │  operation_time: 6.00 ms
│  │  │  │  │  │  cached: false
│  │  │  │  │  │  ┬
│  │  │  │  │  │  │
│  │  │  │  │  │  ├─ IndexScan ?p102 <http://www.wikidata.org/prop/qualifier/P580> ?party_date
│  │  │  │  │  │  │  result_size: 2,382,479 x 2
│  │  │  │  │  │  │  columns: ?p102, ?party_date
│  │  │  │  │  │  │  total_time: 0.00 ms
│  │  │  │  │  │  │  operation_time: 0.00 ms
│  │  │  │  │  │  │  cached: true
│  │  │  │  │  │  │    original_operation_time: 64.00
│  │  │  │  │  │  │    original_total_time: 64.00
│  │  │  │  │  │  │
│  │  │  │  │  │  ├─ IndexScan ?p102 <http://www.wikidata.org/prop/statement/P102> ?party
│  │  │  │  │  │  │  result_size: 347,399 x 2
│  │  │  │  │  │  │  columns: ?p102, ?party
│  │  │  │  │  │  │  total_time: 30.00 ms
│  │  │  │  │  │  │  operation_time: 30.00 ms
│  │  │  │  │  │  │  cached: false
│  │  │  │  │  │
│  │  │  │  │  ├─ IndexScan ?person <http://www.wikidata.org/prop/P102> ?p102
│  │  │  │  │  │  result_size: 348,968 x 2
│  │  │  │  │  │  columns: ?person, ?p102
│  │  │  │  │  │  total_time: 0.00 ms
│  │  │  │  │  │  operation_time: 0.00 ms
│  │  │  │  │  │  cached: true
│  │  │  │  │  │    original_operation_time: 24.00
│  │  │  │  │  │    original_total_time: 24.00

[niklas88]

Ok it looks like the columns get screwed up. In the following query ?person results look like statements and ?party_date like persons. Will investigate further

[niklas88]

Of course the originally reported query also works with PR #259

PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX wikibase: <http://wikiba.se/ontology#>

PREFIX schema: <http://schema.org/>

SELECT ?person_name ?party_name ?party_date ?sitelinks WHERE {
  ?person p:P102 ?p102 .
  ?p102 ps:P102 ?party .
  ?p102 pq:P580 ?party_date .
  {
    SELECT ?person (MAX(?date) AS ?party_date) WHERE {
      ?person p:P102 ?tmp .
      ?tmp pq:P580 ?date .
    }
    GROUP BY ?person
  }
  ?person rdfs:label ?person_name .
  ?about schema:about ?person .
  ?about wikibase:sitelinks ?sitelinks .
  ?party rdfs:label ?party_name .
  FILTER (lang(?person_name) = "en") .
  FILTER (lang(?party_name) = "en") .
}
ORDER BY DESC(?sitelinks)