sort result different by orderasc or not, the count only 1000 by orderasc

[JustinRong]

Produce:

1. prepare data

data.js

const fs = require('fs');

const rdf = (i) => {
    return `_:x${i} <price> "1.5${i}" . #`;
}

const rdfs = (n) => {
    for (var i = 0; i < n; i++) {
        console.log(rdf(i))
    }
}

rdfs(process.argv[2])

// node data.js 1 > 1.rdf
// node data.js 1000 > 1000.rdf

2. load data.
   first, load 1000 data.
   dgraph live -r 1000.rdf
   check the result with query:

{
  A (func: has(price), orderdesc: <price>){
count(uid)
}
B(func: has(price)){
count(uid)
}
}

the result is

"data": {
    "A": [
      {
        "count": 1000
      }
    ],
    "B": [
      {
        "count": 1000
      }
    ]

From now on , It's all ok.

then , load 1 row.
dgraph live -r 1.rdf

re query it

{
  A (func: has(price), orderdesc: <price>){
count(uid)
}
B(func: has(price)){
count(uid)
}
}

the result is :

"data": {
    "A": [
      {
        "count": 1000
      }
    ],
    "B": [
      {
        "count": 1001
      }
    ]
  },

1000 != 1001

Dgraph version Info:

Dgraph version   : v1.0.7
Commit SHA-1     : f1803442
Commit timestamp : 2018-08-10 13:00:21 -0700
Branch           : HEAD

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed under Apache 2.0 + Commons Clause. Copyright 2015-2018 Dgraph Labs, Inc.

[MichelDiz]

Can you show the format of your RDF?
Are you running one line at a time from RDF? why?

RDF should be loaded via LiveLoad or BulkLoad.

Cheers.

[BlankRain]

@MichelDiz the rdf data is genarated by javascript code named data.js
save the code to a file data.js

const rdf = (i) => {
    return `_:x${i} <price> "1.5${i}" . #`;
}

const rdfs = (n) => {
    for (var i = 0; i < n; i++) {
        console.log(rdf(i))
    }
}

rdfs(process.argv[2])

then run command like that:
(you need install nodejs first)
node data.js 1 > 1.rdf
node data.js 1000 > 1000.rdf
now ,you get two data files named 1.rdf and 1000.rdf.

Then ,load data via dgraph live
run dgraph live -r 1000.rdf first
then check the result
run dgraph live -r 1.rdf
check the result and you will find the bug.

[MichelDiz]

Well, I've been able to reproduce in two ways. RDF and JSON. Both gave this strange result. I will consider Bug, but I will request review from the team. It really does not make sense the sorting Indices cut off the others nodes. That looks like a lot of bug.

const fs = require('fs');

const JSONFORMAT = (i) => {
    return `{ "price": 1.5${i} },`;
}

const rdfs = (n) => {
    for (var i = 0; i < n; i++) {
        console.log(JSONFORMAT(i))
    }
}

rdfs(process.argv[2])

Cheers

PS UPDATE

One more test, to see what happen between edges.

PS². It's happening to Int and Float. Whenever it passes 1 thousand nodes.

{
  A (func: has(root_test)){
   root_test (orderasc: price){
    count(uid)
  }

}
B(func: has(root_test)){
   root_test {
    count(uid)
  }
}
}

Mutation

{
    "set": [
      { "root_test": 
       [
              { "price": 1.50 },
              { "price": 1.51 },
              { "price": 1.52 },
              { "price": 1.53 },
              (...)
              { "price": 1.5999 }
            ]
      }
    ]
}

Result

{
  "data": {
    "A": [
      {
        "root_test": [
          {
            "count": 1000
          }
        ]
      }
    ],
    "B": [
      {
        "root_test": [
          {
            "count": 2000
          }
        ]
      }
    ]
  }

[MichelDiz]

Curiously works using pagination oO

{
  A (func: has(root_test)){
   root_test (orderasc: value, first: 10000){
    count(uid)
  }

}
B(func: has(root_test)){
   root_test {
    count(uid)
  }
}
}

Can you guys test this?

Cheers

PS. Result:

{
  "data": {
    "A": [
      {
        "uid": "0x1041f",
        "root_test": [
          {
            "count": 6000
          }
        ]
      }
    ],
    "B": [
      {
        "uid": "0x1041f",
        "root_test": [
          {
            "count": 6000
          }
        ]
      }
    ]
  }

[MichelDiz]

Well folks, it's not a bug. This is a limitation by default.

// Sort and paginate directly as it'd be expensive to iterate over the index which
// might have millions of keys just for retrieving some values.

// Only retrieve up to 1000 results by default.

"if no "first" or "last" etc. argument is specified, it would default to 1000." mrjn.

https://github.com/dgraph-io/dgraph/blob/06ea4c545bc3cf0ed730327557dbff96406e75f8/query/query.go#L2146

Thank you for reporting this.

Cheers.

[jantoine1]

I just ran into this issue and believe it should be classified as a bug. Having a query language that alters the number of results returned when applying a sort is unintuitive, unexpected and can be very difficult to notice and debug when starting with a new query on a small dataset that will grow with time.

[MichelDiz]

Sorry, not a bug. As it was hardcoded in stone. Also it is documented. This helps to avoid OOM with too complex queries runned by mistake.

If you wanna a huge dataset you should do it explicitly(preferably in batches). To not incur cluster-wide overhead. Making it lose its healthy state.