cannot retrieve posting for UID from list with key. readTs less than minTs.

[danielmai]

What version of Dgraph are you using?

v20.03.1

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Ubuntu Linux

Steps to reproduce the issue (command/config used to run Dgraph).

These steps don't reproduce the issue all the time, but I do see it sometimes.

1. Run 6-node Dgraph cluster.
2. Live load 21-million movie data set.
3. Run a query, e.g.,

      {
     director(func: allofterms(name@en, "steven spielberg")) {
       name@en
       director.film {
         name@en
         initial_release_date
         country {
           name@en
         }
         starring {
           performance.actor {
             name@en
           }
           performance.character {
             name@en
           }
         }
         genre {
           name@en
         }
       }
     }
   }
   

4. Restart the same Alpha you ran a query on
5. Run the same query on the restarted Alpha.

Expected behaviour and actual result.

When running step 4, I sometimes get this error

Error Name: t
Message: : cannot find value without language tag from list with key 0000026964000000000000009c41: cannot retrieve posting for UID 18446744073709551615 from list with key 0000026964000000000000009c41: readTs: 61825 less than minTs: 85561 for key: "\x00\x00\x02id\x00\x00\x00\x00\x00\x00\x00\x9cA"
Raw Error:
{
  "name": "t",
  "url": "http://a3c58ff3185ec11eab20b027ea48a6ae-164055270.us-west-2.elb.amazonaws.com:8080/query?timeout=20s&debug=true&ro=true&be=true",
  "errors": [
    {
      "message": ": cannot find value without language tag from list with key 0000026964000000000000009c41: cannot retrieve posting for UID 18446744073709551615 from list with key 0000026964000000000000009c41: readTs: 61825 less than minTs: 85561 for key: \"\\x00\\x00\\x02id\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x9cA\"",
      "extensions": {
        "code": "ErrorInvalidRequest"
      }
    }
  ]
}

After some amount of time (could be seconds or hours), the error goes away and the query works as expected.

[danielmai]

I saw this in test.sh as well.

INFO: Running test for github.com/dgraph-io/dgraph/dgraph/cmd/counter
[Decoder]: Using assembly version of decoder
--- FAIL: TestBestEffortTs (0.67s)
    increment_test.go:157: Initial value: 1
    increment_test.go:58: Last value written by increment in loop: 2
    increment_test.go:131: Last value read by best effort: 1
    increment_test.go:58: Last value written by increment in loop: 3
    increment_test.go:235: 
        	Error Trace:	increment_test.go:235
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = : cannot find value without language tag from list with key 00000b636f756e7465722e76616c0000000000000f44ce: cannot retrieve posting for UID 18446744073709551615 from list with key 00000b636f756e7465722e76616c0000000000000f44ce: readTs: 2482 less than minTs: 2483 for key: "\x00\x00\vcounter.val\x00\x00\x00\x00\x00\x00\x0fD\xce"
        	            	while doing query
        	            	github.com/dgraph-io/dgraph/dgraph/cmd/counter.queryCounter
        	            		/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:88
        	            	github.com/dgraph-io/dgraph/dgraph/cmd/counter.TestBestEffortTs
        	            		/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment_test.go:234
        	            	testing.tRunner
        	            		/usr/local/go/src/testing/testing.go:992
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1373
        	Test:       	TestBestEffortTs

[martinmr]

That seems like it could be a temporary error due to rollup. Notice that the timestamps only differ by one., indicating that the rollup is recent. Retrying in this situation would probably help.

[willem520]

when I run query, I got the similar result
{"errors":[{"message":": rpc error: code = Unknown desc = readTs: 6 less than minTs: 11 for key: \"\\x00\\x00\\x03gid\\x02\\x00\\v\\xbf?L\\x93\\x1d|/P.\\xb1n\\x81i\u003cA\\x89$\\x12\\xf5\\r\\x1a\\x1e\\xbe\\xd8@\\xa1=D\\x13\\xac59\"","extensions":{"code":"ErrorInvalidRequest"}}],"data":null}
I only can query schema or uid

[danielmai]

@willem520 Are you running Dgraph with ludicrous mode enabled? And did you see this error after Dgraph Alpha restarted?

I can easily see this when running ludicrous mode since there's a known bug where write-ahead log snapshots aren't created when ludicrous mode is on, so an Alpha restart causes a long list of write-ahead log entries to be replayed.

[willem520]

@willem520 Are you running Dgraph with ludicrous mode enabled? And did you see this error after Dgraph Alpha restarted?

I can easily see this when running ludicrous mode since there's a known bug where write-ahead log snapshots aren't created when ludicrous mode is on, so an Alpha restart causes a long list of write-ahead log entries to be replayed.

I run Dgraph without ludicrous mode. I got the above result when I bulk a large dataset which cost about 2 hour.

[ashish-goswami]

Hey @willem520, in which version of Dgraph did you see above issue?

[willem520]

Hey @willem520, in which version of Dgraph did you see above issue?
Hi @ashish-goswami ,I used v1.2.1

[minhaj-shakeel]

Github issues have been deprecated.
This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.