Frequent errors when querying during heavy mutations

[slawo]

We tend to see this error during heavy usage. We are doing a fair amount of mutations (+-150/s) and this pops quite often when we're doing queries:

dgraph/posting/list.go

Line 548 in f9ce69c

return x.Errorf("readTs: %d less than minTs: %d for key: %q", readTs, l.minTs, l.key)

[manishrjain]

This would trigger if a certain key is being heavily updated. This most likely would be an index key -- for e.g. if you're setting <a> <type> "something" .. And have a lot of nodes of that type.

You can have a closer look at your schema (or, maybe post it here along with some of your data, so we can identify any potential issues). Otherwise, just re-run the transaction.

[slawo]

@manishrjain, yes we have something northward of 160k nodes which have edges with @count and @reverse, each node also has a @index(exact) @count and multiple int @index(int).

During our mutation phase these indexed keys are very heavily updated.

I suppose disabling indexes during the initial ingestion will help... If you have any other suggestions I would be more than happy to try them for our next v0.9.x test phase.

[manishrjain]

Just generally speaking, fewer indices would result in better write throughput and fewer transaction aborts.

[janardhan1993]

@slawo: Its costly to maintain all versions of indexes during heavy mutation so frequently we remove the old versions. The error you mentioned comes when the query is requesting for a version of index which is not present. In that case you can retry the transaction then dgraph would get latest readTimestamp and try to read the latest version.

Regarding `I suppose disabling indexes during the initial ingestion will help...', Some of the queries would require index to work. You can just retry the read transaction or probably wait till initial ingestion to query.

[janardhan1993]

I am closing this issue now. Please ask on slack or create a post on discuss if you need to discuss further.