schema with @reverse @count in one predicate and without, the speed is total different

[shanghai-Jerry]

If you suspect this could be a bug, follow the template.

• What version of Dgraph are you using?
  v1.0.5

• Have you tried reproducing the issue with latest release?
  No

• What is the hardware spec (RAM, OS)?
  64G, Ubuntu 16.04.2 LTS

• Steps to reproduce the issue (command/config used to run Dgraph).
  nohup dgraph zero -w zw5 --replicas 1 > info1 &
  nohup dgraph server -p p5 -w w5 --lru_mb 20000 --zero localhost:5080 > info2 &
  nohup dgraph-ratel > info3 &

• Expected behaviour and actual result.
  Situation:
  schema with @reverse @count : the dgraph server process consumes 107% cpu, of course the speed
  is slow.
  schema without @reverse @count : : the dgraph server process consumes over 1000% cpu,
  fortunately, we had fast put speed than ever before.

  That's the matter i think, what did i miss something when the dgraph run the schema with @reverse
  @count, or should I first put entities without these and then alter schema ？

  thanks for your help.

[shanghai-Jerry]

@deepakjois @sigma @mohitranka @kostub @voxadam

[shanghai-Jerry]

There are some exceptions : [DEADLINE_EXCEEDED, UNKNOWN: While proposing to RAFT group, err: context deadline exceeded], when this happened, it will miss some datas to put into dgraph.

[manishrjain]

You really want to set the schema upfront, if possible. The slower speed is because reverse and count add extra edges, which take more cycles to process. Reverse itself would double the data size, and count is an expensive index to build, adding 2 more edges per edge. So, this data set might be 3x the number of edges for that predicate. That's why the loading speed appears slow.

If a deadline exceed happens, you can just retry.

[shanghai-Jerry]

ok, Thanks and by the way, is this log normal in server ?
Couldn't take snapshot, txn watermark: [1100420], applied watermark: [1122549] @manishrjain

And retry might be still deadline or not ? what's the exact reason for the deadline?

[shanghai-Jerry]

I retried when i found this exception, but it still happened deadline exceed in retry mutation

[manishrjain]

Are you setting a strict timeout in your queries? Hard to say without more details.

[shanghai-Jerry]

Not in queries, In mutations.

[shanghai-Jerry]

i totally write all data into dgraph with some deadline but with retries there no data miss. And than want to add @reverse in one predicate but with this error:

2018/05/18 02:07:53 list.go:797: Got err in while doing async writes in SyncIfDirty: open p9: too many open files
While opening directory: p9.

What can i do? @manishrjain

[manishrjain]

If you're running into too many open files limit, you might want to adjust your ulimit.

From Badger README:

Are there any linux specific settings that I should use?
We recommend setting max file descriptors to a high number depending upon the expected size of you data.

[shanghai-Jerry]

ulimit -n i tried and it's no problem now.

[shanghai-Jerry]

Hi,
i have a question:
once time i found dgraph server will be unavailable. and i want know if this happend,
restart the dgraph server.

is there a possible: the server missed some data

[manishrjain]

I think if the server just crashed and came back, it shouldn't have lost any data. It has ways to recover all the data that was acknowledged.

[shanghai-Jerry]

i am confused why the dgraph server suddenly crashed with out any logs, what happened to the server, is it not enough memory?