Data missing in Dgraph cluster after bulk loading

[veludurai106]

DGraph Version - 1.0.3

OS - Centos 7

Steps to reproduce the issue

• Started zero server
  nohup dgraph zero --my=10.111.111.101:5080 --replicas=3 --idx=01 &

• Ran bulk loader
  dgraph bulk -r /home/mapr/DgraphResources/sample.rdf -s /home/mapr/DgraphResources/sample.schema --map_shards=6 --reduce_shards=3 -z 10.111.111.101:5080

sample.schema.txt
sample.rdf.txt

• Copied the p folders to all 3 nodes (/opt/dgraph/data)

• Started dgraph server in 3 nodes (inside /opt/dgraph/data)
  nohup dgraph server --memory_mb=16000 --my=10.111.111.101:7080 --zero=10.111.111.101:5080 &
  nohup dgraph server --memory_mb=16000 --my=10.111.111.104:7080 --zero=10.111.111.101:5080 &
  nohup dgraph server --memory_mb=16000 --my=10.111.111.107:7080 --zero=10.111.111.101:5080 &

• Expected behaviour
  I will get result if I query any of the 3 nodes (using IP:port/query API)

• Actual behaviour
  Only one node returns partial result. whereas when I query other 2 nodes , I am getting empty result.

End point - http://10.111.111.107:8080/query?debug=true
Post Body - query.txt
Response - response.txt

Result of cluster state api - /state
cluster state.txt

Node
I initially tried with ~50M edges, faced same issue.

Please help me to understand what is missing!.

[pawanrawal]

I would suggest running bulk loader with --reduce_shards as 1, since you want the 3 Dgraph servers to be replicas of each other. Then give the out/0/p as input to one of the Dgraph servers while starting it.

nohup dgraph server --memory_mb=16000 --my=10.111.111.101:7080 --zero=10.111.111.101:5080 -p out/0/p &

The other replica Dgraph servers would get using the snapshot on starting. I will investigate this issue anyway.

[pawanrawal]

I am going to close the issue. The issue about a node returning partial result should not occur and I couldn't reproduce it. I have opened another issue about not fetching the snapshot if the initial data in p directory is same for replicas. #2202

Feel free to reopen this issue if you still encounter it.

[sriharshaboda]

Hi @pawanrawal , I was able to reproduce the same issue. I wanted to start dgraph with one zero and 3 servers, I do not want 3 replicas however. I'm only interested in data distribution. Hence I started zero with 1 replica and 3 servers with each pointing to the corresponding reduce shard of bulk output p directory.
When I query, only one of the servers gives partial predicates, rest do not show any predicates at all.

[manishrjain]

So, basically, 3 reduce shards. 3 Dgraph servers. Each pointing to a corresponding shard. And you're not able to get full results as expected?

Can you share the commands you used for bulk loading, and then for running the servers?

[sriharshaboda]

dgraph zero
dgraph bulk -r ~/dgraph/21million.rdf.gz -s ~/dgraph/21million.schema --map_shards=6 --reduce_shards=3 --http localhost:8765 --zero=localhost:5080

ls out 
0  1  2

cd out/0
dgraph server --lru_mb 2048 --zero localhost:5080

cd out/1
dgraph server --lru_mb 2048 --zero localhost:5080 -o 200

cd out/2
dgraph server --lru_mb 2048 --zero localhost:5080 -o 400

Attaching the state, where you can see that only a few predicates occupy a decent space.
state.txt

Hey @sriharshaboda , you are not using ( --my=server ) in your commands so it may be able to talk to just one server..
Just try these commands and let me know if you find any issues working with it.

cd out/0
dgraph server --my=server_1:7080 --lru_mb=2048 --zero=zero:5080

cd out/1
dgraph server --my=server_2:7280 --lru_mb=2048 --zero=zero:5080 -o 200

cd out/2
dgraph server --my=server_3:7480 --lru_mb=2048 --zero=zero:5080 -o 400

[sriharshaboda]

Hi @karan28aug , I've tried with the my option enabled, but that didn't change anything. I still see missing predicates. Please let me know if I should be trying something else instead.