Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upPredicate moves can take down Dgraph indefinitely #2397
Comments
|
Any suggestions on how quickly can it be reproduced? And with which test? |
|
I haven't figured out a fast path to reproduce this yet, but the original case was And it took about 4 hours to appear. My guess is that the bank test is more likely to be interesting, since there are more predicates. I'm also going to explore reducing the time between nemeses, and longer test times, so we spend more time migrating. |
|
Okay. I'll try running it at my end as well for long. |
|
Ah.. this is a stupid one. I could see this in Dgraph's own integration tests. Have pushed a potential fix in master. 7f21b82 This could be killing off many Dgraph alphas, which is why the entire cluster is failing. Testing the fix right now. |
|
Commit 7205275 also helps fix this one. Ran it on my computer, with the command above. All tests passed. Closing this. Will send you a binary to test as well at your end. |
In the build @manishrjain provided for testing on 2018-05-16 (sorry, don't have a SHA!), with predicate moves roughly every 10 seconds, Dgraph went down: 20180516T194304.000-0500.zip
In particular, it looks like a predicate move caused a single node's Alpha to crash with the following message:
Oddly, the cluster was never able to recover from this state, even though five Zero nodes and four Alpha nodes were running and connected. All alpha nodes logged
... indefinitely, and Zero nodes logged the same thing:
Every request either timed out (on n3), or returned UNAVAILABLE. Attempts to perform further predicate moves all timed out as well, though Zero would answer
/statequeries:{:counter "1079", :groups {1 {:members {1 {:id "1", :groupId 1, :addr "n1:7080"}, 2 {:id "2", :groupId 1, :addr "n2:7080", :leader true, :lastUpdate "1526517895"}, 4 {:id "4", :groupId 1, :addr "n4:7080"}}, :tablets {:amount_0 {:groupId 1, :predicate "amount_0", :force true}, :key_2 {:groupId 1, :predicate "key_2", :force true}, :type_2 {:groupId 1, :predicate "type_2", :force true}}}, 2 {:members {3 {:id "3", :groupId 2, :addr "n3:7080", :leader true, :lastUpdate "1526517898"}, 5 {:id "5", :groupId 2, :addr "n5:7080"}}, :tablets {:_predicate_ {:groupId 2, :predicate "_predicate_", :force true}, :amount_1 {:groupId 2, :predicate "amount_1", :force true}, :amount_2 {:groupId 2, :predicate "amount_2", :force true}, :key_0 {:groupId 2, :predicate "key_0", :force true}, :key_1 {:groupId 2, :predicate "key_1", :force true}, :type_0 {:groupId 2, :predicate "type_0", :force true}, :type_1 {:groupId 2, :predicate "type_1", :force true}}}}, :zeros {1 {:id "1", :addr "n1:5080"}, 2 {:id "2", :addr "n2:5080"}, 3 {:id "3", :addr "n3:5080", :leader true}, 4 {:id "4", :addr "n4:5080"}, 5 {:id "5", :addr "n5:5080"}}, :maxLeaseId "20000", :maxTxnTs "20000"}