Server crash & corruption with large mutations

[alexmilowski]

What version of Dgraph are you using?

1.0.4

What is the hardware spec (RAM, OS)?

Mac OS / Docker

Steps to reproduce the issue (command/config used to run Dgraph).

my setup

Running dgraph locally via docker:

zero:

docker run -it -p 5080:5080 -p 6080:6080 -p 8080:8080 -p 9080:9080 -p 8000:8000 -v ~/workspace/dgraph/data/:/dgraph --name dgraph-local dgraph/dgraph dgraph zero

one server:

docker exec -it dgraph-local dgraph server --memory_mb 16384 --zero localhost:5080

Generating Data

The following program generates mutations like:

{
set {
_:n1 <count> "1"^^<xs:int> .
_:n2 <count> "2"^^<xs:int> .
_:n3 <count> "3"^^<xs:int> .
_:n4 <count> "4"^^<xs:int> .
_:n5 <count> "5"^^<xs:int> .
_:n6 <count> "6"^^<xs:int> .
_:n7 <count> "7"^^<xs:int> .
_:n8 <count> "8"^^<xs:int> .
_:n9 <count> "9"^^<xs:int> .
_:n10 <count> "10"^^<xs:int> .
}
}

import sys

def blank_id(prefix='n'):
    n = 0
    while True:
       n += 1
       yield '_:' + prefix + str(n)

ids = blank_id()

print('{')
print('set {')
for i in range(1,1 + (int(sys.argv[1]) if len(sys.argv)>1 else 1000)):
   id = next(ids)
   print('{} <count> "{}"^^<xs:int> .'.format(id,i))
print('}')
print('}')

Procedure

1. Start with an empty database.

2. This will succeed:

   python generate.py 100 | curl -X POST -H "X-Dgraph-CommitNow: true" --data-binary @- -o /dev/null -v http://localhost:8080/mutate

3. This will fail and the server will crash:

   python generate.py 1000000 | curl -X POST -H "X-Dgraph-CommitNow: true" --data-binary @- -o /dev/null -v http://localhost:8080/mutate

4. Restart the server:

   docker exec -it dgraph-local dgraph server --memory_mb 16384 --zero localhost:5080

5. Now the simple mutation will not work:

   python generate.py 100 | curl -X POST -H "X-Dgraph-CommitNow: true" --data-binary @- -o /dev/null -v http://localhost:8080/mutate

Expected behaviour and actual result.

The server should gracefully fail if the mutation is too large. Instead, the server process crashes and the database is left in some unrecoverable state.

[janardhan1993]

@alexmilowski Can you please share the error you were getting and the database logs before crash and after restart.

[pawanrawal]

Seems like a docker issue on Mac, related to #1989.

[alexmilowski]

From the discussion thread:

I tried increasing the memory and changed the way I run the server via docker. That seems to have helped but I still believe there is some kind of OOM problem. I have tested the server and it seems top have recovered from the crash when I restart the container now so the “corruption” may have been more of an issue of the way I ran it via docker and tried to restart.

I changed my bulk loader process to allow me to resume and I was able to restart the crashed docker container for the server and finish the import. With a small batch size for each transaction, that makes me wonder if memory is not be reclaimed fast enough. I hit the server in rapid succession as each chunk is ready.

Running the zero, server, and ratel as follows seemed to help my local testing:

Zero:

docker run -it -p 5080:5080 -p 6080:6080 -v ~/workspace/dgraph/data/zero:/dgraph --name dgraph-zero dgraph/dgraph dgraph zero --my=docker.for.mac.localhost:5080

Server:

docker run -it -p 7080:7080 -p 8080:8080 -v ~/workspace/dgraph/data/server-1:/dgraph --name dgraph-server-1 dgraph/dgraph dgraph server --memory_mb 4096 --zero docker.for.mac.localhost:5080

Ratel:

docker run -it -p 8000:8000 --name dgraph-ratel dgraph/dgraph dgraph-ratel

I think the main issue here is low memory. I don't expect the server to crash in a low memory condition but this may be hard to determine given the interaction between Docker and the container and possible poor behavior by Docker on the Mac.

I am going to move this whole test to a more substantial machine running much more like it would in production and test again. As you have mentioned, could be a docker/Mac issue.

[pawanrawal]

Also, I would recommend memory_mb to something like 1/3 - 1/2 of the total RAM. Closing this issue, feel free to reopen if you see any corruption issues with the database.

[alexmilowski]

I don't understand why you closed this issue. The server container still just gets randomly terminated. We have no data that indicates this is a docker issue versus dgraph (although, I suspect Docker on the Mac is the cause). This is rather basic for testing dgraph in a Mac development environment.

[pawanrawal]

Can you please check the reason for why the container exited? You can run docker inspect $container_id to do so. Please share the State object from the result of the command.

I think this is a docker issue because if dgraph were to crash you would see some logs in the server mentioning the reason.