Data loss in case of miss configuration

[hubyhuby]

When configuring dgraph alpha :

1. you are allowed to set --lru_mb higher than actual RAM on the system.
2. Doing so will crash the instance AND DELETE ALL THE DATAS. It deletes the directories and the datas of degraph.
   We had the bug twice with deleting of the folders and data, before we understood that.
   Note that our Ubuntu system stays stable. With lru to 1024 we have been stable for 5 days.

What version of Dgraph are you using?

1.1.1

What is the hardware spec (RAM, OS)?

Scaleway DEV instance 2GB RAM . Ubuntu 18.04

Steps to reproduce the issue (command/config used to run Dgraph).

1 ) Install Dgraph as a service following this tutorial : https://computingforgeeks.com/install-dgraph-on-centos-ubuntu/
2) Wait for some time : 24hours in our tests
3) Check the services : dgraph alpha is down + its folder are now deleted !

Expected behaviour and actual result.

-It shouldn t be possible to set the memory direcive higher than the system memory.

• In case of a crash it shouldn t delete all my dgraph data and folders.

[danielmai]

Can you share the Dgraph Zero and Dgraph Alpha logs? What's happening in the step "wait for some time: 24 hours in our tests"?

[hubyhuby]

Hi @danielmai ,
Thanks for coming back to me.
During the 24H nothing impressive was done ;) We impute max 50 nodes... Sending the basic "hello word" of degraph to learn how it works.

Strangely I get no logs since the first setup of Dgraph as you can see in the log folder....
( here are the perms for the log folder drwxr-xr-x 2 dgraph dgraph 4096 Dec 27 10:36 dgraph/ )

Here is the current Log of the now WORKING configuration :

root@scw-wonderful-mcnulty:/var/log/dgraph# ll
total 8
drwxr-xr-x  2 dgraph dgraph 4096 Dec 27 10:36 ./
drwxr-xr-x 13 root   root   4096 Dec 27 15:32 ../
root@scw-wonderful-mcnulty:/var/log/dgraph# systemctl status dgraph dgraph-zero dgraph-ui
● dgraph.service - dgraph.io data server
   Loaded: loaded (/etc/systemd/system/dgraph.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 16:23:46 CET; 4 days ago
 Main PID: 15127 (dgraph)
    Tasks: 25 (limit: 2288)
   CGroup: /system.slice/dgraph.service
           └─15127 /usr/local/bin/dgraph alpha --lru_mb 1024 -p /var/run/dgraph/p -w /var/run/dgraph/w

Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.804502   15127 log.go:34] DropPrefix done
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.804554   15127 log.go:34] Resuming writes
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.804583   15127 log.go:34] Writes flushed. Stopping compactions now...
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834266   15127 log.go:34] LOG Compact 1->1, del 1 tables, add 1 tables, took 1.506656ms
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834299   15127 log.go:34] DropPrefix done
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834310   15127 log.go:34] Resuming writes
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834322   15127 index.go:890] Rebuilding reverse index for doc
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834499   15127 log.go:34] Rebuilding index for predicate doc: Sent 0 keys
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834572   15127 mutation.go:171] Done schema update predicate:"doc" value_type:UID directive:REVERSE
Jan 13 16:04:48 scw-wonderful-mcnulty dgraph[15127]: I0113 16:04:48.834699   15127 server.go:211] ALTER op: schema:"<doc>: uid @reverse ."  done

● dgraph-zero.service - dgraph.io zero server
   Loaded: loaded (/etc/systemd/system/dgraph-zero.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 16:20:56 CET; 4 days ago
 Main PID: 14922 (dgraph)
    Tasks: 29 (limit: 2288)
   CGroup: /system.slice/dgraph-zero.service
           └─14922 /usr/local/bin/dgraph zero --wal /var/run/dgraph/zw

Jan 09 16:20:59 scw-wonderful-mcnulty dgraph[14922]: I0109 16:20:59.673629   14922 log.go:34] 1 received MsgVoteResp from 1 at term 3
Jan 09 16:20:59 scw-wonderful-mcnulty dgraph[14922]: I0109 16:20:59.673690   14922 log.go:34] 1 became leader at term 3
Jan 09 16:20:59 scw-wonderful-mcnulty dgraph[14922]: I0109 16:20:59.673705   14922 log.go:34] raft.node: 1 elected leader 1 at term 3
Jan 09 16:20:59 scw-wonderful-mcnulty dgraph[14922]: I0109 16:20:59.673761   14922 raft.go:663] I've become the leader, updating leases.
Jan 09 16:20:59 scw-wonderful-mcnulty dgraph[14922]: I0109 16:20:59.673772   14922 assign.go:41] Updated Lease id: 1. Txn Ts: 10001
Jan 09 16:21:00 scw-wonderful-mcnulty dgraph[14922]: W0109 16:21:00.173691   14922 node.go:669] [0x1] Read index context timed out
Jan 09 16:23:46 scw-wonderful-mcnulty dgraph[14922]: I0109 16:23:46.978047   14922 zero.go:415] Got connection request: cluster_info_only:true
Jan 09 16:23:46 scw-wonderful-mcnulty dgraph[14922]: I0109 16:23:46.978629   14922 zero.go:433] Connected: cluster_info_only:true
Jan 09 16:23:46 scw-wonderful-mcnulty dgraph[14922]: I0109 16:23:46.981083   14922 zero.go:415] Got connection request: id:1 addr:"localhost:7080"
Jan 09 16:23:46 scw-wonderful-mcnulty dgraph[14922]: I0109 16:23:46.981324   14922 zero.go:542] Connected: id:1 addr:"localhost:7080"

● dgraph-ui.service - dgraph.io UI server
   Loaded: loaded (/etc/systemd/system/dgraph-ui.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-09 15:08:53 CET; 4 days ago
 Main PID: 4585 (dgraph-ratel)
    Tasks: 6 (limit: 2288)
   CGroup: /system.slice/dgraph-ui.service
           └─4585 /usr/local/bin/dgraph-ratel

[hubyhuby]

Some more infos .
After the crash those folders including /var/run/dgraph it self are deleted.
This was quite unbelievable at first... I thought it might be the server instance itself ... But now that dgraph has the proper memory setup it didn t happen again ...
IMHO it is worth notifying, since we need to be really confident in our data storage.

root@scw-wonderful-mcnulty:/var/run/dgraph# ll
total 0
drwxr-xr-x 5 dgraph dgraph 100 Jan 9 15:01 ./
drwxr-xr-x 27 root root 1040 Jan 14 08:47 ../
drwxr-xr-x 2 dgraph dgraph 140 Jan 13 16:04 p/
drwxr-xr-x 2 dgraph dgraph 140 Jan 9 16:23 w/
drwxr-xr-x 2 dgraph dgraph 160 Jan 9 16:20 zw/

---

root@scw-wonderful-mcnulty:~# cd /var/log/dgraph/
root@scw-wonderful-mcnulty:/var/log/dgraph# ll
total 8
drwxr-xr-x 2 dgraph dgraph 4096 Dec 27 10:36 ./
drwxr-xr-x 13 root root 4096 Dec 27 15:32 ../

---

/var/log# ll
total 48396
drwxr-xr-x 13 root root 4096 Dec 27 15:32 ./
drwxr-xr-x 14 root root 4096 Dec 25 18:55 ../
-rw-r--r-- 1 root root 29107 Jan 7 15:32 alternatives.log
drwxr-xr-x 2 root root 4096 Jan 14 06:55 apt/
-rw-rw---- 1 root utmp 48183936 Jan 14 09:40 btmp
-rw-r--r-- 1 root adm 353119 Jan 9 14:21 cloud-init.log
-rw-r--r-- 1 root root 26064 Jan 9 14:21 cloud-init-output.log
drwxr-xr-x 2 dgraph dgraph 4096 Dec 27 10:36 dgraph/

[hubyhuby]

Working environment.
For info , no idea if that help but Alpha gets over the "lru" directive :

[MichelDiz]

It doesn't make much sense that this is a problem with Dgraph. LRU cache is currently disabled. As far as I know, no matter what value you put into it, there are no effects. We're waiting for Ristretto to come in.

This has more to do with the way Dgraph is being used on Linux than something in Dgraph itself.

For example, logs are handled by the OS. If they are gone, it has nothing to do with Dgraph. Because Dgraph does not deals with logs, just generate it.

If you can reproduce this without following the tutorial that uses systemctl/d would be helpful.

[MichelDiz]

One update on this:

I'm testing this since yesterday. I noticed after doing a restart that the paths /var/run/dgraph/{p,w,zw} were gone. But this was without doing anything else. Just a restart.

After that, I thought it was a mistake from my part. Maybe I forgot to make those directories. Or something.

Today I'm redoing the tests. I'll let the test running for 24 hours. And check it again. And then I'll do a restart to check if the directories still there.

This time, I'm increasing the load per second - it is sending two nodes in a single transaction every 1 millisecond - Which means 1000 calls for second and 2000 nodes created per second. In total it will create 172.800.000 nodes in 24 hours.

The VM has 2GB of RAM. And I set up the lru_mb to 4096. It is running for one hour. And the logs seem fine. Let's see it until tomorrow.

[hubyhuby]

Hi @MichelDiz ,
A colleague had a 3rd reset of the database this week (~5 days working ok) . I will debreif with him Monday.
I am very puzzeled aswell on what is exaclty the cause; but I think it is the memory constrain that is a edge case.
It is quite OK that Dgraph may go down with little memory (also other services nginx, postgres ...doesn t) , what is not Ok is that it deletes all the datas.
What we notice is that using a Golang visual code plugin install on the server a service that consumes extra memory. So the server instance starts to swap before crashing I guess.

[MichelDiz]

hey @hubyhuby, we think your case about the data loss. It is due to the fact that the steps used to configure the systemd service is using a temporary path. Which will be erased every reboot.

@mangalaman93 clarified to me that:
/dev type devtmpfs
/run is a tmpfs
/var/run is symlink to /run

So, all paths used in the tutorial will be erased every reboot. We are working on a fix for this. Until there, my VM still doing great xD By the way, I already forced a reboot while he was running just to see if it brokes. It is firm and strong. And keeping rolling.

Cheers.

Update:

/var/run/ contains various temporary system info which changes after a reboot. For instance, a list of the active PIDs and system services is stored in here. Such data will be different the next time the system reboots. Source: https://www.linux.org/threads/temporary-files.10817/

[danielmai]

@MichelDiz has merged #4610 to fix the systemd configuration to write to persistent filesystem paths (/var/lib) instead of writing them to a temporary filesystem path (/var/run). Please use this latest configuration for systemd.

[hubyhuby]

Will try it out with new config and report if any issue.
Could you delete or update this google indexed page , which id now outdated ,
https://computingforgeeks.com/install-dgraph-on-centos-ubuntu/

[sleto-it]

@hubyhuby thanks

We have created PRs to update our repositories. Some have been merged, others will be soon
We do not own that tutorial, but we have contacted the author - we will follow-up with him again

Thanks again,