Problem when change datatype and index of predicate with huge data

[thanhphi0401]

If you suspect this could be a bug, follow the template.

• What version of Dgraph are you using?
  v1.0.14

• Have you tried reproducing the issue with latest release?
  Yes

• What is the hardware spec (RAM, OS)?
  Linux Ubuntu 18.04 , 31GiB

• Steps to reproduce the issue (command/config used to run Dgraph).

1. Go to dgraph-ui page
2. Change type predicate from string to float and make index
   
3. The Dgraph server freeze some minutes. I had restart Dgraph server

systemctl restart dgraph dgraph-ui.service dgraph-zero.service

4. Go back ui page (port 8000) , make query and not response. The error: context deadline exceeded (see image)

• Detail:

I have Dgraph Database with huge data (over 10 million nodes). After i change price predicate from string to float (current with over 5 million nodes has price data) and index it, Dgraph server has been freeze and i can’t query or mutate. This is the error:

It seem to be hang forever and can’t excute any query.
I have restart Dgraph service, but still got the problem.
What happen, is it the bug on Dgraph when run with huge data?

[martinmr]

This is not a bug. Unfortunately, in order to keep all our consistency guarantees, Dgraph blocks while rebuilding an index. So adding an index to a predicate with lots of data might result in Dgraph being unavailable for a while. Restarting the server doesn't help because the index rebuild process starts right after restart. It's better to just let it complete.

Two strategies to reduce the impact:

1. Think though your schema and desired queries thoroughly before you start adding data. This minimizes the need to change the indices/schema.
2. If the need arises, perform the change at some point with low traffic (e.g at night).

Unfortunately this is the current state of affairs. Once version 1.1 is released we can start looking into improving this process.

[d4l3k]

This is a really unfortunate state to be in. There should be some way to cancel building the index since it's can be hideously slow. I left it running for 10 hours without it finishing and doesn't seem to be any way to view the progress.

From a recent blog post "One of the cornerstones of Dgraph is that it allows a flexible schema, which can be modified in a live system, without any downtime. This involves changing data types and adding or removing indices with a single ALTER command to match the needs of an application developer." https://blog.dgraph.io/post/indexing-in-dgraph/

This post states that there is no downtime which according to this issue is not the case. Ideally this would happen in the background like how CockroachDB does it and only use the index once it's fully built. CockroachDB also provides info on status of the index build. https://www.cockroachlabs.com/docs/stable/online-schema-changes.html

[manishrjain]

Yes. We should be fixing this problem. Allow queries to use the index once it's built and also allow indices to be built using disk, instead of entirely in memory. Feel free to file an issue to track that. We'll get to it with priority after v1.1 release. CC: @campoy .

For the time being, if you use best effort queries, you'd still be able to get the queries to work.

[manishrjain]

Opening this issue as a placeholder. Close this issue once the other issue is filed.

[d4l3k]

For anyone else running into this issue, here's the hacky solution to cancelling an index build.

Applied these changes to v1.0.16: d4l3k@6a11b74

Launched alpha with these changes, removed the index from schema and then launched from stable again.

[d4l3k]

For the time being, if you use best effort queries, you'd still be able to get the queries to work.

Reading still works with this but it was blocking me from adding new data.