Encryption-at-rest cannot be enabled after starting Alpha without encryption

[danielmai]

What version of Dgraph are you using?

v20.03.1

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Ubuntu Linux.

Steps to reproduce the issue (command/config used to run Dgraph).

1. Generate an encryption key: dd if=/dev/random bs=1 count=32 of=enc_key_file
2. Run Dgraph Zero: dgraph zero
3. Run Dgraph Alpha without encryption: dgraph alpha --lru_mb=1024
4. Stop Dgraph Alpha
5. Restart Dgraph Alpha with the encryption key: dgraph alpha --lru_mb=1024 --encryption_key_file ./enc_key_file

Expected behaviour and actual result.

I expect that Alpha should start successfully with the encryption key file and start to encrypt new data. This comes from the encryption blog post:

If you have an existing Badger datastore that is not encrypted, enabling encryption on it will not immediately encrypt your existing data all at once. Instead, only new files are encrypted, which will happen as new data is added. As older data gets compacted and newer files generated, those would also get encrypted over time. Badger can run in this hybrid mode easily, as each SSTable and value log file stores the information about the data key used for encryption.

Instead, Alpha cannot start with encryption enabled if it has already started previously without encryption. Doing so results in this panic:

2020/04/29 19:54:56 Encryption key mismatch
github.com/dgraph-io/badger/v2.init
	/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200421062606-cddf7c03451c/errors.go:115
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5414
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5409
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5409
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5409
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5409
runtime.main
	/usr/local/go/src/runtime/proc.go:190
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373
Error while creating badger KV WAL store
github.com/dgraph-io/dgraph/x.Checkf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:145
github.com/dgraph-io/dgraph/worker.InitServerState
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:57
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:613
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:90
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

[danielmai]

Closing this because encryption has never been able to be turned on with existing unencrypted Alpha. Encryption needs to be enabled from the beginning.

The blog post and docs need to be updated.