feat(Restore): Retry restore proposals.

[martinmr]

This change adds retries when sending proposals to each group to start a restore for
temporary and/or recoverable errors.

Fixes DGRAPH-1232

---

This change is 

Docs Preview:

[parasssh]

Reviewable status: 0 of 1 files reviewed, 3 unresolved discussions (waiting on @manishrjain, @martinmr, and @vvbalaji-dgraph)

a discussion (no related file):
minor comments. Feel free to resolve.

---

---

worker/online_restore_ee.go, line 78 at r1 (raw file):

		go func() {
			errCh <- retryRestoreProposal(ctx, reqCopy)

retryRestoreProposal is confusing. Maybe just rename to tryRestoreProposal()

---

worker/online_restore_ee.go, line 114 at r1 (raw file):

		return true
	case strings.Contains(err.Error(), errRestoreProposal):
		// Do not try to recover from errors when sending the proposal.

Can proposal failure also be recoverable?

[martinmr]

Reviewable status: 0 of 1 files reviewed, 3 unresolved discussions (waiting on @manishrjain, @martinmr, @parasssh, and @vvbalaji-dgraph)

---

worker/online_restore_ee.go, line 78 at r1 (raw file):

Previously, parasssh wrote…

retryRestoreProposal is confusing. Maybe just rename to tryRestoreProposal()

Done.

---

worker/online_restore_ee.go, line 114 at r1 (raw file):

Previously, parasssh wrote…

Can proposal failure also be recoverable?

I checked and the only error that seems recoverable is if raft has not been initialized yet. I added a condition to check for this error.