Restore can complete but fail writing to etcd with request too large

[dgoodwin]

I created a backup for a specific namespace, and restored it into a rebuilt cluster. My particular cluster may contain a larger than normal amount of cluster objects. (cluster role bindings, etc) I saw quite a bit of throttling request messages in the log.
ark-restore.log

The restore completed, my namespace is restored (though failed with service account problems probably due to the new cluster, I will dig into that next), but fails to write status because etcd finds the request too large. My restore then is hung in progress.

I0929 12:56:11.805636       1 restore_controller.go:255] restore heptio-ark/test1-11-20170929125444 completed      
I0929 12:56:11.805661       1 restore_controller.go:258] updating restore heptio-ark/test1-11-20170929125444 final status                                                                                                              
I0929 12:56:11.991224       1 restore_controller.go:260] error updating restore heptio-ark/test1-11-20170929125444 final status: etcdserver: request is too large    

$ ark restore get
NAME                      BACKUP     STATUS       WARNINGS   ERRORS    CREATED                         SELECTOR
test1-11-20170929125444   test1-11   InProgress   0          0         2017-09-29 12:54:45 +0000 UTC   <none>

Environment was OpenShift 3.7 based on Kubernetes 1.7. Etcd 3.2.5.

[ncdc]

Can you grab the log for the restore itself (ark restore logs <restore name>)?

I'm thinking there are a couple of issues here:

1. We need to make the qps and burst configurable for the client the ark server uses
2. We may need to rethink how/where we store warnings and errors for a restore. I'm guessing that you had a ton of warnings and errors such that the overall Restore object was over the etcd raft message size limit (1.5MB).

I'm going to create a separate issue for 1 and we'll keep this to track 2

[dgoodwin]

Adding restore log.
restore.log

Definitely looks like some interplay here with reloading all that cluster data, in this case it was an entirely new cluster and things are not entirely happy.

[timothysc]

@dgoodwin are these really large lists?
If so, we should dbl-check where clayton's paging work is at.

[ncdc]

@timothysc the problem is we store the "lists" as structured data within restore.status

[ncdc]

I'm thinking of some ways around this:

1. Store only # of warnings & # of errors in restore.status
2. Users can download the restore log if they want to see warnings & errors (they'd have to sift through it), OR
3. We could store the structured data for warnings & errors in object storage and make it easy to combine that information with what's in etcd; e.g., ark restore get --details could grab the warnings & errors and display them.

[ncdc]

Here's what I have so far:

$ ark restore describe b2-20171101135335
Name:         b2-20171101135335
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Backup:  b2

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  true

Phase:  Completed

Validation errors:  <none>

Warnings:
  Ark:        <none>
  Cluster:    <none>
  Namespaces:
    default:      services "kubernetes" already exists
    heptio-ark:   services "minio" already exists
                  services "some-svc" already exists
    kube-system:  services "kube-dns" already exists

Errors:
  Ark:        <none>
  Cluster:    <none>
  Namespaces: <none>

WDYT?

[nrb]

@ncdc The format here looks good to me. I assume you're using option 3 of your previous comment for storing the data? That one seems like the best UX, as opposed to just throwing a potentially huge log at the user.

[dgoodwin]

Really nice functionality and output, how would this work around the etcd request size issue?

[ncdc]

The warnings and errors are stored in a gzipped file in object storage and retrieved on the fly

…

On Wed, Nov 1, 2017 at 2:08 PM Devan Goodwin ***@***.***> wrote: Really nice functionality and output, how would this work around the etcd request size issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYjE-weEDoS7m9le29VPgUDERoNm4ks5syLOSgaJpZM4Poqni> .

[dgoodwin]

Love it, seeing them broken out per ns/cluster/ark is fantastic. The describe subcommand threw me for a loop but forgot you can do whatever you want as it's not kubectl directly.

[ncdc]

@nrb yes this is option 3, although using a new describe subcommand.