Failed to take backup when there were about 500 secrets in local cluster

[sowmyav27]

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

• Create about 500 secrets in local cluster.
• create a backup CR
• Backup fails with error
• Backup CR error:

status:
  conditions:
  - lastUpdateTime: "2020-08-29T22:34:25Z"
    message: the server was unable to return a response in the time allotted, but
      may still be processing the request
    reason: Error
    status: "False"
    type: Reconciling

• backup-restore operator logs:

INFO[2020/08/29 22:33:25] Processing backup b2                         
INFO[2020/08/29 22:33:25] For backup CR b2, filename: default-b2-28e36ff0-c48a-41fd-9d71-5b760c5748e7-2020-08-29T22#33#25Z 
INFO[2020/08/29 22:33:25] Temporary backup path for storing all contents for backup CR b2 is /tmp/default-b2-28e36ff0-c48a-41fd-9d71-5b760c5748e7-2020-08-29T22#33#25Z680481891 
INFO[2020/08/29 22:33:25] Using resourceSet ecm-resource-set for gathering resources for backup CR b2 
INFO[2020/08/29 22:33:25] Gathering resources for backup CR b2         
INFO[2020/08/29 22:33:25] Gathering resources for groupVersion: v1     
INFO[2020/08/29 22:33:25] resource kind namespaces, matched regex ^namespaces$ 
INFO[2020/08/29 22:33:25] Gathering resources for groupVersion: v1     
INFO[2020/08/29 22:33:25] resource kind secrets, matched regex ^Secret$|^serviceaccounts$ 
INFO[2020/08/29 22:33:25] resource kind serviceaccounts, matched regex ^Secret$|^serviceaccounts$ 
ERRO[2020/08/29 22:34:25] error syncing 'default/b2': handler backups: the server was unable to return a response in the time allotted, but may still be processing the request, requeuing 
INFO[2020/08/29 22:34:25] Processing backup b2                         

Expected Result:
Backup should happen with NO error.

Other details that may be helpful:
After this the rancher server also crashed.

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): master-head - commit id: 8d9cedde9
• Installation option (single install/HA): HA

[mrajashree]

I had slightly more than 500 secrets in a cluster and I was able to do a backup of them. Is this always reproducible on your setup?

[sowmyav27]

@mrajashree I think these secrets had big values.

[mrajashree]

With the fix, we make multiple calls to the apiserver, so we are paginating the list call response to get a certain number (currently 200) of resources per call. So the call may seem to take time but it shouldn't timeout

[sowmyav27]

Verified on master-head with backup restore operator tag: rc7

• Create about 500 secrets in local cluster.
• create a backup CR (resource modified to included all namespaces/resources created ^^^)
• Bakcup has been created successfully.
• Backup is available in S3.
• User is able to restore from this backup successfully.