Velero restore with include-resources to restore only namespace resource is not working

[Frank51]

What steps did you take and what happened:
Velero user could use '--include-resources' flag to specify which kind of resources need to be restored from backup. But when '--include-resources' only contains 'namespaces', the backup namespace could not be restored successfully to the Kubernetes cluster. If adding 'namespaces' and 'services' into the 'include resources', the backup could be restored correctly at this time.

velero get backups
NAME        STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
yelb-back   Completed   2019-09-20 12:39:34 -0700 PDT   3d        default            <none>

velero describe backup yelb-back
Name:         yelb-back
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  yelb-ns
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2019-09-20 12:39:34 -0700 PDT
Completed:  2019-09-20 12:39:35 -0700 PDT

Expiration:  2019-10-20 12:39:34 -0700 PDT

Persistent Volumes: <none included>


kubectl get ns
NAME              STATUS   AGE
default           Active   5h6m
kube-node-lease   Active   5h6m
kube-public       Active   5h6m
kube-system       Active   5h6m
velero            Active   152m

velero restore create --from-backup yelb-back --include-resources namespaces
Restore request "yelb-back-20191016140424" submitted successfully.
Run `velero restore describe yelb-back-20191016140424` or `velero restore logs yelb-back-20191016140424` for more details.

kubectl get ns
NAME              STATUS   AGE
default           Active   5h7m
kube-node-lease   Active   5h7m
kube-public       Active   5h7m
kube-system       Active   5h7m
velero            Active   152m

# yelb-ns is not restored from the backup.
# However, when I restore both the namespace and service, yelb-ns is restored:

velero restore create --from-backup yelb-back --include-resources namespaces,services
Restore request "yelb-back-20191016140932" submitted successfully.
Run `velero restore describe yelb-back-20191016140932` or `velero restore logs yelb-back-20191016140932` for more details.

kubectl get ns
NAME              STATUS   AGE
default           Active   5h12m
kube-node-lease   Active   5h12m
kube-public       Active   5h12m
kube-system       Active   5h12m
velero            Active   157m
yelb-ns           Active   4s

What did you expect to happen:
When running 'velero restore create --from-backup --include-resources namespaces ', velero should restore only the namespace resource from the backup on the Kubernetes cluster successfully.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
  https://gist.github.com/Frank51/5c18363b55bb593fc780068134684da7
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

Name:         yelb-back
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  yelb-ns
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2019-09-20 12:39:34 -0700 PDT
Completed:  2019-09-20 12:39:35 -0700 PDT

Expiration:  2019-10-20 12:39:34 -0700 PDT

Persistent Volumes: <none included>

• velero backup logs <backupname>
  https://gist.github.com/Frank51/62aaa98ea6a38418eb159b926e101e17
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml

Name:         yelb-back-20191016140424
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Backup:  yelb-back

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        namespaces
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

• velero restore logs <restorename>
  https://gist.github.com/Frank51/3e2bbdc44a5c1f1cdee91e0d65d70dd4
  Anything else you would like to add:
  [Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version): v1.0.0
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): (Client Version : v1.16.0), (Server Version :v1.14.5 )
• Kubernetes installer & version:
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Ubuntu

[skriss]

Yeah, this is likely being caused by https://github.com/vmware-tanzu/velero/blob/master/pkg/restore/restore.go#L403-L407. This will take a little bit of investigation as I don't remember exactly why we treated namespaces that way, though likely has to do with included/excluded namespaces, namespace mappings, etc.

[Frank51]

Yeah, this is likely being caused by https://github.com/vmware-tanzu/velero/blob/master/pkg/restore/restore.go#L403-L407. This will take a little bit of investigation as I don't remember exactly why we treated namespaces that way, though likely has to do with included/excluded namespaces, namespace mappings, etc.

Hi, Kriss. Could you give me more detail related to this namespace restore issue? Because I am a little bit confused about that part of the code. Why the namespace will be treated in this way? But when restoring namespace with other resources like PV, PVC, it works.

[skriss]

@Frank51 sorry, I remember you asked me for this before in Slack :) I'll try to add more context tomorrow - I'll have to jog my own memory as to why it's implemented this way.

[Frank51]

@Frank51 sorry, I remember you asked me for this before in Slack :) I'll try to add more context tomorrow - I'll have to jog my own memory as to why it's implemented this way.

Hi, Kriss. Sorry to disturb you. Do you have any idea about the implementation of the restore of namespace at that time?

[skriss]

So I think there are a couple of things to consider:

• if we're restoring a namespaced resource, we first need to ensure that the namespace itself exists, so the resource can be restored into it
• when restoring Namespace resources themselves, we need to consider the namespace includes/excludes, as well as the namespace mappings

I believe that the implementation approach of lazily restoring Namespace resources only if we are restoring a resource into it was the easiest way to deal with the above two items. That said, alternate approaches would work fine as long as all of the permutations of the above are properly dealt with.

Hope that helps!

[skriss]

@Frank51 were you interested in working on a fix here?

[Frank51]

@Frank51 were you interested in working on a fix here?

Sure, I am quite interested in it.

[ashish-amarnath]

/assign @Frank51

[nrb]

This also has implications for users wanting to write RestoreItemActions for namespaces. Right now, since namespaces aren't created as part of the normal restore process but rather directly with a client call, writing a RestoreItemAction plugin for them will not get invoked.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing the stale issue.

[reasonerjt]

not a stale

[sseago]

@reasonerjt I wonder whether we want a restore field to control this behavior. Currently, velero doesn't restore empty namespaces. With the change proposed here, empty namespaces would be restored. Since that's a behavior change, would we want to give users an option to preserve prior behavior, or should we consider "velero doesn't restore an empty namespace" to be a bug instead, in which case, we wouldn't want a field, we'd just make the change for all restores.

[reasonerjt]

We'll double-check to determine if the current behavior "velero doesn't restore an empty namespace" is a bug or not, and may leave it as-is.

[reasonerjt]

@reasonerjt I wonder whether we want a restore field to control this behavior. Currently, velero doesn't restore empty namespaces. With the change proposed here, empty namespaces would be restored. Since that's a behavior change, would we want to give users an option to preserve prior behavior, or should we consider "velero doesn't restore an empty namespace" to be a bug instead, in which case, we wouldn't want a field, we'd just make the change for all restores.

@sseago
IMO "velero doesn't restore an empty namespace" is a bug, it was not documented anywhere