Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup restore for Event Bus fails #5942

Closed
sayanh opened this issue Oct 9, 2019 · 5 comments · Fixed by kyma-project/test-infra#1703

Comments

@sayanh
Copy link
Contributor

@sayanh sayanh commented Oct 9, 2019

Description

Backup restore for Event Bus fails here:
https://storage.googleapis.com/kyma-prow-logs/logs/kyma-gke-backup-nightly/1181796650318827522/build-log.txt

Expected result

Actual result

Steps to reproduce

Troubleshooting

@sayanh sayanh added the area/eventing label Oct 9, 2019
@sayanh sayanh added this to the Backlog_SkydivingTunas milestone Oct 9, 2019
@suleymanakbas91

This comment has been minimized.

Copy link
Member

@suleymanakbas91 suleymanakbas91 commented Oct 10, 2019

Event bus test has been disabled from the end-to-end backup scenario until it is fixed.

@k15r

This comment has been minimized.

Copy link
Contributor

@k15r k15r commented Oct 17, 2019

The problem:

During restore the owner references on channels created by nats-channels do not get updated to the correct values, but these owner references just get removed.

k get service -n kyma-system -l messaging.knative.dev/role=natss-channel

options for the fix:

  • create a velero plugin that adds valid owner references on the restored services
  • don't backup these services at all, as the controller creates them if they cannot be found (maybe using velero.io/exclude-from-backup)
  • use hooks to delete the service after restore check here
@montaro

This comment has been minimized.

Copy link
Contributor

@montaro montaro commented Oct 18, 2019

Option # 2 The velero.io/exclude-from-backup labeling option is not supported in the version we use of Velero 1.0.0

@montaro

This comment has been minimized.

Copy link
Contributor

@montaro montaro commented Oct 24, 2019

Option # 3 doesn't look like a valid option because:

  1. The hooks are for backup only, and we need a post.restore hook to delete the service after creation.
  2. This requires a living a kubectl container to execute the commands
@montaro

This comment has been minimized.

Copy link
Contributor

@montaro montaro commented Oct 24, 2019

Option # 1
I created a plugin to search for the corresponding NATS Channel and set the Owner Ref of the Service with the right value.
The problem is that backup tests never finish because Velero fails to restore for unknow reason yet.
Example: https://status.build.kyma-project.io/log?id=1187302077185921024&job=pre-master-kyma-gke-backup

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
You can’t perform that action at this time.