Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix for a bug #2226 #2231

Merged
merged 8 commits into from Feb 3, 2020
Merged

fix for a bug #2226 #2231

merged 8 commits into from Feb 3, 2020

Conversation

koehn
Copy link
Contributor

@koehn koehn commented Jan 30, 2020

This fixes a race condition that happens when velero looks at restic's stdout before it has output any lines. The bug (an array index out of bounds) causes velero to crash, which takes out the entire container including restic, which causes restic to fail to run its backup.

The fix checks for null or empty strings and handles them correctly. There are tests to make sure.

closes #2226

Signed-off-by: Brad Koehn <brad@koehn.com>
@koehn koehn changed the title fix for a bug #2226 I think fix for a bug #2226 Jan 30, 2020
Copy link
Contributor

@ashish-amarnath ashish-amarnath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick PR!
I made a couple of minor comments.
Considering you are able to get a consistent repro of this, would you be willing to run a test image of Velero with this fix in your environment to confirm that the problem is fully fixed?

Signed-off-by: Brad Koehn <brad@koehn.com>
@koehn
Copy link
Contributor Author

koehn commented Jan 31, 2020

Considering you are able to get a consistent repro of this, would you be willing to run a test image of Velero with this fix in your environment to confirm that the problem is fully fixed?

I made some of the changes you mentioned and commented on the others.

I'm happy to run it for testing; how do I build an image? make all doesn't seem to do it.

@koehn
Copy link
Contributor Author

koehn commented Jan 31, 2020

OK, it looks like docker build -f Dockerfile-velero _output will build a workable image.

@ashish-amarnath
Copy link
Contributor

Building a container image is not very friendly atm. Once you've pushed all your changes I can build a container and share it with you. That may be faster. lmk..

@ashish-amarnath
Copy link
Contributor

@koehn this PR #2233 should make it easy to build container images.

Brad Koehn added 2 commits January 30, 2020 20:47
…f statement

Signed-off-by: Brad Koehn <brad@koehn.com>
Signed-off-by: Brad Koehn <brad@koehn.com>
This reverts commit 39a5751.

Signed-off-by: Brad Koehn <brad@koehn.com>
Signed-off-by: Brad Koehn <brad@koehn.com>
Copy link
Contributor

@ashish-amarnath ashish-amarnath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Let me get you a container image with these changes that you can run :)

Brad Koehn added 2 commits January 30, 2020 21:10
Signed-off-by: Brad Koehn <brad@koehn.com>
Signed-off-by: Brad Koehn <brad@koehn.com>
@ashish-amarnath
Copy link
Contributor

@koehn you should be able to pull this image to test.

docker pull quay.io/ashish_amarnath/velero:koehn-pr-2231

If you are unable to pull that image for any reason, you can make a local change similar to #2233 and build your container image like
E.g.:

IMAGE=quay.io/ashish_amarnath/velero VERSION=koehn-pr-2231 make container

IMAGE will be your image repository and
VERSION will be your tag for that image.

Then you can use docker push to push it to your remote image repository

@koehn
Copy link
Contributor Author

koehn commented Jan 31, 2020

One more fix I uncovered in testing.

@koehn
Copy link
Contributor Author

koehn commented Jan 31, 2020

Thanks I was able to build and test it myself. It works fine; just added a change to avoid decoding a line w/o and JSON and logging an error.

@ashish-amarnath
Copy link
Contributor

LGTM so far. I'll take a closer look tomorrow!

Copy link
Contributor

@ashish-amarnath ashish-amarnath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGMT.
Please let us know if you would like me to build you a container image for testing!

Copy link
Member

@skriss skriss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two minor comments mainly for your own edification, not merge-blocking. LGTM. Thanks for the PR!

TotalBytes: stat.TotalBytes,
BytesDone: stat.BytesDone,
})
if len(lastLine) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd be more consistent with go style to invert this, so e.g.

if len(lastLine) == 0 {
    break
}

...

so the main logic can be less indented. this is not a merge-blocking comment, though.

@@ -153,6 +155,9 @@ func decodeBackupStatusLine(lastLine []byte) (backupStatusLine, error) {
// have a newline at the end of it, so this returns the substring between the
// last two newlines.
func getLastLine(b []byte) []byte {
if b == nil || len(b) == 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should be able to just do if len(b) == 0 { since a nil slice also has a length of zero. Not blocking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Some Restic Backups timeout
3 participants