New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: store new item in finalized indexB #1251

Closed
darkskiez opened this Issue Sep 17, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@darkskiez

darkskiez commented Sep 17, 2017

Output of restic version

restic 0.7.2 (v0.7.2-0-g5b1e4df1)
compiled with go1.8.3 on linux/amd64

How did you run restic exactly?

restic -r /home/restic backup /mnt/media

What backend/server/service did you use?

localfile

Expected behavior

not to panic

Actual behavior

scanned 270 directories, 7193 files in 0:00
panic: store new item in finalized indexB / 1.341 TiB  2719 / 7463 items  0 errors  ETA 24:32 

goroutine 1266676 [running]:
github.com/restic/restic/internal/repository.(*Index).Store(0xc43b47e1c0, 0x1, 0x137244, 0xed5be426c2b4400a, 0xf0d573c30a46e61a, 0xd8395f58b032a2a0, 0x524e794418239f8c, 0x2d4c74, 0x221ad52c4ab6c05d, 0xf2342afbc820faf8, ...)
	src/github.com/restic/restic/internal/repository/index.go:103 +0x441
github.com/restic/restic/internal/repository.(*MasterIndex).Store(0xc4201cc720, 0x1, 0x137244, 0xed5be426c2b4400a, 0xf0d573c30a46e61a, 0xd8395f58b032a2a0, 0x524e794418239f8c, 0x2d4c74, 0x221ad52c4ab6c05d, 0xf2342afbc820faf8, ...)
	src/github.com/restic/restic/internal/repository/master_index.go:127 +0x2a2
github.com/restic/restic/internal/repository.(*Repository).savePacker(0xc420075680, 0xc462e90240, 0xd573c30a46e61aed, 0x395f58b032a2a0f0)
	src/github.com/restic/restic/internal/repository/packer_manager.go:151 +0x778
github.com/restic/restic/internal/repository.(*Repository).SaveAndEncrypt(0xc420075680, 0xdc8e00, 0xc4200183f8, 0x6d859b01, 0xc49420a000, 0x137224, 0x4ae000, 0xc46d859d10, 0x0, 0x0, ...)
	src/github.com/restic/restic/internal/repository/repository.go:203 +0x34e
github.com/restic/restic/internal/repository.(*Repository).SaveBlob(0xc420075680, 0xdc8e00, 0xc4200183f8, 0xd8395f58b032a201, 0xc49420a000, 0x137224, 0x4ae000, 0xed5be426c2b4400a, 0xf0d573c30a46e61a, 0xd8395f58b032a2a0, ...)
	src/github.com/restic/restic/internal/repository/repository.go:529 +0x18b
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc46d82a230, 0xdc8e00, 0xc4200183f8, 0xed5be426c2b44001, 0xc49420a000, 0x137224, 0x4ae000, 0xed5be426c2b4400a, 0xf0d573c30a46e61a, 0xd8395f58b032a2a0, ...)
	src/github.com/restic/restic/internal/archiver/archiver.go:104 +0x1de
github.com/restic/restic/internal/archiver.(*Archiver).saveChunk(0xc46d82a230, 0xdc8e00, 0xc4200183f8, 0xd2a0714d, 0x137224, 0x74c2ce1e00000, 0xc49420a000, 0x137224, 0x4ae000, 0xc47a41a840, ...)
	src/github.com/restic/restic/internal/archiver/archiver.go:161 +0x135
created by github.com/restic/restic/internal/archiver.(*Archiver).SaveFile
	src/github.com/restic/restic/internal/archiver/archiver.go:239 +0x38c

Steps to reproduce the behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Sep 18, 2017

Member

Dang, this is a bug, thanks for reporting it! I thought we had resolved this error by now, meh.

Member

fd0 commented Sep 18, 2017

Dang, this is a bug, thanks for reporting it! I thought we had resolved this error by now, meh.

@fd0 fd0 added the bug label Sep 18, 2017

@felix9

This comment has been minimized.

Show comment
Hide comment
@felix9

felix9 Oct 5, 2017

Contributor

I took a look at this, and my first guess is that it's a race here:
https://github.com/restic/restic/blob/master/internal/repository/master_index.go#L125

Two worker threads find the same index that isn't final yet.
One writes to it, filling it up.
The index saver thread finds the full index and finalizes it.
Then the other worker thread tries to write to it, and panics.

One straightforward fix is to have MasterIndex.Store do Lock instead of RLock.
Another fix is to optimistically try Index.Store, and retry if the index became finalized.

I can code up either if you like, but I'm unsure how to write a test for this.

Contributor

felix9 commented Oct 5, 2017

I took a look at this, and my first guess is that it's a race here:
https://github.com/restic/restic/blob/master/internal/repository/master_index.go#L125

Two worker threads find the same index that isn't final yet.
One writes to it, filling it up.
The index saver thread finds the full index and finalizes it.
Then the other worker thread tries to write to it, and panics.

One straightforward fix is to have MasterIndex.Store do Lock instead of RLock.
Another fix is to optimistically try Index.Store, and retry if the index became finalized.

I can code up either if you like, but I'm unsure how to write a test for this.

@fd0

This comment has been minimized.

Show comment
Hide comment
@fd0

fd0 Oct 7, 2017

Member

I think you're right, thanks a lot for the analysis!

Let's use Lock instead of RLock and submit that as a PR. And I don't have an idea on how to test this either.

I'd like to refactor/rework the whole index handling thing anyway.

Member

fd0 commented Oct 7, 2017

I think you're right, thanks a lot for the analysis!

Let's use Lock instead of RLock and submit that as a PR. And I don't have an idea on how to test this either.

I'd like to refactor/rework the whole index handling thing anyway.

@fd0 fd0 closed this in cc5ada6 Oct 7, 2017

fd0 added a commit that referenced this issue Oct 7, 2017

Merge pull request #1339 from felix9/fix_1251
fixes #1251, race when writing indexes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment