Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: panic during json unmarshal #18403

Closed
cpuguy83 opened this issue Dec 21, 2016 · 6 comments

Comments

Projects
None yet
4 participants
@cpuguy83
Copy link

commented Dec 21, 2016

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.7.3

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

Unmarshal a byte slice via json.Unmarshal.
Paniced code is here: https://github.com/docker/docker/blob/6ef1060cd0acb847e06db890abb335faa837a9e2/volume/store/restore.go#L35

What did you expect to see?

I would not expect the panic here

What did you see instead?

Panic is here: https://gist.github.com/cpuguy83/715079190b7e00a367c3b2cc4a415c9c

Potentially there was some data corruption here as the system in question shows an unclean shutdown, however at the point that we are unmarshalling we've already pulled the data from boltdb, we're just unmarshalling the raw bytes to a struct.

Relevant struct is here:

type volumeMetadata struct {
	Name    string
	Driver  string
	Labels  map[string]string
	Options map[string]string
}

Tracking in moby/moby#29636

@ianlancetaylor ianlancetaylor changed the title panic during json unmarshal encoding/json: panic during json unmarshal Dec 21, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 21, 2016

The crash is occurring in the assembler implementation for bytes.Equal. It is comparing two byte slices whose length is the same and is greater than 0 and less than 8. It is executing a MOVQ instruction to load the bytes of the second slice. That MOVQ is failing with a SIGSEGV.

The second slice is the key field of the JSON string, as read by unquoteBytes in encoding/json/decode.go. That function always reads the first and last bytes of the slice and normally reads every byte of the slice. In some cases it copies the slice to a newly allocated []byte.

This crash frankly seems completely impossible unless there has already been some memory corruption. You say "potentially there was some data corruption here"; I would have to agree. The Go runtime can't operate normally in the presence of data corruption.

Is this problem repeatable?

@cpuguy83

This comment has been minimized.

Copy link
Author

commented Dec 21, 2016

It seems repeatable with the bolt db in question (which I'm still trying to get my hands on).
I've setup a test to make sure our usage of boltdb is not causing the panic here and it doesn't seem to be the case.

So, until I can get a copy of the affected db, I can't replicate.

@josharian

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

I think this may be due to a misuse of bolt.

The byte slice you are unmarshalling, entry.Value, comes from entries, which is generated in listEntries by directly preserving references to []byte from bolt.

However, the bolt docs state:

Keys and values retrieved from the database are only valid for the life of the transaction. When used outside the transaction, these byte slices can point to different data or can point to invalid memory which will cause a panic.

I believe the correct fix is in listEntries. When you are creating each new entry, make new k and v byte slices of the correct length and copy in the data from bolt.

You will probably want to audit the other uses of bolt for similar bugs.

@josharian

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

Something like this (untested, typed into github comment) should do the trick:

func listEntries(tx *bolt.Tx) []*dbEntry {
	var entries []*dbEntry
	b := tx.Bucket(volumeBucketName)
	b.ForEach(func(k, v []byte) error {
		// Make copies of k and v, since they will outlive tx.
		var entry dbEntry
		entry.k = append(entry.k, k...)
		entry.v = append(entry.v, v...)
		entries = append(entries, &entry)
		return nil
	})
	return entries
}
@cpuguy83

This comment has been minimized.

Copy link
Author

commented Dec 22, 2016

Ah thanks, I was wondering about that but hadn't seen anything, and couldn't repro the issue with the same useage.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

Closing based on @josharian's analysis. Please comment if that turns out to be incorrect.

@golang golang locked and limited conversation to collaborators Dec 22, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.