Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image/png: fatal error: runtime: out of memory via (*decoder).parseIDAT #53778

Open
secsys-go opened this issue Jul 11, 2022 · 13 comments
Open
Labels
NeedsInvestigation

Comments

@secsys-go
Copy link

@secsys-go secsys-go commented Jul 11, 2022

What version of Go are you using (go version)?

$ go version
go version go1.17.8 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN="/home/zjx/workspace/gowork/bin"
GOCACHE="/home/zjx/.cache/go-build"
GOENV="/home/zjx/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/zjx/workspace/gowork/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/zjx/workspace/gowork"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/home/zjx/.local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/zjx/.local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.8"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/zjx/workspace/gowork/src/go-fdg-exmaples/std/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1878510231=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I attempt to decode an png data, and the details can be seen at:

go.dev/play

What did you expect to see?

It decodes the data successfully or just returns an error

What did you see instead?

It triggered a fatal error like:

go env Output
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x4af45e, 0x31ce0000000})
/home/zjx/.local/go/src/runtime/panic.go:1198 +0x71
runtime.sysMap(0xc000400000, 0x426780, 0x7fff08857758)
/home/zjx/.local/go/src/runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x5652e0, 0x18e70000)
/home/zjx/.local/go/src/runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x5652e0, 0x18e70000, 0x0, 0x1)
/home/zjx/.local/go/src/runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
/home/zjx/.local/go/src/runtime/mheap.go:913 +0x69
runtime.systemstack()
/home/zjx/.local/go/src/runtime/asm_amd64.s:383 +0x49

goroutine 1 [running]:
runtime.systemstack_switch()
/home/zjx/.local/go/src/runtime/asm_amd64.s:350 fp=0xc00010da00 sp=0xc00010d9f8 pc=0x458e20
runtime.(*mheap).alloc(0x0, 0xc00010daa8, 0x6f, 0x0)
/home/zjx/.local/go/src/runtime/mheap.go:907 +0x73 fp=0xc00010da50 sp=0xc00010da00 pc=0x422ab3
runtime.(*mcache).allocLarge(0x40b3fe, 0x31ce0000000, 0x87, 0x1)
/home/zjx/.local/go/src/runtime/mcache.go:227 +0x89 fp=0xc00010dab0 sp=0xc00010da50 pc=0x413949
runtime.mallocgc(0x31ce0000000, 0x499220, 0x1)
/home/zjx/.local/go/src/runtime/malloc.go:1088 +0x5c5 fp=0xc00010db30 sp=0xc00010dab0 pc=0x40bb45
runtime.makeslice(0xc0001b6000, 0x0, 0xc00010db98)
/home/zjx/.local/go/src/runtime/slice.go:98 +0x52 fp=0xc00010db58 sp=0xc00010db30 pc=0x444892
image.NewNRGBA64({{0x0, 0x0}, {0xc00010dc18, 0x1000000465c25}})
/home/zjx/.local/go/src/image/image.go:601 +0x66 fp=0xc00010dbd0 sp=0xc00010db58 pc=0x489fa6
image/png.(*decoder).readImagePass(0xc0000d7000, {0x7f125c277030, 0xc000180050}, 0x0, 0x0)
/home/zjx/.local/go/src/image/png/reader.go:493 +0x7b6 fp=0xc00010dda0 sp=0xc00010dbd0 pc=0x48c7f6
image/png.(*decoder).decode(0xc0000d7000)
/home/zjx/.local/go/src/image/png/reader.go:372 +0x1af fp=0xc00010de70 sp=0xc00010dda0 pc=0x48bb0f
image/png.(*decoder).parseIDAT(0xc0000d7000, 0xd7078)
/home/zjx/.local/go/src/image/png/reader.go:849 +0x25 fp=0xc00010de88 sp=0xc00010de70 pc=0x48f6e5
image/png.(*decoder).parseChunk(0xc0000d7000)
/home/zjx/.local/go/src/image/png/reader.go:908 +0x128 fp=0xc00010def8 sp=0xc00010de88 pc=0x48f888
image/png.Decode({0x4cb540, 0xc0001a6000})
/home/zjx/.local/go/src/image/png/reader.go:967 +0x11b fp=0xc00010df60 sp=0xc00010def8 pc=0x48fe5b
main.main()
/home/zjx/workspace/gowork/src/go-fdg-exmaples/std/image/png/pocTestIncolplete/poc.go:9 +0x56 fp=0xc00010df80 sp=0xc00010df60 pc=0x4903b6
runtime.main()
/home/zjx/.local/go/src/runtime/proc.go:255 +0x227 fp=0xc00010dfe0 sp=0xc00010df80 pc=0x4324a7
runtime.goexit()
/home/zjx/.local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x45af01
exit status 2

Found by go-fuzz,

@robpike
Copy link
Contributor

@robpike robpike commented Jul 11, 2022

Your program says it exits without error, but it doesn't. My version, which checks the error returned by Decode, reports

% go run x.go
<nil> flate: corrupt input before offset 5
% 

https://go.dev/play/p/bAa9vbuJfgT

It may be running out of memory or CPU time on the playground, but the input is invalid and the error is reported on a proper machine.

Did you find this example by fuzzing?

@bcmills
Copy link
Member

@bcmills bcmills commented Jul 11, 2022

I am able to reproduce the reported behavior on Go at HEAD.

$ go version
go version devel go1.19-59ab6f351a Mon Jul 11 08:23:57 2022 +0000 linux/amd64

$ go mod init example
go: creating new go.mod: module example

$ cat > main.go
package main

import (
        "fmt"
        "image/png"
        "runtime"
        "strings"
)

func main() {
        fmt.Println(runtime.Version())
        _, err := png.Decode(strings.NewReader("\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\xffd\x00\x00\x00\x10\x06\x00\x00\x00\x8bӧ\xed0000IDATx\x9c\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\xffd\x00\x00\x00\x10\x06\x00\x00"))
        fmt.Println("decoded:", err)
}

$ go run .
devel go1.19-59ab6f351a Mon Jul 11 08:23:57 2022 +0000
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x4b3483?, 0x80000?})
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/panic.go:1047 +0x5d fp=0x7fff582f0740 sp=0x7fff582f0710 pc=0x431afd
runtime.sysMapOS(0xc000400000, 0x31ce0000000?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mem_linux.go:187 +0x11b fp=0x7fff582f0788 sp=0x7fff582f0740 pc=0x4152fb
runtime.sysMap(0x5695a0?, 0x7f721d373000?, 0x428520?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mem.go:142 +0x35 fp=0x7fff582f07b8 sp=0x7fff582f0788 pc=0x414cd5
runtime.(*mheap).grow(0x5695a0, 0x18e70000?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mheap.go:1459 +0x23d fp=0x7fff582f0828 sp=0x7fff582f07b8 pc=0x4255dd
runtime.(*mheap).allocSpan(0x5695a0, 0x18e70000, 0x0, 0x1)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mheap.go:1191 +0x1be fp=0x7fff582f08c0 sp=0x7fff582f0828 pc=0x424d3e
runtime.(*mheap).alloc.func1()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mheap.go:910 +0x65 fp=0x7fff582f0908 sp=0x7fff582f08c0 pc=0x4247c5
runtime.systemstack()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:492 +0x49 fp=0x7fff582f0910 sp=0x7fff582f0908 pc=0x45b0a9

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:459 fp=0xc00005f9d0 sp=0xc00005f9c8 pc=0x45b040
runtime.(*mheap).alloc(0x413b2b?, 0x57fd20?, 0x7?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mheap.go:904 +0x65 fp=0xc00005fa18 sp=0xc00005f9d0 pc=0x424705
runtime.(*mcache).allocLarge(0x7?, 0x31ce0000000, 0x1)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mcache.go:233 +0x85 fp=0xc00005fa68 sp=0xc00005fa18 pc=0x413c65
runtime.mallocgc(0x31ce0000000, 0x49c4c0, 0x1)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/malloc.go:1029 +0x57e fp=0xc00005fae0 sp=0xc00005fa68 pc=0x40bd5e
runtime.makeslice(0xc00005fb38?, 0xc00005fb38?, 0x446b72?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/slice.go:103 +0x52 fp=0xc00005fb08 sp=0xc00005fae0 pc=0x446b72
image.NewNRGBA64({{0xc000104bb8?, 0x484d6e?}, {0x100c000121088?, 0x4d1f08?}})
        /usr/local/google/home/bcmills/sdk/gotip/src/image/image.go:603 +0x66 fp=0xc00005fb80 sp=0xc00005fb08 pc=0x48da26
image/png.(*decoder).readImagePass(0x4d1f68?, {0x7f71f5ec6020, 0xc000078050}, 0x0?, 0x0)
        /usr/local/google/home/bcmills/sdk/gotip/src/image/png/reader.go:495 +0x845 fp=0xc00005fd58 sp=0xc00005fb80 pc=0x4902c5
image/png.(*decoder).decode(0xc000052c00)
        /usr/local/google/home/bcmills/sdk/gotip/src/image/png/reader.go:374 +0x1ae fp=0xc00005fe28 sp=0xc00005fd58 pc=0x48f56e
image/png.(*decoder).parseIDAT(0xc000052c00, 0x52c78?)
        /usr/local/google/home/bcmills/sdk/gotip/src/image/png/reader.go:859 +0x25 fp=0xc00005fe40 sp=0xc00005fe28 pc=0x493085
image/png.(*decoder).parseChunk(0xc000052c00)
        /usr/local/google/home/bcmills/sdk/gotip/src/image/png/reader.go:918 +0x129 fp=0xc00005feb0 sp=0xc00005fe40 pc=0x493229
image/png.Decode({0x4d1fe8?, 0xc00006e020})
        /usr/local/google/home/bcmills/sdk/gotip/src/image/png/reader.go:977 +0x11b fp=0xc00005ff18 sp=0xc00005feb0 pc=0x4937fb
main.main()
        /tmp/tmp.oRxzJ95Mq5/main.go:12 +0xa5 fp=0xc00005ff80 sp=0xc00005ff18 pc=0x493d65
runtime.main()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:250 +0x212 fp=0xc00005ffe0 sp=0xc00005ff80 pc=0x434352
runtime.goexit()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x45d121

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:363 +0xd6 fp=0xc00004efb0 sp=0xc00004ef90 pc=0x434716
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:369
runtime.forcegchelper()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:302 +0xad fp=0xc00004efe0 sp=0xc00004efb0 pc=0x4345ad
runtime.goexit()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00004efe8 sp=0xc00004efe0 pc=0x45d121
created by runtime.init.6
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:363 +0xd6 fp=0xc00004f790 sp=0xc00004f770 pc=0x434716
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgcsweep.go:278 +0x8e fp=0xc00004f7c8 sp=0xc00004f790 pc=0x4217ae
runtime.gcenable.func1()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgc.go:178 +0x26 fp=0xc00004f7e0 sp=0xc00004f7c8 pc=0x416686
runtime.goexit()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00004f7e8 sp=0xc00004f7e0 pc=0x45d121
created by runtime.gcenable
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x4d18c8?, 0x1?, 0x0?, 0x0?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:363 +0xd6 fp=0xc00004ff70 sp=0xc00004ff50 pc=0x434716
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x550e60)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc00004ffa0 sp=0xc00004ff70 pc=0x41f853
runtime.bgscavenge(0x0?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgcscavenge.go:617 +0x45 fp=0xc00004ffc8 sp=0xc00004ffa0 pc=0x41fe25
runtime.gcenable.func2()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgc.go:179 +0x26 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0x416626
runtime.goexit()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0x45d121
created by runtime.gcenable
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x434a97?, 0x49?, 0xe0?, 0xe7?, 0xc00004e770?)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:363 +0xd6 fp=0xc00004e628 sp=0xc00004e608 pc=0x434716
runtime.goparkunlock(...)
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/proc.go:369
runtime.runfinq()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mfinal.go:180 +0x10f fp=0xc00004e7e0 sp=0xc00004e628 pc=0x41578f
runtime.goexit()
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00004e7e8 sp=0xc00004e7e0 pc=0x45d121
created by runtime.createfing
        /usr/local/google/home/bcmills/sdk/gotip/src/runtime/mfinal.go:157 +0x45
exit status 2

@bcmills bcmills changed the title image/png: fatal error: runtime: out of memory image/png: fatal error: runtime: out of memory in NewNRGBA64 Jul 11, 2022
@bcmills bcmills changed the title image/png: fatal error: runtime: out of memory in NewNRGBA64 image/png: fatal error: runtime: out of memory via (*decoder).parseIDAT Jul 11, 2022
@bcmills bcmills added the NeedsInvestigation label Jul 11, 2022
@bcmills
Copy link
Member

@bcmills bcmills commented Jul 11, 2022

(CC @nigeltao)

@robpike
Copy link
Contributor

@robpike robpike commented Jul 12, 2022

Interesting. I am on an M1 with 128GB of memory and see an error, not a crash.

@rsc
Copy link
Contributor

@rsc rsc commented Jul 12, 2022

I ran Rob's version and Ctrl-'ed it after a few seconds and it looks like it is allocating 3.4 TB. Probably different systems handle that differently.

@rsc
Copy link
Contributor

@rsc rsc commented Jul 12, 2022

png.DecodeConfig runs fine and says 255x1677721600, so this may be another instance where untrusted inputs should be checked with DecodeConfig before being passed to Decode.

@robpike
Copy link
Contributor

@robpike robpike commented Jul 12, 2022

I tried it on a smaller M1 and still saw the error, not the crash.

My summary: this is a buggy program that does not protect against bad inputs as instructed, but the variant behavior on different CPU types and/or operating systems is worth investigating.

@secsys-go
Copy link
Author

@secsys-go secsys-go commented Jul 13, 2022

Your program says it exits without error, but it doesn't. My version, which checks the error returned by Decode, reports

% go run x.go
<nil> flate: corrupt input before offset 5
% 

https://go.dev/play/p/bAa9vbuJfgT

It may be running out of memory or CPU time on the playground, but the input is invalid and the error is reported on a proper machine.

Did you find this example by fuzzing?

Yeah, I found it by go-fuzz, and the program exceeded the memory limit of the system when applying for a slice according to mutated size of the image.

In fact, what I want to say at the last println in playground is "If it exits without crashes, you should see this sentence", and the link of playground has been updated as go.dev/play

@nigeltao
Copy link
Contributor

@nigeltao nigeltao commented Jul 13, 2022

this may be another instance where untrusted inputs should be checked with DecodeConfig before being passed to Decode.

I'd agree with that.

I'd also say that the bug is a dupe of #5050 (whose title says image/gif but really applies to all the image formats).

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 13, 2022

One approach we could use for this kind of problem is a incremental image build, as in https://go.dev/cl/417477. @nigeltao Any opinions about that approach? Thanks.

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 13, 2022

Change https://go.dev/cl/417477 mentions this issue: image/png: build large images incrementally

@nigeltao
Copy link
Contributor

@nigeltao nigeltao commented Jul 13, 2022

One approach we could use for this kind of problem is a incremental image build, as in https://go.dev/cl/417477. @nigeltao Any opinions about that approach?

I dropped a code review comment on that page.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 14, 2022

Bringing the discussion back to the issue.

It's a fair point that the CL doesn't protect well against zlib. Although I do think of this kind of change more as aimed at a corrupt file than as a defense against an attacker.

I also think it's mildly unfortunate to expect people to check the DecodeConfig result first, as there is no obvious limit to check for, and there is no natural way to call DecodeConfig, check values, and then call Decode, as you need to rewind the reader. Or so it seems to me, I don't know what much about these packages.

Certainly the CL needs more work, it's just a sketch.

What concerns me about these issues is the unrecoverable crash that occurs on many systems, particularly when running in a container. It's not a friendly response to corrupt data. This is related to the discussion at #20169, which remains unresolved. I've been trying to tackle a subset of these issues in the CLs starting at https://go.dev/cl/408678. Basically I'm trying to work within our existing constraints to return an error rather than crashing. This CL, https://go.dev/cl/417477, is an attempt at taking this approach to image/png, but it may be a poor fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation
Projects
None yet
Development

No branches or pull requests

7 participants