Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: invalid pointer found on stack #27278

Closed
Funzinator opened this issue Aug 27, 2018 · 19 comments

Comments

Projects
None yet
8 participants
@Funzinator
Copy link

commented Aug 27, 2018

What did you do?

I have used blevesearch/bleve in my application and was accessing the index concurrently. One goroutine was indexing documents while another was querying.
At this point in time, I am unable to provide the sources since it's part of a bigger code base. I am trying to isolate it and provide a minimal example and will then update this issue if required.

The code was working fine with go 1.10.4 but isn't with go 1.11.

A similar issue has been filed at blevesearch/bleve#993 but they asked me to create an issue here instead because of runtime.morestack().

I have bisected the issue by recompiling my application with particular commits from the golang git repository.
The first commit, introducing this crash is f31a18d

commit f31a18ded405bdbc7b44a011d1434c83e7c39347
Date:   Wed Apr 11 22:47:24 2018 +0100

    cmd/compile: add some generic composite type optimizations

What did you expect to see?

The application works fine as it does with 1.10.4 - concurrent access to the cache is possible.

What did you see instead?

The runtime crashes with the following panic:

fatal error: invalid pointer found on stack

runtime stack:
runtime.throw(0xa698a4, 0x1e)
	/usr/local/go/go-1.11/src/runtime/panic.go:608 +0x72 fp=0x7f4dd3de1750 sp=0x7f4dd3de1720 pc=0x42c812
runtime.adjustpointers(0xc000e35348, 0x7f4dd3de1808, 0x7f4dd3de1bc8, 0xeb0998, 0xecfa60)
	/usr/local/go/go-1.11/src/runtime/stack.go:591 +0x221 fp=0x7f4dd3de17b0 sp=0x7f4dd3de1750 pc=0x4436c1
runtime.adjustframe(0x7f4dd3de1ad8, 0x7f4dd3de1bc8, 0xecfa60)
	/usr/local/go/go-1.11/src/runtime/stack.go:633 +0x188 fp=0x7f4dd3de1838 sp=0x7f4dd3de17b0 pc=0x443858
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000000300, 0x0, 0x0, 0x7fffffff, 0xa83ab8, 0x7f4dd3de1bc8, 0x0, ...)
	/usr/local/go/go-1.11/src/runtime/traceback.go:325 +0x1326 fp=0x7f4dd3de1b40 sp=0x7f4dd3de1838 pc=0x44daf6
runtime.copystack(0xc000000300, 0x4000, 0x20300000000001)
	/usr/local/go/go-1.11/src/runtime/stack.go:845 +0x26e fp=0x7f4dd3de1cf8 sp=0x7f4dd3de1b40 pc=0x443f0e
runtime.newstack()
	/usr/local/go/go-1.11/src/runtime/stack.go:1017 +0x312 fp=0x7f4dd3de1e90 sp=0x7f4dd3de1cf8 pc=0x444322
runtime.morestack()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:429 +0x8f fp=0x7f4dd3de1e98 sp=0x7f4dd3de1e90 pc=0x456acf

goroutine 1 [copystack]:
regexp/syntax.(*compiler).compile(0xc000e34a50, 0xc00015c1c0, 0x7ffffff)
	/usr/local/go/go-1.11/src/regexp/syntax/compile.go:98 +0x1832 fp=0xc000e34528 sp=0xc000e34520 pc=0x6b49d2
regexp/syntax.(*compiler).compile(0xc000e34a50, 0xc00015c2a0, 0xc000e9a840)
	/usr/local/go/go-1.11/src/regexp/syntax/compile.go:142 +0xe14 fp=0xc000e34790 sp=0xc000e34528 pc=0x6b3fb4
regexp/syntax.(*compiler).compile(0xc000e34a50, 0xc00015c620, 0x0)
	/usr/local/go/go-1.11/src/regexp/syntax/compile.go:154 +0xadf fp=0xc000e349f8 sp=0xc000e34790 pc=0x6b3c7f
regexp/syntax.Compile(0xc00015c620, 0xc00015c620, 0x1, 0x1)
	/usr/local/go/go-1.11/src/regexp/syntax/compile.go:83 +0x105 fp=0xc000e34ac0 sp=0xc000e349f8 pc=0x6b2f85
regexp.compile(0xc000684620, 0x9, 0xc0000000d4, 0x7, 0xc000684620, 0x9)
	/usr/local/go/go-1.11/src/regexp/regexp.go:178 +0xc7 fp=0xc000e34b48 sp=0xc000e34ac0 pc=0x6cc2c7
regexp.Compile(0xc000684620, 0x9, 0x7, 0xc000684620, 0x9)
	/usr/local/go/go-1.11/src/regexp/regexp.go:133 +0x41 fp=0xc000e34b88 sp=0xc000e34b48 pc=0x6cc1c1
funzi.org/vendor/github.com/blevesearch/bleve/search/query.(*WildcardQuery).convertToRegexp(0xc000c56030, 0xc000237d50, 0x4, 0xc00063e1c0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/search/query/wildcard.go:105 +0x5b fp=0xc000e34bc0 sp=0xc000e34b88 pc=0x8f63db
funzi.org/vendor/github.com/blevesearch/bleve/search/query.(*WildcardQuery).Searcher(0xc000c56030, 0xb04c00, 0xc00063e1c0, 0xb02d20, 0xc00014ef80, 0x0, 0xb03980, 0xc0012fc700, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/search/query/wildcard.go:88 +0x100 fp=0xc000e34c38 sp=0xc000e34bc0 pc=0x8f6250
funzi.org/vendor/github.com/blevesearch/bleve/search/query.(*ConjunctionQuery).Searcher(0xc000c56060, 0xb04c00, 0xc00063e1c0, 0xb02d20, 0xc00014ef80, 0x7f0000, 0xc00014ef58, 0xc00063e1c0, 0xc00014ef58, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/search/query/conjunction.go:58 +0x122 fp=0xc000e34cf0 sp=0xc000e34c38 pc=0x8e63b2
funzi.org/vendor/github.com/blevesearch/bleve/search/query.(*BooleanQuery).Searcher(0xc000e50000, 0xb04c00, 0xc00063e1c0, 0xb02d20, 0xc00014ef80, 0xc000f50000, 0x0, 0x0, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/search/query/boolean.go:132 +0x410 fp=0xc000e34d90 sp=0xc000e34cf0 pc=0x8e58d0
funzi.org/vendor/github.com/blevesearch/bleve.(*indexImpl).SearchInContext(0xc000242770, 0xb00b00, 0xc000024128, 0xc00015c070, 0x0, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index_impl.go:391 +0x2a9 fp=0xc000e35268 sp=0xc000e34d90 pc=0x8fdf09
funzi.org/vendor/github.com/blevesearch/bleve.(*indexImpl).Search(0xc000242770, 0xc00015c070, 0xf3b440, 0x8, 0xc000110180)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index_impl.go:363 +0x4d fp=0xc000e352b0 sp=0xc000e35268 pc=0x8fdc2d
funzi.org/index/indexer.(*BleveIndex).SearchActor(0xc000275100, 0xa58697, 0x5, 0xc00090c000, 0x268b, 0x3000, 0xafe8e0, 0xc000275100)
	/home/funzi/go/src/funzi.org/index/indexer/bleve.go:130 +0x9ca fp=0xc000e355b0 sp=0xc000e352b0 pc=0x914b2a
main.main()
	/home/funzi/go/src/funzi.org/index/main.go:196 +0x144a fp=0xc000e35f98 sp=0xc000e355b0 pc=0x91797a
runtime.main()
	/usr/local/go/go-1.11/src/runtime/proc.go:201 +0x207 fp=0xc000e35fe0 sp=0xc000e35f98 pc=0x42e187
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000e35fe8 sp=0xc000e35fe0 pc=0x458a61

goroutine 2 [force gc (idle)]:
runtime.gopark(0xa83cf8, 0xf3ab70, 0x1410, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000048f80 sp=0xc000048f60 pc=0x42e57b
runtime.goparkunlock(0xf3ab70, 0x1410, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc000048fb0 sp=0xc000048f80 pc=0x42e623
runtime.forcegchelper()
	/usr/local/go/go-1.11/src/runtime/proc.go:251 +0xb3 fp=0xc000048fe0 sp=0xc000048fb0 pc=0x42e3f3
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000048fe8 sp=0xc000048fe0 pc=0x458a61
created by runtime.init.4
	/usr/local/go/go-1.11/src/runtime/proc.go:240 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0xa83cf8, 0xf3aea0, 0x45140c, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000049780 sp=0xc000049760 pc=0x42e57b
runtime.goparkunlock(0xf3aea0, 0xaf140c, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc0000497b0 sp=0xc000049780 pc=0x42e623
runtime.bgsweep(0xc000072000)
	/usr/local/go/go-1.11/src/runtime/mgcsweep.go:71 +0x102 fp=0xc0000497d8 sp=0xc0000497b0 pc=0x421412
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000497e0 sp=0xc0000497d8 pc=0x458a61
created by runtime.gcenable
	/usr/local/go/go-1.11/src/runtime/mgc.go:216 +0x58

goroutine 4 [finalizer wait]:
runtime.gopark(0xa83cf8, 0xf590e8, 0x140f, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000049f28 sp=0xc000049f08 pc=0x42e57b
runtime.goparkunlock(0xf590e8, 0xa8140f, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc000049f58 sp=0xc000049f28 pc=0x42e623
runtime.runfinq()
	/usr/local/go/go-1.11/src/runtime/mfinal.go:175 +0x99 fp=0xc000049fe0 sp=0xc000049f58 pc=0x418d79
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x458a61
created by runtime.createfing
	/usr/local/go/go-1.11/src/runtime/mfinal.go:156 +0x61

goroutine 5 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000161dc8 sp=0xc000161da8 pc=0x42e57b
runtime.selectgo(0xc000161f70, 0xc000161f50, 0x2, 0x1, 0xc000048701)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000161f28 sp=0xc000161dc8 pc=0x43d7dd
funzi.org/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc0000342a0, 0xc000034300)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc000161fd0 sp=0xc000161f28 pc=0x78cc62
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000161fd8 sp=0xc000161fd0 pc=0x458a61
created by funzi.org/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 6 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000e9ddc8 sp=0xc000e9dda8 pc=0x42e57b
runtime.selectgo(0xc000e9df70, 0xc000e9df50, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000e9df28 sp=0xc000e9ddc8 pc=0x43d7dd
funzi.org/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc0000342a0, 0xc000034300)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc000e9dfd0 sp=0xc000e9df28 pc=0x78cc62
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000e9dfd8 sp=0xc000e9dfd0 pc=0x458a61
created by funzi.org/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 7 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000e99dc8 sp=0xc000e99da8 pc=0x42e57b
runtime.selectgo(0xc000e99f70, 0xc000e99f50, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000e99f28 sp=0xc000e99dc8 pc=0x43d7dd
funzi.org/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc0000342a0, 0xc000034300)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc000e99fd0 sp=0xc000e99f28 pc=0x78cc62
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000e99fd8 sp=0xc000e99fd0 pc=0x458a61
created by funzi.org/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 8 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000167dc8 sp=0xc000167da8 pc=0x42e57b
runtime.selectgo(0xc000167f70, 0xc000167f50, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000167f28 sp=0xc000167dc8 pc=0x43d7dd
funzi.org/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc0000342a0, 0xc000034300)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc000167fd0 sp=0xc000167f28 pc=0x78cc62
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000167fd8 sp=0xc000167fd0 pc=0x458a61
created by funzi.org/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 9 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00004bdd8 sp=0xc00004bdb8 pc=0x42e57b
runtime.selectgo(0xc00004bf68, 0xc00004bf60, 0x2, 0x0, 0x0)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc00004bf38 sp=0xc00004bdd8 pc=0x43d7dd
database/sql.(*DB).connectionOpener(0xc000110480, 0xb00ac0, 0xc000070dc0)
	/usr/local/go/go-1.11/src/database/sql/sql.go:1001 +0xe8 fp=0xc00004bfc8 sp=0xc00004bf38 pc=0x4ddb08
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004bfd0 sp=0xc00004bfc8 pc=0x458a61
created by database/sql.OpenDB
	/usr/local/go/go-1.11/src/database/sql/sql.go:671 +0x15d

goroutine 10 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc0000445c0 sp=0xc0000445a0 pc=0x42e57b
runtime.selectgo(0xc000044768, 0xc000044748, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000044720 sp=0xc0000445c0 pc=0x43d7dd
database/sql.(*DB).connectionResetter(0xc000110480, 0xb00ac0, 0xc000070dc0)
	/usr/local/go/go-1.11/src/database/sql/sql.go:1014 +0xfb fp=0xc0000447c8 sp=0xc000044720 pc=0x4ddc3b
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000447d0 sp=0xc0000447c8 pc=0x458a61
created by database/sql.OpenDB
	/usr/local/go/go-1.11/src/database/sql/sql.go:672 +0x193

goroutine 13 [select]:
runtime.gopark(0xa83d38, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000045530 sp=0xc000045510 pc=0x42e57b
runtime.selectgo(0xc0000456f0, 0xc0000456c4, 0x2, 0x0, 0x0)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000045690 sp=0xc000045530 pc=0x43d7dd
funzi.org/vendor/github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc0000750e0, 0xc000110540, 0xc000034660)
	/home/funzi/go/src/funzi.org/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xbf fp=0xc0000457c8 sp=0xc000045690 pc=0x7139bf
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000457d0 sp=0xc0000457c8 pc=0x458a61
created by funzi.org/vendor/github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
	/home/funzi/go/src/funzi.org/vendor/github.com/go-sql-driver/mysql/connection_go18.go:176 +0xbe

goroutine 14 [GC worker (idle)]:
runtime.gopark(0xa83ba8, 0xc0004511d0, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000044f60 sp=0xc000044f40 pc=0x42e57b
runtime.gcBgMarkWorker(0xc000036000)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc000044fd8 sp=0xc000044f60 pc=0x41c64c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000044fe0 sp=0xc000044fd8 pc=0x458a61
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 15 [GC worker (idle)]:
runtime.gopark(0xa83ba8, 0xc000468000, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000045f60 sp=0xc000045f40 pc=0x42e57b
runtime.gcBgMarkWorker(0xc000038500)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc000045fd8 sp=0xc000045f60 pc=0x41c64c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000045fe0 sp=0xc000045fd8 pc=0x458a61
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 18 [GC worker (idle)]:
runtime.gopark(0xa83ba8, 0xc000472000, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00046e760 sp=0xc00046e740 pc=0x42e57b
runtime.gcBgMarkWorker(0xc00003aa00)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc00046e7d8 sp=0xc00046e760 pc=0x41c64c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00046e7e0 sp=0xc00046e7d8 pc=0x458a61
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 34 [GC worker (idle)]:
runtime.gopark(0xa83ba8, 0xc000472010, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00046a760 sp=0xc00046a740 pc=0x42e57b
runtime.gcBgMarkWorker(0xc00003cf00)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc00046a7d8 sp=0xc00046a760 pc=0x41c64c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00046a7e0 sp=0xc00046a7d8 pc=0x458a61
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 16 [runnable]:
sync.(*Pool).Put(0xc00018c330, 0x971c00, 0xc0006478e0)
	/usr/local/go/go-1.11/src/sync/pool.go:88 +0x19f fp=0xc000e393c8 sp=0xc000e393c0 pc=0x46f59f
funzi.org/vendor/github.com/boltdb/bolt.(*Tx).write(0xc000226e00, 0x4be7c478a, 0xf3b440)
	/home/funzi/go/src/funzi.org/vendor/github.com/boltdb/bolt/tx.go:538 +0x45d fp=0xc000e39510 sp=0xc000e393c8 pc=0x8965bd
funzi.org/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc000226e00, 0xc000f57900, 0xc000e39678)
	/home/funzi/go/src/funzi.org/vendor/github.com/boltdb/bolt/tx.go:198 +0x310 fp=0xc000e39650 sp=0xc000e39510 pc=0x894470
funzi.org/vendor/github.com/blevesearch/bleve/index/store/boltdb.(*Writer).ExecuteBatch.func1(0xc000e397f0, 0xc000226e00)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/store/boltdb/writer.go:52 +0x4e fp=0xc000e39678 sp=0xc000e39650 pc=0x8b165e
funzi.org/vendor/github.com/blevesearch/bleve/index/store/boltdb.(*Writer).ExecuteBatch(0xc00000f0c0, 0xb01600, 0xc00096ff00, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/store/boltdb/writer.go:90 +0x503 fp=0xc000e397d8 sp=0xc000e39678 pc=0x8b11b3
funzi.org/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).batchRows(0xc00014ef00, 0xb00f40, 0xc00000f0c0, 0x0, 0x0, 0x0, 0xc00096fe80, 0x1, 0x1, 0x0, ...)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:292 +0x78e fp=0xc000e39c98 sp=0xc000e397d8 pc=0x7f075e
funzi.org/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Update(0xc00014ef00, 0xc0013d9600, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:484 +0x511 fp=0xc000e39e28 sp=0xc000e39c98 pc=0x7f2351
funzi.org/vendor/github.com/blevesearch/bleve.(*indexImpl).Index(0xc000242770, 0xc00075d908, 0x5, 0x9b5fe0, 0xc000226b60, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/vendor/github.com/blevesearch/bleve/index_impl.go:252 +0x21e fp=0xc000e39ea8 sp=0xc000e39e28 pc=0x8fd34e
funzi.org/index/indexer.(*BleveIndex).IndexActor(0xc000275100, 0xc00000ef30, 0x1, 0x1, 0x0, 0x0)
	/home/funzi/go/src/funzi.org/index/indexer/bleve.go:158 +0xa5 fp=0xc000e39ef8 sp=0xc000e39ea8 pc=0x9150d5
main.main.func1(0xc000237d70, 0xc00090c000, 0x268b, 0x3000, 0xafe8e0, 0xc000275100, 0xc000240e60)
	/home/funzi/go/src/funzi.org/index/main.go:177 +0x17a fp=0xc000e39fa8 sp=0xc000e39ef8 pc=0x91845a
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000e39fb0 sp=0xc000e39fa8 pc=0x458a61
created by main.main
	/home/funzi/go/src/funzi.org/index/main.go:170 +0x140e

goroutine 50 [timer goroutine (idle)]:
runtime.gopark(0xa83cf8, 0xf3f900, 0x1414, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00046cf28 sp=0xc00046cf08 pc=0x42e57b
runtime.goparkunlock(0xf3f900, 0xc000001414, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc00046cf58 sp=0xc00046cf28 pc=0x42e623
runtime.timerproc(0xf3f900)
	/usr/local/go/go-1.11/src/runtime/time.go:280 +0x288 fp=0xc00046cfd8 sp=0xc00046cf58 pc=0x44a1d8
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00046cfe0 sp=0xc00046cfd8 pc=0x458a61
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/go-1.11/src/runtime/time.go:170 +0x114

System details

go version go1.11 linux/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/funzi/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/funzi/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go/go-1.11"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/go-1.11/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version go1.11 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.11
uname -sr: Linux 4.16.11-100.fc26.x86_64
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	Fedora
Description:	Fedora release 26 (Twenty Six)
Release:	26
Codename:	TwentySix
/lib64/libc.so.6: GNU C Library (GNU libc) stable release version 2.25, by Roland McGrath et al.
gdb --version: GNU gdb (GDB) Fedora 8.0.1-36.fc26
@agnivade

This comment has been minimized.

Copy link
Member

commented Aug 27, 2018

@Funzinator - Can you post a sample code with steps for us to reproduce this ?

@heschik

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2018

This sounds a lot like #26407.

There should be a message saying which stack frame the bad pointer was in from

go/src/runtime/stack.go

Lines 590 to 591 in f43aa1d

print("runtime: bad pointer in frame ", funcname(f), " at ", pp, ": ", hex(p), "\n")
throw("invalid pointer found on stack")

-- can you get that? I think I might be able to figure this out without a repro but I really need to know which function is bad.

cc @mundaym

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2018

@heschik
Looking at adjustpointers' first arg, 0xc000e35348, and the SPs and FPs in the stack trace, it suggests the bad pointer is in the frame of

funzi.org/index/indexer.(*BleveIndex).SearchActor(0xc000275100, 0xa58697, 0x5, 0xc00090c000, 0x268b, 0x3000, 0xafe8e0, 0xc000275100)
	/home/funzi/go/src/funzi.org/index/indexer/bleve.go:130 +0x9ca fp=0xc000e355b0 sp=0xc000e352b0 pc=0x914b2a

@agnivade agnivade changed the title fatal error: invalid pointer found on stack with go 1.11 runtime: invalid pointer found on stack Aug 27, 2018

@agnivade agnivade added this to the Go1.12 milestone Aug 27, 2018

@heschik

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2018

@cherrymui oh, true. I thought that adjustpointers was called on the whole stack but I guess it's frame-by-frame. I'll take a look.

@heschik

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2018

...but I will need the source. @Funzinator

@Funzinator

This comment has been minimized.

Copy link
Author

commented Aug 27, 2018

It took me a while to strip the source code down to something that I can share and that still has the issue.
You can find it at https://github.com/Funzinator/bleve-go-one-eleven
To reproduce the issue with that, I run the following:

rm -rf /tmp/test-index ; go build -o ./index.bin synchronkartei.de/index && ./index.bin

And that's the output I get:

2018/08/27 23:59:36 unable to open index, trying to create a new one
2018/08/27 23:59:36 Start Indexing
runtime: bad pointer in frame synchronkartei.de/index/indexer.(*BleveIndex).SearchSprecher at 0xc00078de00: 0x2
fatal error: invalid pointer found on stack

runtime stack:
runtime.throw(0x93b1fa, 0x1e)
	/usr/local/go/go-1.11/src/runtime/panic.go:608 +0x72 fp=0x7ffe0646aba8 sp=0x7ffe0646ab78 pc=0x42c732
runtime.adjustpointers(0xc00078dc48, 0x7ffe0646ac60, 0x7ffe0646b020, 0xcc15f8, 0xcd86c0)
	/usr/local/go/go-1.11/src/runtime/stack.go:591 +0x221 fp=0x7ffe0646ac08 sp=0x7ffe0646aba8 pc=0x4435e1
runtime.adjustframe(0x7ffe0646af30, 0x7ffe0646b020, 0xcd86c0)
	/usr/local/go/go-1.11/src/runtime/stack.go:633 +0x188 fp=0x7ffe0646ac90 sp=0x7ffe0646ac08 pc=0x443778
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000000300, 0x0, 0x0, 0x7fffffff, 0x94b768, 0x7ffe0646b020, 0x0, ...)
	/usr/local/go/go-1.11/src/runtime/traceback.go:325 +0x1326 fp=0x7ffe0646af98 sp=0x7ffe0646ac90 pc=0x44da16
runtime.copystack(0xc000000300, 0x2000, 0x20300000000001)
	/usr/local/go/go-1.11/src/runtime/stack.go:845 +0x26e fp=0x7ffe0646b150 sp=0x7ffe0646af98 pc=0x443e2e
runtime.newstack()
	/usr/local/go/go-1.11/src/runtime/stack.go:1017 +0x312 fp=0x7ffe0646b2e8 sp=0x7ffe0646b150 pc=0x444242
runtime.morestack()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:429 +0x8f fp=0x7ffe0646b2f0 sp=0x7ffe0646b2e8 pc=0x4569ef

goroutine 1 [copystack]:
runtime.(*mspan).nextFreeIndex(0xd3eee0, 0x1)
	/usr/local/go/go-1.11/src/runtime/mbitmap.go:195 +0x177 fp=0xc00078d358 sp=0xc00078d350 pc=0x414e27
runtime.(*mcache).nextFree(0x7fe6938d5440, 0xb, 0x0, 0x0, 0x8)
	/usr/local/go/go-1.11/src/runtime/malloc.go:741 +0x55 fp=0xc00078d3b0 sp=0xc00078d358 pc=0x40cd55
runtime.mallocgc(0x40, 0x910360, 0x1, 0x910220)
	/usr/local/go/go-1.11/src/runtime/malloc.go:903 +0x793 fp=0xc00078d450 sp=0xc00078d3b0 pc=0x40d703
runtime.newobject(0x910360, 0xd8)
	/usr/local/go/go-1.11/src/runtime/malloc.go:1032 +0x38 fp=0xc00078d480 sp=0xc00078d450 pc=0x40dae8
synchronkartei.de/vendor/github.com/boltdb/bolt.(*Tx).init(0xc0004700e0, 0xc000196000)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/tx.go:49 +0x73 fp=0xc00078d520 sp=0xc00078d480 pc=0x7a8f03
synchronkartei.de/vendor/github.com/boltdb/bolt.(*DB).beginTx(0xc000196000, 0xc000392060, 0xc000053620, 0x69549a)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/db.go:486 +0x99 fp=0xc00078d590 sp=0xc00078d520 pc=0x7a18b9
synchronkartei.de/vendor/github.com/boltdb/bolt.(*DB).Begin(0xc000196000, 0x0, 0x1, 0xc000392060, 0x0)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/db.go:463 +0x6e fp=0xc00078d5c0 sp=0xc00078d590 pc=0x7a17ee
synchronkartei.de/vendor/github.com/blevesearch/bleve/index/store/boltdb.(*Store).Reader(0xc000098460, 0x1, 0x1, 0xc000392060, 0x1)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/store/boltdb/store.go:110 +0x38 fp=0xc00078d630 sp=0xc00078d5c0 pc=0x7c5a28
synchronkartei.de/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Reader(0xc000182100, 0x0, 0x0, 0x0, 0x0)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:1006 +0x48 fp=0xc00078d6a0 sp=0xc00078d630 pc=0x6f98c8
synchronkartei.de/vendor/github.com/blevesearch/bleve.(*indexImpl).SearchInContext(0xc000186070, 0x9b15c0, 0xc000024138, 0xc000792000, 0x0, 0x0, 0x0)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index_impl.go:381 +0x135 fp=0xc00078db78 sp=0xc00078d6a0 pc=0x825a05
synchronkartei.de/vendor/github.com/blevesearch/bleve.(*indexImpl).Search(0xc000186070, 0xc000792000, 0xd3ec40, 0x8, 0xc000794000)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index_impl.go:363 +0x4d fp=0xc00078dbc0 sp=0xc00078db78 pc=0x82589d
synchronkartei.de/index/indexer.(*BleveIndex).SearchSprecher(0xc0001ae0d0, 0x93170b, 0x5, 0xc00048c000, 0x2328, 0x2400, 0x9afb20, 0xc0001ae0d0)
	/home/funzi/go/src/synchronkartei.de/index/indexer/bleve.go:50 +0x678 fp=0xc00078de60 sp=0xc00078dbc0 pc=0x82f9c8
main.main()
	/home/funzi/go/src/synchronkartei.de/index/main.go:61 +0x256 fp=0xc00078df98 sp=0xc00078de60 pc=0x830116
runtime.main()
	/usr/local/go/go-1.11/src/runtime/proc.go:201 +0x207 fp=0xc00078dfe0 sp=0xc00078df98 pc=0x42e0a7
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00078dfe8 sp=0xc00078dfe0 pc=0x458981

goroutine 2 [force gc (idle)]:
runtime.gopark(0x94b9a8, 0xd3e440, 0x1410, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000042f80 sp=0xc000042f60 pc=0x42e49b
runtime.goparkunlock(0xd3e440, 0x1410, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc000042fb0 sp=0xc000042f80 pc=0x42e543
runtime.forcegchelper()
	/usr/local/go/go-1.11/src/runtime/proc.go:251 +0xb3 fp=0xc000042fe0 sp=0xc000042fb0 pc=0x42e313
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x458981
created by runtime.init.4
	/usr/local/go/go-1.11/src/runtime/proc.go:240 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x94b9a8, 0xd3e720, 0x45140c, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000043780 sp=0xc000043760 pc=0x42e49b
runtime.goparkunlock(0xd3e720, 0x9a140c, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc0000437b0 sp=0xc000043780 pc=0x42e543
runtime.bgsweep(0xc000080000)
	/usr/local/go/go-1.11/src/runtime/mgcsweep.go:71 +0x102 fp=0xc0000437d8 sp=0xc0000437b0 pc=0x421332
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000437e0 sp=0xc0000437d8 pc=0x458981
created by runtime.gcenable
	/usr/local/go/go-1.11/src/runtime/mgc.go:216 +0x58

goroutine 4 [finalizer wait]:
runtime.gopark(0x94b9a8, 0xd5c4a8, 0x140f, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000043f28 sp=0xc000043f08 pc=0x42e49b
runtime.goparkunlock(0xd5c4a8, 0x140f, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc000043f58 sp=0xc000043f28 pc=0x42e543
runtime.runfinq()
	/usr/local/go/go-1.11/src/runtime/mfinal.go:175 +0x99 fp=0xc000043fe0 sp=0xc000043f58 pc=0x418c99
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000043fe8 sp=0xc000043fe0 pc=0x458981
created by runtime.createfing
	/usr/local/go/go-1.11/src/runtime/mfinal.go:156 +0x61

goroutine 5 [select]:
runtime.gopark(0x94b9e8, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00073ddc8 sp=0xc00073dda8 pc=0x42e49b
runtime.selectgo(0xc00073df70, 0xc00073df50, 0x2, 0x1, 0x9ad401)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc00073df28 sp=0xc00073ddc8 pc=0x43d6fd
synchronkartei.de/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc000084120, 0xc000084180)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc00073dfd0 sp=0xc00073df28 pc=0x68e652
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00073dfd8 sp=0xc00073dfd0 pc=0x458981
created by synchronkartei.de/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 6 [select]:
runtime.gopark(0x94b9e8, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000741dc8 sp=0xc000741da8 pc=0x42e49b
runtime.selectgo(0xc000741f70, 0xc000741f50, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000741f28 sp=0xc000741dc8 pc=0x43d6fd
synchronkartei.de/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc000084120, 0xc000084180)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc000741fd0 sp=0xc000741f28 pc=0x68e652
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000741fd8 sp=0xc000741fd0 pc=0x458981
created by synchronkartei.de/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 7 [select]:
runtime.gopark(0x94b9e8, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000791dc8 sp=0xc000791da8 pc=0x42e49b
runtime.selectgo(0xc000791f70, 0xc000791f50, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc000791f28 sp=0xc000791dc8 pc=0x43d6fd
synchronkartei.de/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc000084120, 0xc000084180)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc000791fd0 sp=0xc000791f28 pc=0x68e652
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000791fd8 sp=0xc000791fd0 pc=0x458981
created by synchronkartei.de/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 8 [select]:
runtime.gopark(0x94b9e8, 0x0, 0x1809, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00073fdc8 sp=0xc00073fda8 pc=0x42e49b
runtime.selectgo(0xc00073ff70, 0xc00073ff50, 0x2, 0x1, 0x1)
	/usr/local/go/go-1.11/src/runtime/select.go:313 +0xcbd fp=0xc00073ff28 sp=0xc00073fdc8 pc=0x43d6fd
synchronkartei.de/vendor/github.com/blevesearch/bleve/index.AnalysisWorker(0xc000084120, 0xc000084180)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:83 +0x102 fp=0xc00073ffd0 sp=0xc00073ff28 pc=0x68e652
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00073ffd8 sp=0xc00073ffd0 pc=0x458981
created by synchronkartei.de/vendor/github.com/blevesearch/bleve/index.NewAnalysisQueue
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/analysis.go:75 +0xc8

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x94b858, 0xc00044b140, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000045f60 sp=0xc000045f40 pc=0x42e49b
runtime.gcBgMarkWorker(0xc000030000)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc000045fd8 sp=0xc000045f60 pc=0x41c56c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000045fe0 sp=0xc000045fd8 pc=0x458981
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 10 [GC worker (idle)]:
runtime.gopark(0x94b858, 0xc00045a000, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00003e760 sp=0xc00003e740 pc=0x42e49b
runtime.gcBgMarkWorker(0xc000032500)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc00003e7d8 sp=0xc00003e760 pc=0x41c56c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00003e7e0 sp=0xc00003e7d8 pc=0x458981
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x94b858, 0xc00045a010, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000462760 sp=0xc000462740 pc=0x42e49b
runtime.gcBgMarkWorker(0xc000034a00)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc0004627d8 sp=0xc000462760 pc=0x41c56c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004627e0 sp=0xc0004627d8 pc=0x458981
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x94b858, 0xc000466000, 0x1417, 0x0)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc000462f60 sp=0xc000462f40 pc=0x42e49b
runtime.gcBgMarkWorker(0xc000036f00)
	/usr/local/go/go-1.11/src/runtime/mgc.go:1772 +0xfc fp=0xc000462fd8 sp=0xc000462f60 pc=0x41c56c
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000462fe0 sp=0xc000462fd8 pc=0x458981
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/go-1.11/src/runtime/mgc.go:1720 +0x77

goroutine 11 [runnable]:
synchronkartei.de/vendor/github.com/boltdb/bolt.(*Cursor).searchNode(0xc00066d618, 0xc0003de180, 0xc, 0x10, 0xc0007b1340)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/cursor.go:274 +0x135 fp=0xc00066d4b0 sp=0xc00066d4a8 pc=0x79fc25
synchronkartei.de/vendor/github.com/boltdb/bolt.(*Cursor).search(0xc00066d618, 0xc0003de180, 0xc, 0x10, 0x559)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/cursor.go:268 +0x142 fp=0xc00066d558 sp=0xc00066d4b0 pc=0x79f8b2
synchronkartei.de/vendor/github.com/boltdb/bolt.(*Cursor).seek(0xc00066d618, 0xc0003de180, 0xc, 0x10, 0x0, 0x0, 0xc0003de180, 0x10, 0xc0003de180, 0xc00066d638, ...)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/cursor.go:159 +0xa5 fp=0xc00066d5a8 sp=0xc00066d558 pc=0x79f135
synchronkartei.de/vendor/github.com/boltdb/bolt.(*Bucket).Get(0xc0004782c0, 0xc0003de180, 0xc, 0x10, 0xc, 0x10, 0xa)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/boltdb/bolt/bucket.go:267 +0xcd fp=0xc00066d648 sp=0xc00066d5a8 pc=0x79c19d
synchronkartei.de/vendor/github.com/blevesearch/bleve/index/store/boltdb.(*Writer).ExecuteBatch(0xc00047aa68, 0x9b1f40, 0xc00075cec0, 0x0, 0x0)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/store/boltdb/writer.go:65 +0x26a fp=0xc00066d7a8 sp=0xc00066d648 pc=0x7c645a
synchronkartei.de/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).batchRows(0xc000182100, 0x9b1a00, 0xc00047aa68, 0xc00075ce60, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:292 +0x78e fp=0xc00066dc68 sp=0xc00066d7a8 pc=0x6f1fde
synchronkartei.de/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Update(0xc000182100, 0xc000905e40, 0x0, 0x0)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:484 +0x511 fp=0xc00066ddf8 sp=0xc00066dc68 pc=0x6f3bd1
synchronkartei.de/vendor/github.com/blevesearch/bleve.(*indexImpl).Index(0xc000186070, 0xc0002013b8, 0x4, 0x8af2e0, 0xc00020bb20, 0x0, 0x0)
	/home/funzi/go/src/synchronkartei.de/vendor/github.com/blevesearch/bleve/index_impl.go:252 +0x21e fp=0xc00066de78 sp=0xc00066ddf8 pc=0x824fbe
synchronkartei.de/index/indexer.(*BleveIndex).IndexSprecher(0xc0001ae0d0, 0xc00047a920, 0x1, 0x1, 0x0, 0x0)
	/home/funzi/go/src/synchronkartei.de/index/indexer/bleve.go:72 +0x7e fp=0xc00066dec8 sp=0xc00066de78 pc=0x82fe0e
main.main.func1(0xc000025b70, 0xc00048c000, 0x2328, 0x2400, 0x9afb20, 0xc0001ae0d0, 0xc00001f120)
	/home/funzi/go/src/synchronkartei.de/index/main.go:42 +0x160 fp=0xc00066dfa8 sp=0xc00066dec8 pc=0x830910
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00066dfb0 sp=0xc00066dfa8 pc=0x458981
created by main.main
	/home/funzi/go/src/synchronkartei.de/index/main.go:35 +0x21d

goroutine 12 [timer goroutine (idle)]:
runtime.gopark(0x94b9a8, 0xd42ce0, 0x1414, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:302 +0xeb fp=0xc00003ef28 sp=0xc00003ef08 pc=0x42e49b
runtime.goparkunlock(0xd42ce0, 0xc000001414, 0x1)
	/usr/local/go/go-1.11/src/runtime/proc.go:308 +0x53 fp=0xc00003ef58 sp=0xc00003ef28 pc=0x42e543
runtime.timerproc(0xd42ce0)
	/usr/local/go/go-1.11/src/runtime/time.go:280 +0x288 fp=0xc00003efd8 sp=0xc00003ef58 pc=0x44a0f8
runtime.goexit()
	/usr/local/go/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00003efe0 sp=0xc00003efd8 pc=0x458981
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/go-1.11/src/runtime/time.go:170 +0x114
@heschik

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2018

Thanks.

I'm away from my desk this week and this is too much for me to figure out on a laptop. If nobody gets to it before next week I'll take a harder look then.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

The bad pointer is autotmp_100, which is live at the call site.

The SSA before deadautoelim looks like

v324 (47) = VarDef <mem> {.autotmp_100} v323
v325 (47) = LocalAddr  {.autotmp_100} v2 v324
v326 (+47) = Zero  [224] v325 v324
v327 (47) = LocalAddr {.autotmp_100} v2 v326 
v332 (51) = OffPtr  [0] v327 
read v332 later

deadautoelim found a LocalAddr, v325, is used only in a store, and assumed nothing would read from it, and elided it, without knowing that v327 is also pointing to the same memory.

I think this is because LocalAddr (Addr at that time) didn't take memory operand before, when deadautoelim was written. So CSE makd it only one copy of LocalAddr per local variable. But this is no longer true after LocalAddr starts taking memory operand.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

Reading the code, I think my argument about LocalAddr is probably not correct. But somehow deadautoelim didn't find .autotmp_100 is used through the v327-v332 path. Still investigating...

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

Ok, I think I see what's going on. Besides the stores that are elided, the only use of v332 is NilCheck, which is special in that it has no use but deadcode doesn't elide it. deadautoelim didn't count it as a use, so it ends up compiling to nil check uninitialized memory. I think deadautoelim should count nil checks as uses.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 28, 2018

Change https://golang.org/cl/131955 mentions this issue: cmd/compile: count nil check as use in dead auto elim

@randall77

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

so it ends up compiling to nil check uninitialized memory.

I don't think that's quite the right description.
I think what happens is that there remains a nil check of an uninitialized local variable. If there is a safe point between when the variable is declared (the VarDef) and the nil check, then GC scanning at that safepoint scans uninitialized memory.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

You're right. The nil check applies on the address of some uninitialized memory, not the content.

@josharian

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

Seems like a backport candidate.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

Seems like a backport candidate.

I agree.

@cherrymui cherrymui modified the milestones: Go1.12, Go1.11.1 Aug 29, 2018

@heschik

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

@gopherbot please backport this fix for a serious compiler bug to 1.10.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 29, 2018

Backport issue(s) opened: #27342 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

@heschik I think the bug is introduced in Go 1.11. It doesn't need to be backported to Go 1.10.

@heschik

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

Ugh, I'm too sleepy. I'll retarget the backport bug.

@FiloSottile FiloSottile added this to the Go1.12 milestone Aug 31, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.