Out of Memory Error

[jamiegwatkin]

I encountered this error over the weekend (Prometheys version 0.16.0rc2 (branch: HEAD, revision: e7c60c2)):

time="2015-10-13T02:28:06Z" level=info msg="Done checkpointing in-memory metrics and chunks in 5.050651997s." file=persistence.go line=563 
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xcabce0, 0x16)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sysMap(0xc88cf00000, 0x100000, 0x1d2100, 0x107a1d8)
    /usr/local/go/src/runtime/mem_linux.go:143 +0x9b
runtime.mHeap_SysAlloc(0x105a700, 0x100000, 0x0)
    /usr/local/go/src/runtime/malloc.go:423 +0x160
runtime.mHeap_Grow(0x105a700, 0x8, 0x0)
    /usr/local/go/src/runtime/mheap.go:628 +0x63
runtime.mHeap_AllocSpanLocked(0x105a700, 0x7, 0x7f92e701a728)
    /usr/local/go/src/runtime/mheap.go:532 +0x5f1
runtime.mHeap_Alloc_m(0x105a700, 0x7, 0x39, 0x7f92e701a728)
    /usr/local/go/src/runtime/mheap.go:425 +0x1ac
runtime.mHeap_Alloc.func1()
    /usr/local/go/src/runtime/mheap.go:484 +0x41
runtime.systemstack(0x7f92f5d2fd58)
    /usr/local/go/src/runtime/asm_amd64.s:278 +0xab
runtime.mHeap_Alloc(0x105a700, 0x7, 0x10000000039, 0x414b94)
    /usr/local/go/src/runtime/mheap.go:485 +0x63
runtime.mCentral_Grow(0x1065e38, 0x0)
    /usr/local/go/src/runtime/mcentral.go:190 +0x93
runtime.mCentral_CacheSpan(0x1065e38, 0xc88b24d130)
    /usr/local/go/src/runtime/mcentral.go:86 +0x4d4
runtime.mCache_Refill(0x7f92f96ac000, 0x39, 0xc88b24d100)
    /usr/local/go/src/runtime/mcache.go:118 +0xcf
runtime.mallocgc.func2()
    /usr/local/go/src/runtime/malloc.go:611 +0x2b
runtime.systemstack(0xc820018000)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 8909948 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc88b3c0798 sp=0xc88b3c0790
runtime.mallocgc(0x3700, 0xb857c0, 0x0, 0xc81c99409b)
    /usr/local/go/src/runtime/malloc.go:612 +0x842 fp=0xc88b3c0868 sp=0xc88b3c0798
runtime.newarray(0xb857c0, 0x35d, 0xc800000000)
    /usr/local/go/src/runtime/malloc.go:777 +0xc9 fp=0xc88b3c08a8 sp=0xc88b3c0868
runtime.makeslice(0x9e0600, 0x0, 0x35d, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/slice.go:32 +0x165 fp=0xc88b3c08f8 sp=0xc88b3c08a8
github.com/prometheus/prometheus/storage/local.(*memorySeries).newIterator(0xc82152dc20, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/series.go:425 +0x5b fp=0xc88b3c09c0 sp=0xc88b3c08f8
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).NewIterator(0xc8200c82d0, 0xb5a85a28e131e950, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:325 +0x108 fp=0xc88b3c0a20 sp=0xc88b3c09c0
github.com/prometheus/prometheus/promql.(*Analyzer).Prepare.func2(0x7f92f9507498, 0xc87d82e040, 0x7f92f9507701)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/analyzer.go:156 +0x258 fp=0xc88b3c0b58 sp=0xc88b3c0a20
github.com/prometheus/prometheus/promql.inspector.Visit(0xc88c8b9540, 0x7f92f9507498, 0xc87d82e040, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:305 +0x40 fp=0xc88b3c0b88 sp=0xc88b3c0b58
github.com/prometheus/prometheus/promql.Walk(0x7f92f5d75598, 0xc88c8b9540, 0x7f92f9507498, 0xc87d82e040)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:254 +0x65 fp=0xc88b3c0d10 sp=0xc88b3c0b88
github.com/prometheus/prometheus/promql.Walk(0x7f92f5d75598, 0xc88c8b9540, 0x7f92f95077b8, 0xc87d82e0c0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:277 +0x643 fp=0xc88b3c0e98 sp=0xc88b3c0d10
github.com/prometheus/prometheus/promql.Walk(0x7f92f5d75598, 0xc88c8b9540, 0x7f92f9507760, 0xc88b85a910)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:287 +0x9b2 fp=0xc88b3c1020 sp=0xc88b3c0e98
github.com/prometheus/prometheus/promql.Walk(0x7f92f5d75598, 0xc88c8b9540, 0x7f92f9507440, 0xc87d82ea40)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:281 +0x8e9 fp=0xc88b3c11a8 sp=0xc88b3c1020
github.com/prometheus/prometheus/promql.Walk(0x7f92f5d75598, 0xc88c8b9540, 0x7f92f9507440, 0xc87d82ebc0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:280 +0x881 fp=0xc88b3c1330 sp=0xc88b3c11a8
github.com/prometheus/prometheus/promql.Walk(0x7f92f5d75598, 0xc88c8b9540, 0x7f92f9507440, 0xc888c5a100)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:280 +0x881 fp=0xc88b3c14b8 sp=0xc88b3c1330
github.com/prometheus/prometheus/promql.Inspect(0x7f92f9507440, 0xc888c5a100, 0xc88c8b9540)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/ast.go:315 +0x60 fp=0xc88b3c14f8 sp=0xc88b3c14b8
github.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc888c5a1c0, 0x7f92f5d75530, 0xc88b3e6720, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/analyzer.go:164 +0x77a fp=0xc88b3c1710 sp=0xc88b3c14f8
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc820142d00, 0x7f92f5d75530, 0xc88b3e6720, 0xc888c5a140, 0xc88b835b30, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/engine.go:386 +0x3be fp=0xc88b3c1a80 sp=0xc88b3c1710
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc820142d00, 0xc888c5a140, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4ca fp=0xc88b3c1b98 sp=0xc88b3c1a80
github.com/prometheus/prometheus/promql.(*query).Exec(0xc888c5a140, 0xc88b1bf000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/promql/engine.go:195 +0x2e fp=0xc88b3c1bf8 sp=0xc88b3c1b98
github.com/prometheus/prometheus/rules.(*AlertingRule).eval(0xc8200c7d50, 0x1505f078f2e, 0xc820142d00, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/rules/alerting.go:157 +0x140 fp=0xc88b3c1db8 sp=0xc88b3c1bf8
github.com/prometheus/prometheus/rules.(*Manager).runIteration.func1(0xc88b3b90b0, 0x1505f078f2e, 0xc820017440, 0x7f92f9507548, 0xc8200c7d50)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/rules/manager.go:252 +0xd3 fp=0xc88b3c1f58 sp=0xc88b3c1db8
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88b3c1f60 sp=0xc88b3c1f58
created by github.com/prometheus/prometheus/rules.(*Manager).runIteration
    /home/julius/gosrc/src/github.com/prometheus/prometheus/rules/manager.go:278 +0x209

goroutine 1 [select, 6188 minutes]:
main.Main(0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:176 +0x1ae6
main.main()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x18

goroutine 17 [syscall, 6188 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 102 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820072000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 19 [syscall, 6188 minutes]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 103 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820072000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 98 [select, 6188 minutes, locked to thread]:
runtime.gopark(0xd6f240, 0xc82001ef28, 0xc22d00, 0x6, 0x434b18, 0x2)
    /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82001ef28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82001ef28)
    /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 99 [select, 6188 minutes]:
main.Main.func2(0xc8202d5ec0, 0xc8202d5e60, 0xc820084a80, 0xc820412d20, 0x5, 0x5)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:127 +0x134
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:133 +0xebd

goroutine 100 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8200b8000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 104 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820072000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 105 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820072000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 106 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820072000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 107 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8200b87e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 108 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820072160)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 109 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820072160)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 110 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820072160)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 111 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820072160)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 112 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820072160)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 113 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8200b8a80)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 114 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820072840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 115 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820072840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 116 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820072840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 117 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820072840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 118 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820072840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 119 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8200b8c40)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 120 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820072c60)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 121 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820072c60)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 122 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820072c60)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 123 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820072c60)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 124 [select, 6188 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820072c60)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 125 [select]:
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc82047ec00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/persistence.go:1341 +0xcaf
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc82047ec00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/persistence.go:280 +0x21
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:252 +0x137

goroutine 126 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc8200c82d0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:658 +0x52d
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:275 +0x499

goroutine 127 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc8200c82d0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:875 +0x59b
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:276 +0x4be

goroutine 132 [semacquire]:
sync.runtime_Semacquire(0xc88b3b90bc)
    /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*WaitGroup).Wait(0xc88b3b90b0)
    /usr/local/go/src/sync/waitgroup.go:126 +0xb4
github.com/prometheus/prometheus/rules.(*Manager).runIteration(0xc820017440)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/rules/manager.go:280 +0x236
github.com/prometheus/prometheus/rules.(*Manager).Run(0xc820017440)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/rules/manager.go:158 +0x28a
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:158 +0x129b

goroutine 133 [chan receive]:
github.com/prometheus/prometheus/notification.(*NotificationHandler).Run(0xc8200e14a0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/notification/notification.go:208 +0xa3
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:161 +0x12eb

goroutine 146 [select]:
github.com/prometheus/prometheus/retrieval/discovery.(*FileDiscovery).Run(0xc82000ba10, 0xc825e82ae0, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/discovery/file.go:129 +0x552
created by github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:389 +0xc9

goroutine 135 [IO wait, 5093 minutes]:
net.runtime_pollWait(0x7f92f659b490, 0x72, 0xc820010140)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8200b0680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8200b0680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8200b0620, 0x0, 0x7f92f659b588, 0xc840576b00)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820024650, 0x4598a0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820024650, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc825e82d80, 0x7f92f659b550, 0xc820024650, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc825e82d80, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0xc1a6e0, 0x5, 0x7f92f659a4b8, 0xc8200dcf00, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
github.com/prometheus/prometheus/web.(*Handler).Run(0xc820084a80)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/web/web.go:224 +0x144
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:169 +0x13b6

goroutine 136 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc8200c82d0, 0x10c8, 0x3feeca346ca346ca, 0x1)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:762 +0x19a
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints.func1(0xc825e82600, 0xc8200c82d0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:802 +0x252
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:812 +0x5d

goroutine 137 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc8200c82d0, 0x0, 0x3ff0000000000000, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:762 +0x19a
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints.func1(0xc825e82660, 0xc8200c82d0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:835 +0x2bc
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/storage.go:855 +0x5d

goroutine 138 [select]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc825e82960, 0xc825e82900, 0xc8200e8000, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:141 +0x154
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:151 +0x4f2

goroutine 139 [select, 6188 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc825e82a20, 0xc825e829c0, 0xc8200e8000, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:141 +0x154
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:151 +0x4f2

goroutine 140 [chan receive]:
github.com/prometheus/prometheus/retrieval.merge.func1(0xc825e82960)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:83 +0x98
created by github.com/prometheus/prometheus/retrieval.merge
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:94 +0x151

goroutine 141 [chan receive, 6188 minutes]:
github.com/prometheus/prometheus/retrieval.merge.func1(0xc825e82a20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:83 +0x98
created by github.com/prometheus/prometheus/retrieval.merge
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:94 +0x151

goroutine 142 [semacquire, 6188 minutes]:
sync.runtime_Semacquire(0xc825e88bac)
    /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*WaitGroup).Wait(0xc825e88ba0)
    /usr/local/go/src/sync/waitgroup.go:126 +0xb4
github.com/prometheus/prometheus/retrieval.merge.func2(0xc825e88ba0, 0xc825e82a80)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:99 +0x21
created by github.com/prometheus/prometheus/retrieval.merge
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:101 +0x198

goroutine 143 [select]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).handleUpdates(0xc8200137c0, 0xc825e82a80, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:177 +0x356
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:157 +0x5ac

goroutine 144 [select, 6188 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc820143b80, 0xc825e829c0, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:392 +0x247
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 145 [select]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc820143b40, 0xc825e82900, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:392 +0x247
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 147 [syscall]:
syscall.Syscall6(0xe8, 0x16, 0xc825f0dc24, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x6, 0xc88b834318, 0x2)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.EpollWait(0x16, 0xc825f0dc24, 0x7, 0x7, 0xffffffffffffffff, 0x1, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:365 +0x89
github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify%2ev1.(*fdPoller).wait(0xc825e94dc0, 0xc825e82c00, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify.v1/inotify_poller.go:85 +0xbc
github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc825d3fdb0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify.v1/inotify.go:179 +0x1af
created by github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify%2ev1.NewWatcher
    /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify.v1/inotify.go:58 +0x315

goroutine 148 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea06c0, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 150 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea0900, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 151 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea0a20, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 152 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea0b40, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 154 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea0d80, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 155 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea0ea0, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 156 [chan receive, 6188 minutes]:
github.com/prometheus/prometheus/retrieval.(*StaticProvider).Run(0xc8203a76e0, 0xc825e834a0, 0xc825e827e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:547 +0x18c
created by github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:389 +0xc9

goroutine 157 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea0fc0, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 158 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea10e0, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 159 [select]:
net/http.(*persistConn).roundTrip(0xc88b3e8000, 0xc88b3cb0c0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc8579f7c20, 0xc88b35c8c0, 0xc800000000, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc88b35c8c0, 0x7f92f9500530, 0xc8579f7c20, 0xc88b3dc4a0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc8454473b0, 0xc88b35c8c0, 0x1f, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc8454473b0, 0xc88b35c8c0, 0xd6ee38, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc8454473b0, 0xc88b35c8c0, 0x6, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:175 +0x9c
github.com/prometheus/prometheus/retrieval.(*Target).scrape(0xc825ea1200, 0x7f92f659b5b0, 0xc88b3dc460, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:444 +0x428
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1200, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:364 +0x623
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 160 [select]:
net/http.(*persistConn).roundTrip(0xc88b3e8420, 0xc88b4418b0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc8579f7b00, 0xc88b35cc40, 0xc800000000, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc88b35cc40, 0x7f92f9500530, 0xc8579f7b00, 0xc88b7e7360, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc845447170, 0xc88b35cc40, 0x1f, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc845447170, 0xc88b35cc40, 0xd6ee38, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc845447170, 0xc88b35cc40, 0x6, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:175 +0x9c
github.com/prometheus/prometheus/retrieval.(*Target).scrape(0xc825ea1320, 0x7f92f659b5b0, 0xc88b7e7320, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:444 +0x428
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1320, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:364 +0x623
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 161 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1440, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 162 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1560, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 163 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1680, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 1574930 [IO wait, 5093 minutes]:
net.runtime_pollWait(0x7f92f659b310, 0x72, 0xc820010140)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc83c1fd020, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc83c1fd020, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc83c1fcfc0, 0xc839a30000, 0x1000, 0x1000, 0x0, 0x7f92f94fc050, 0xc820010140)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8304a1a60, 0xc839a30000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.(*liveSwitchReader).Read(0xc831aaa7d8, 0xc839a30000, 0x1000, 0x1000, 0xc8341524f0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:219 +0xa4
io.(*LimitedReader).Read(0xc840576a20, 0xc839a30000, 0x1000, 0x1000, 0xe00000000, 0x0, 0x0)
    /usr/local/go/src/io/io.go:427 +0xbd
bufio.(*Reader).fill(0xc83a711140)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc83a711140, 0xc81fffa30a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc83a711140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc83eb31fb0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc83eb31fb0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.ReadRequest(0xc83a711140, 0xc8200b9340, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:653 +0xb6
net/http.(*conn).readRequest(0xc831aaa790, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:633 +0x32f
net/http.(*conn).serve(0xc831aaa790)
    /usr/local/go/src/net/http/server.go:1319 +0x727
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 165 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea18c0, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 166 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea19e0, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 167 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1b00, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 1571910 [IO wait, 5093 minutes]:
net.runtime_pollWait(0x7f92f659af50, 0x72, 0xc820010140)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc83796c1b0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc83796c1b0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc83796c150, 0xc82ceb6000, 0x1000, 0x1000, 0x0, 0x7f92f94fc050, 0xc820010140)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc83bf4db10, 0xc82ceb6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.(*liveSwitchReader).Read(0xc83d7439b8, 0xc82ceb6000, 0x1000, 0x1000, 0xc83796c150, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:219 +0xa4
io.(*LimitedReader).Read(0xc843317400, 0xc82ceb6000, 0x1000, 0x1000, 0xe00000000, 0x0, 0x0)
    /usr/local/go/src/io/io.go:427 +0xbd
bufio.(*Reader).fill(0xc831f8e000)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc831f8e000, 0xc81fffa30a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc831f8e000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc835994f60, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc835994f60, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.ReadRequest(0xc831f8e000, 0xc8200b9260, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:653 +0xb6
net/http.(*conn).readRequest(0xc83d743970, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:633 +0x32f
net/http.(*conn).serve(0xc83d743970)
    /usr/local/go/src/net/http/server.go:1319 +0x727
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 169 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1d40, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 170 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc825ea1e60, 0x7f92f9501398, 0xc8200dce20)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:308 +0x780

goroutine 8610967 [chan send]:
github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter.func1(0xc8200dd9c0, 0xc872a8f920)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xdc
created by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter
    /home/julius/gosrc/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0x5d

goroutine 8910011 [IO wait]:
net.runtime_pollWait(0x7f92f659a590, 0x72, 0xc820010140)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc88b7ea290, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc88b7ea290, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc88b7ea230, 0xc88ab33000, 0x1000, 0x1000, 0x0, 0x7f92f94fc050, 0xc820010140)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc88a9ec3a0, 0xc88ab33000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f92f659b660, 0xc88a9ec3a0, 0xc88b3e8058, 0xc88ab33000, 0x1000, 0x1000, 0xc88b3e7a40, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc88b8cb400, 0xc88ab33000, 0x1000, 0x1000, 0xc825eddcf0, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc88b3e6fc0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc88b3e6fc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc88b3e8000)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 8910007 [IO wait]:
net.runtime_pollWait(0x7f92f659ab90, 0x72, 0xc820010140)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc88b7ea530, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc88b7ea530, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc88b7ea4d0, 0xc88ab23000, 0x1000, 0x1000, 0x0, 0x7f92f94fc050, 0xc820010140)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc88a9ec390, 0xc88ab23000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f92f659b660, 0xc88a9ec390, 0xc88b3e8478, 0xc88ab23000, 0x1000, 0x1000, 0xc88b3e7980, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc88b8cb300, 0xc88ab23000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc88b3e6d80)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc88b3e6d80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc88b3e8420)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 8910012 [select]:
net/http.(*persistConn).writeLoop(0xc88b3e8000)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 8910008 [select]:
net/http.(*persistConn).writeLoop(0xc88b3e8420)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

Prometheus is running on an EC2 instance with only 2GB of memory, I'm going to move it to a bigger instance but I thought it might be a good idea to log the issue here.

[fabxc]

2GB is very little for a serious deployment. Prometheus keeps a fair amount of data in memory to have good ingestion and query performance.

Only advice I can give is tuning according to the description here: http://prometheus.io/docs/operating/storage/#memory-usage

This has its limits though. How many targets are you scraping and how many metrics in total?

[jamiegwatkin]

We're really only testing it out at the moment hence the small size. We're scraping 18 targets - I can't find a total metric count.

We'll move to a bigger memory instance and see how it goes.

[fabxc]

If you let Prometheus scrape itself as in the getting started examples, you can check the number of currently scraped series via the prometheus_local_storage_memory_series metric.

[fabxc]

For 18 targets 2GB should still be fine – but you might have to turn the knobs a bit as described in the documentation.

[juliusv]

Yes, you'll want to tune the -storage.local.memory-chunks from the default of keeping 1 million chunks in memory to maybe a value of around 100000 (well, at least as large as the query prometheus_local_storage_memory_series currently reports to allow at least one chunk for every series in memory - though even if you set it lower than that, it would keep at least one chunk for every active series because that flag is only a lower bound on how many chunks should be kept in memory).

If you still get an OOM error after that, we'll have to look deeper.

[jamiegwatkin]

Seems to be working better with more ram, cheers!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.