Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failure in armel: parser panic: runtime error: integer divide by zero #2058

Closed
TheTincho opened this Issue Oct 6, 2016 · 9 comments

Comments

Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
3 participants
@TheTincho @beorn7 @brian-brazil
@TheTincho
Copy link
Contributor

TheTincho commented Oct 6, 2016

Hi,

I am getting build failures in armel due to a test failing with an "unexpected error", that turns out to be a division by zero in the parser:

time="2016-10-02T18:56:22Z" level=error msg="parser panic: runtime error: integer divide by zero
goroutine 386 [running]:
github.com/prometheus/prometheus/promql.(*evaluator).recover(0x135a2b30, 0x1084bb70)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:557 +0xb8
panic(0x3d13d0, 0x10828010)
    /usr/lib/go-1.7/src/runtime/panic.go:458 +0x454
github.com/prometheus/prometheus/promql.vectorElemBinop(0x17, 0x0, 0x40a45000, 0x0, 0x45000000, 0x13d93638, 0x0, 0x2)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:1060 +0x1f0
github.com/prometheus/prometheus/promql.(*evaluator).vectorScalarBinop(0x135a2b30, 0x17, 0x13d93630, 0x2, 0x2, 0x135a2c60, 0x135a0000, 0x0, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:991 +0x104
github.com/prometheus/prometheus/promql.(*evaluator).eval(0x135a2b30, 0x579200, 0x13f290c0, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:669 +0xe2c
github.com/prometheus/prometheus/promql.(*evaluator).Eval(0x135a2b30, 0x579200, 0x13f290c0, 0x0, 0x0, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:630 +0x94
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0x135a6360, 0xb5380000, 0x1474ca00, 0x1084be88, 0x13642980, 0x0, 0x0, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:383 +0x250
github.com/prometheus/prometheus/promql.(*Engine).exec(0x135a6360, 0x1084be88, 0x0, 0x0, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4b4
github.com/prometheus/prometheus/promql.(*query).Exec(0x1084be88, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:196 +0x28
github.com/prometheus/prometheus/promql.(*Test).exec(0x1231a4b0, 0x576c90, 0x1232f920, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/test.go:466 +0x29c
github.com/prometheus/prometheus/promql.(*Test).Run(0x1231a4b0, 0x0, 0x0)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/test.go:444 +0xa8
github.com/prometheus/prometheus/promql.TestEvaluations(0x1080aa80)
    /«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/promql_test.go:31 +0x274
testing.tRunner(0x1080aa80, 0x45ad90)
    /usr/lib/go-1.7/src/testing/testing.go:610 +0xa4
created by testing.(*T).Run
    /usr/lib/go-1.7/src/testing/testing.go:646 +0x304
" source="engine.go:559"

Full test log output:

=== RUN   TestEvaluations
time="2016-10-02T18:56:19Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:19Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:19Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:19Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:19Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:19Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:19Z" level=info msg="Done checkpointing in-memory metrics and chunks in 822.831µs." source="persistence.go:572" 
time="2016-10-02T18:56:19Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:19Z" level=info msg="Done checkpointing fingerprint mappings in 134.238µs." source="persistence.go:1394" 
time="2016-10-02T18:56:19Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:19Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:19Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:19Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:19Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:19Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:19Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:19Z" level=info msg="Done checkpointing in-memory metrics and chunks in 372.957µs." source="persistence.go:572" 
time="2016-10-02T18:56:19Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:19Z" level=info msg="Done checkpointing fingerprint mappings in 137.358µs." source="persistence.go:1394" 
time="2016-10-02T18:56:19Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:19Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:19Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:19Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:19Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:19Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:19Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:19Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:19Z" level=info msg="Done checkpointing in-memory metrics and chunks in 644.914µs." source="persistence.go:572" 
time="2016-10-02T18:56:19Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:19Z" level=info msg="Done checkpointing fingerprint mappings in 139.039µs." source="persistence.go:1394" 
time="2016-10-02T18:56:19Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 867.511µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 208.958µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 856.712µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 267.478µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 867.631µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 493.555µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 365.716µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 163.398µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 318.316µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 144.079µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 335.717µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 145.159µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 312.157µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 142.558µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 301.797µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 192.878µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 637.834µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 205.358µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 313.077µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 164.959µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 333.957µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 143.639µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 343.196µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 223.158µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 328.637µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 155.558µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:20Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:20Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:20Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:20Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing in-memory metrics and chunks in 692.953µs." source="persistence.go:572" 
time="2016-10-02T18:56:20Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:20Z" level=info msg="Done checkpointing fingerprint mappings in 141.318µs." source="persistence.go:1394" 
time="2016-10-02T18:56:20Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:20Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:20Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 530.635µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 147.079µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 460.675µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 159.359µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 410.796µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 261.518µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 358.676µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 145.279µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 214.357µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 142.319µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 1.669344ms." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 146.318µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 1.618744ms." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 141.438µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 389.716µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 140.679µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 552.954µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 228.158µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 279.397µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 142.399µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:21Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:21Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:21Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:21Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:21Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:21Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 249.197µs." source="persistence.go:572" 
time="2016-10-02T18:56:21Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:21Z" level=info msg="Done checkpointing fingerprint mappings in 145.839µs." source="persistence.go:1394" 
time="2016-10-02T18:56:21Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:22Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:22Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:22Z" level=error msg="parser panic: runtime error: integer divide by zero\ngoroutine 386 [running]:\ngithub.com/prometheus/prometheus/promql.(*evaluator).recover(0x135a2b30, 0x1084bb70)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:557 +0xb8\npanic(0x3d13d0, 0x10828010)\n\t/usr/lib/go-1.7/src/runtime/panic.go:458 +0x454\ngithub.com/prometheus/prometheus/promql.vectorElemBinop(0x17, 0x0, 0x40a45000, 0x0, 0x45000000, 0x13d93638, 0x0, 0x2)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:1060 +0x1f0\ngithub.com/prometheus/prometheus/promql.(*evaluator).vectorScalarBinop(0x135a2b30, 0x17, 0x13d93630, 0x2, 0x2, 0x135a2c60, 0x135a0000, 0x0, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:991 +0x104\ngithub.com/prometheus/prometheus/promql.(*evaluator).eval(0x135a2b30, 0x579200, 0x13f290c0, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:669 +0xe2c\ngithub.com/prometheus/prometheus/promql.(*evaluator).Eval(0x135a2b30, 0x579200, 0x13f290c0, 0x0, 0x0, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:630 +0x94\ngithub.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0x135a6360, 0xb5380000, 0x1474ca00, 0x1084be88, 0x13642980, 0x0, 0x0, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:383 +0x250\ngithub.com/prometheus/prometheus/promql.(*Engine).exec(0x135a6360, 0x1084be88, 0x0, 0x0, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4b4\ngithub.com/prometheus/prometheus/promql.(*query).Exec(0x1084be88, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/engine.go:196 +0x28\ngithub.com/prometheus/prometheus/promql.(*Test).exec(0x1231a4b0, 0x576c90, 0x1232f920, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/test.go:466 +0x29c\ngithub.com/prometheus/prometheus/promql.(*Test).Run(0x1231a4b0, 0x0, 0x0)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/test.go:444 +0xa8\ngithub.com/prometheus/prometheus/promql.TestEvaluations(0x1080aa80)\n\t/«BUILDDIR»/prometheus-1.1.3+ds/build/src/github.com/prometheus/prometheus/promql/promql_test.go:31 +0x274\ntesting.tRunner(0x1080aa80, 0x45ad90)\n\t/usr/lib/go-1.7/src/testing/testing.go:610 +0xa4\ncreated by testing.(*T).Run\n\t/usr/lib/go-1.7/src/testing/testing.go:646 +0x304\n" source="engine.go:559" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:22Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:22Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:22Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:22Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:22Z" level=info msg="Done checkpointing in-memory metrics and chunks in 868.672µs." source="persistence.go:572" 
time="2016-10-02T18:56:22Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:22Z" level=info msg="Done checkpointing fingerprint mappings in 142.558µs." source="persistence.go:1394" 
time="2016-10-02T18:56:22Z" level=info msg="Local storage stopped." source="storage.go:406" 
time="2016-10-02T18:56:22Z" level=info msg="Loading series map and head chunks..." source="storage.go:358" 
time="2016-10-02T18:56:22Z" level=info msg="0 series loaded." source="storage.go:363" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping local storage..." source="storage.go:381" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" 
time="2016-10-02T18:56:22Z" level=info msg="Maintenance loop stopped." source="storage.go:1143" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping series quarantining..." source="storage.go:387" 
time="2016-10-02T18:56:22Z" level=info msg="Series quarantining stopped." source="storage.go:1554" 
time="2016-10-02T18:56:22Z" level=info msg="Stopping chunk eviction..." source="storage.go:391" 
time="2016-10-02T18:56:22Z" level=info msg="Chunk eviction stopped." source="storage.go:963" 
time="2016-10-02T18:56:22Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" 
time="2016-10-02T18:56:22Z" level=info msg="Done checkpointing in-memory metrics and chunks in 511.114µs." source="persistence.go:572" 
time="2016-10-02T18:56:22Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1371" 
time="2016-10-02T18:56:22Z" level=info msg="Done checkpointing fingerprint mappings in 158.759µs." source="persistence.go:1394" 
time="2016-10-02T18:56:22Z" level=info msg="Local storage stopped." source="storage.go:406" 
--- FAIL: TestEvaluations (2.76s)
    promql_test.go:33: error running test testdata/operators.test: error evaluating query: unexpected error
FAIL
exit status 1
FAIL    github.com/prometheus/prometheus/promql 2.938s
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Oct 14, 2016

Unfortunately, neither the stack trace nor the test log tell us which query was evaluated when the error occurred. And it's happening only on that platform, I assume.

Is there any way to reproduce this locally on x86_64 hardware? (I'm not familiar with the Debian crossbuild environment and how it runs the tests on those platforms. I guess it's some emulated hardware, or do you have the physical hardware hooked up?)
@TheTincho

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

TheTincho commented Oct 15, 2016

Hi!

Yes, this is only happening in armel right now. We use real hardware, and I can log in to one of the porter boxes if you want me to try something.
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Oct 17, 2016

The problem here is that the testing code doesn't spit out the expression it was testing if an unexpected panic happens.

If the test ran (and failed) on my local machine, I'd insert some Println statements to interactively find out what's going on. But that requires a quick roundtrip.

Ideally, we would improve the test to better report in the case of that kind of error. But it looks like a really weird error, quite possible a Go compile issue on that platform.

@brian-brazil brian-brazil added the kind/bug label Oct 26, 2016

@TheTincho

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

TheTincho commented Nov 3, 2016

This seems to be the failing code block, so it does not make any sense to have a div0 error...

        case itemMOD:
                if rhs != 0 {
                        return model.SampleValue(int(lhs) % int(rhs)), true
                }
                return model.SampleValue(math.NaN()), true
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Nov 3, 2016

Oh yes, it does.

This is actually a real bug. The fact that it only happens on some architectures has to do with rounding precision depending on size of an int or something.

Brilliant, I'll have a fix ready in 5m and merge it into 1.2.

One more reason to cut 1.2.3. ;-/

@beorn7 beorn7 referenced this issue Nov 3, 2016

Merged

Fix MOD binop for scalars and vectors #2156

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Nov 3, 2016

I'm pretty sure #2156 solves this.
@TheTincho please reopen if not.

@beorn7 beorn7 closed this Nov 3, 2016

@TheTincho

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

TheTincho commented Nov 4, 2016

It did fix it, but immediately I've got a similar issue, also in armel:

    promql_test.go:33: error running test testdata/operators.test: error in eval sum(http_requests) BY (job) % 2 ^ 3 ^ 2 ^ 2: expected 1000 for {job="api-server"} but got NaN
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Nov 4, 2016

yeah, that's when the float overflows the int (it would even overflow int64 here, just coincidence again that it only throws up on a particular architecture). I'll file a separate bug.

@beorn7 beorn7 referenced this issue Nov 4, 2016

Closed

promql: int overflow not checked #2165

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.