Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: unable to parse bool value #3230

Closed
jasonrm opened this issue Jul 3, 2015 · 10 comments
Closed

panic: unable to parse bool value #3230

jasonrm opened this issue Jul 3, 2015 · 10 comments

Comments

@jasonrm
Copy link

jasonrm commented Jul 3, 2015

Happens with both the collectd and graphite inputs. Haven't tested the others.

Add graphite to InfluxDB config
[[graphite]]
  enabled = true
  bind-address = ":2003"
  protocol = "tcp"
  separator = "_"
  tags = []
  templates = [
    "host.measurement.device.type.type_instance"
  ]
Test script
#!/bin/bash
SERVER=localhost
HTTP_PORT=8086
GRAPHITE_PORT=2003

# Write simple numerical value
echo "localhost.test.test.client.type_instance 4 `date +%s`" | nc -c ${SERVER} ${GRAPHITE_PORT}

# Attempt to write a string to that same key...
echo "localhost.test.test.client.type_instance Ping `date +%s`" | nc -c ${SERVER} ${GRAPHITE_PORT}
# No issue so far, InfludDB correctly handled it as invalid

# Now attempt to write 'NaN'
echo "localhost.test.test.client.type_instance NaN `date +%s`" | nc -c ${SERVER} ${GRAPHITE_PORT}
# And panic
Selected portion of log
2015/07/03 09:44:51 InfluxDB starting, version 0.9.0-HEAD, commit af00eb32a59345d8ac04f9f721de249ce88fd28c
2015/07/03 09:44:51 GOMAXPROCS set to 4
2015/07/03 09:44:51 Sending anonymous usage statistics to m.influxdb.com
[graphite] 2015/07/03 09:44:52 unable to parse data: field "localhost.test.test.client.type_instance" value: strconv.ParseFloat: parsing "Ping": invalid syntax
panic: unable to parse bool value 'NaN': strconv.ParseBool: parsing "NaN": invalid syntax


goroutine 68 [running]:
github.com/influxdb/influxdb/tsdb.newFieldsFromBinary(0xc20806a080, 0x9, 0x10, 0x30)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/tsdb/points.go:1038 +0x8ac
github.com/influxdb/influxdb/tsdb.(*point).unmarshalBinary(0xc2080ae000, 0x30)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/tsdb/points.go:928 +0x3b
github.com/influxdb/influxdb/tsdb.(*point).Fields(0xc2080ae000, 0xc208104120)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/tsdb/points.go:875 +0x28
github.com/influxdb/influxdb/tsdb.(*Shard).validateSeriesAndFields(0xc2080460b0, 0xc2080d02c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/tsdb/shard.go:658 +0x9c9
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc2080460b0, 0xc2080d02c0, 0x1, 0x1, 0x0, 0x0)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/tsdb/shard.go:220 +0x75
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc20802c480, 0x1, 0xc2080d02c0, 0x1, 0x1, 0x0, 0x0)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/tsdb/store.go:281 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x1, 0x1, 0xc2080d02c0, 0x1, 0x1)
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /private/tmp/influxdb20150703-10818-8fgmmk/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b
Versions
$ uname -a
Darwin localhost 14.4.0 Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64 x86_64
$ go version
go version go1.4.2 darwin/amd64
$ influx -version
InfluxDB shell 0.9.0-HEAD
@anryko
Copy link

anryko commented Jul 6, 2015

I have the same issue in v0.9.1.
The issue is in https://github.com/influxdb/influxdb/blob/master/tsdb/points.go#L1035-L1040
According to docs strconv.ParseBool (http://golang.org/pkg/strconv/#ParseBool) can parse "1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False" and NaN is not one of those.

@veverkap
Copy link

veverkap commented Jul 6, 2015

I have this issue as well, but I have graphite on UDP

###
### [[graphite]]
###
### Controls one or many listeners for Graphite data.
###

[[graphite]]
  enabled = true
  bind-address = ":2003"
  protocol = "udp"
  # consistency-level = "one"
  # name-separator = "."

  ## "name-schema" configures tag names for parsing the metric name from graphite protocol;
  ## separated by `name-separator`.
  ## The "measurement" tag is special and the corresponding field will become
  ## the name of the metric.
  ## e.g. "type.host.measurement.device" will parse "server.localhost.cpu.cpu0" as
  ## {
  ##     measurement: "cpu",
  ##     tags: {
  ##         "type": "server",
  ##         "host": "localhost,
  ##         "device": "cpu0"
  ##     }
  ## }
  # name-schema = "type.host.measurement.device"

  ## If set to true, when the input metric name has more fields than `name-schema` specified,
  ## the extra fields will be ignored.
  ## Otherwise an error will be logged and the metric rejected.
  # ignore-unnamed = true

log:

GOMAXPROCS set to 4
2015/07/06 17:56:40 [INFO] raft: Node at 127.0.0.1:8088 [Follower] entering Follower state
2015/07/06 17:56:42 [WARN] raft: Heartbeat timeout reached, starting election
2015/07/06 17:56:42 [INFO] raft: Node at 127.0.0.1:8088 [Candidate] entering Candidate state
2015/07/06 17:56:42 [DEBUG] raft: Votes needed: 1
2015/07/06 17:56:42 [DEBUG] raft: Vote granted. Tally: 1
2015/07/06 17:56:42 [INFO] raft: Election won. Tally: 1
2015/07/06 17:56:42 [INFO] raft: Node at 127.0.0.1:8088 [Leader] entering Leader state
2015/07/06 17:56:42 [INFO] raft: Disabling EnableSingleNode (bootstrap)
2015/07/06 17:56:42 [DEBUG] raft: Node 127.0.0.1:8088 updated peer set (2): [127.0.0.1:8088]
2015/07/06 17:56:42 created local node: id=1, host=127.0.0.1:8088
[httpd] 2015/07/06 17:56:42 listening on HTTP: [::]:8086
[collectd] 2015/07/06 17:56:43 collectd UDP started
[graphite] 2015/07/06 17:56:44 ensured target database graphite exists
[graphite] 2015/07/06 17:56:44 udp Graphite input opened on [::]:2003
[http] 2015/07/06 17:56:54 162.221.25.36 - - [06/Jul/2015:17:56:54 +0000] GET /query?q=SHOW+DATABASES&db= HTTP/1.1 200 105 http://52.0.193.167:8083/ Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36 62c8f17b-2408-11e5-8001-000000000000 1.019332ms
[http] 2015/07/06 17:56:55 162.221.25.36 - - [06/Jul/2015:17:56:54 +0000] GET /query?q=SHOW+DATABASES&db= HTTP/1.1 200 105 http://52.0.193.167:8083/ Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36 635c28f2-2408-11e5-8002-000000000000 1.564707ms
[http] 2015/07/06 17:56:59 162.221.25.36 - - [06/Jul/2015:17:56:59 +0000] GET /query?q=SHOW+MEASUREMENTS&db=collectd HTTP/1.1 200 40 http://52.0.193.167:8083/ Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36 65c9792f-2408-11e5-8003-000000000000 5.647616ms
[http] 2015/07/06 17:56:59 162.221.25.36 - - [06/Jul/2015:17:56:59 +0000] GET /query?q=SHOW+DATABASES&db=collectd HTTP/1.1 200 105 http://52.0.193.167:8083/ Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36 65cf22c2-2408-11e5-8004-000000000000 799.953µs
panic: unable to parse bool value 'NaN': strconv.ParseBool: parsing "NaN": invalid syntax


goroutine 8550 [running]:
github.com/influxdb/influxdb/tsdb.newFieldsFromBinary(0xc208481870, 0x9, 0x10, 0x10)
    /home/philip/build/src/github.com/influxdb/influxdb/tsdb/points.go:1038 +0x8ac
github.com/influxdb/influxdb/tsdb.(*point).unmarshalBinary(0xc208101400, 0x10)
    /home/philip/build/src/github.com/influxdb/influxdb/tsdb/points.go:928 +0x3b
github.com/influxdb/influxdb/tsdb.(*point).Fields(0xc208101400, 0xc20913f590)
    /home/philip/build/src/github.com/influxdb/influxdb/tsdb/points.go:875 +0x28
github.com/influxdb/influxdb/tsdb.(*Shard).validateSeriesAndFields(0xc208be4000, 0xc208722000, 0x61, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/philip/build/src/github.com/influxdb/influxdb/tsdb/shard.go:651 +0x5f7
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc208be4000, 0xc208722000, 0x61, 0x80, 0x0, 0x0)
    /home/philip/build/src/github.com/influxdb/influxdb/tsdb/shard.go:220 +0x75
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc208046480, 0x2, 0xc208722000, 0x61, 0x80, 0x0, 0x0)
    /home/philip/build/src/github.com/influxdb/influxdb/tsdb/store.go:281 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x2, 0x1, 0xc208722000, 0x61, 0x80)
    /home/philip/build/src/github.com/influxdb/influxdb/cluster/points_writer.go:250 +0x28b
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /home/philip/build/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

@shish
Copy link

shish commented Jul 8, 2015

I'm getting this too, is there any workaround while we wait for a new build? Looks like one bad packet kills the whole daemon and it doesn't recover ;(

@ghost
Copy link

ghost commented Jul 8, 2015

@shish you could figure out plugins that send this value and disable or fix those. I decided to postpone my migration to 0.9 until this fix is in stable release.

@shish
Copy link

shish commented Jul 8, 2015

Graphite is the only plugin that I use; disabling that means I have no data at all >_< I'd stick with 0.8, but my 0.8 installation just managed to corrupt itself somehow so I have no data there either...

@veverkap
Copy link

veverkap commented Jul 8, 2015

@shish that was the same problem that I ran into. But the graphite plugin didn't cause this issue for me - only the collectd plugin.

@otoolep
Copy link
Contributor

otoolep commented Jul 8, 2015

@jwilder -- will the latest Graphite PR address this?

@shish
Copy link

shish commented Jul 8, 2015

Just built from source after applying the above "Parse NaN as float" patch and copy-pasted ~/gocodez/bin/influxd to /opt/influxdb/influx -- it's been running for 5 minutes and not crashed, which is considerably better than the crashing after ~5 seconds that 0.9.1.deb had \o/

@jwilder jwilder closed this as completed in a3ab093 Jul 8, 2015
@jwilder
Copy link
Contributor

jwilder commented Jul 8, 2015

@otoolep yes. Not sure if it will fix collectd yet.

@jasonrm
Copy link
Author

jasonrm commented Jul 8, 2015

I've been running 3b815e3 for about 45 minutes now without hitting this issue. Used to crash every 30sec-5min. My guess is that this will also fix collectd, but I won't have time to test it for a few days at best.

@beckettsean beckettsean mentioned this issue Jul 29, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@veverkap @jasonrm @shish @jwilder @otoolep @anryko