Hitting open files limit causes influxdb to create shards in loop

[dgnorton]

Default install on Debian, 0.8.3, no tuning done except changing data dir path.

After InfluxDB hit the open files limit it started to create new shards in the loop, and of course could not write data, resulting in creation of few thousand dirs.
It should either:

• exit, so if it is monitored user can notice before wondering why there is no stats from last shard
• close old/least used shards

Logs:

[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:796) No matching shards for write at time 1413064200000000u, creating...
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:827) createShards for space long_term: start: Tue Oct 7 02:00:00 +0200 CEST 2014. end: Thu Nov 6 01:00:00 +0100 CET 2014
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /var/lib/influxdb/db/shard_db_v2/04960
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1029) AddShards: error setting local store: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).doOrProxyCommandOnce:169) Cannot run command &coordinator.CreateShardsCommand{Shards:[]*cluster.NewShardData{(*cluster.NewShardData)(0xc20983ba40)}, SpaceName:""}. open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).CreateShards:749) RAFT: CreateShards: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit:626) Couldn't write data for continuous query: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files)
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:796) No matching shards for write at time 1413064200000000u, creating...
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:827) createShards for space long_term: start: Tue Oct 7 02:00:00 +0200 CEST 2014. end: Thu Nov 6 01:00:00 +0100 CET 2014
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /var/lib/influxdb/db/shard_db_v2/04961
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1029) AddShards: error setting local store: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).doOrProxyCommandOnce:169) Cannot run command &coordinator.CreateShardsCommand{Shards:[]*cluster.NewShardData{(*cluster.NewShardData)(0xc208edaee0)}, SpaceName:""}. open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).CreateShards:749) RAFT: CreateShards: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit:626) Couldn't write data for continuous query: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files)
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:796) No matching shards for write at time 1413064200000000u, creating...
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:827) createShards for space long_term: start: Tue Oct 7 02:00:00 +0200 CEST 2014. end: Thu Nov 6 01:00:00 +0100 CET 2014
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /var/lib/influxdb/db/shard_db_v2/04962
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1029) AddShards: error setting local store: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).doOrProxyCommandOnce:169) Cannot run command &coordinator.CreateShardsCommand{Shards:[]*cluster.NewShardData{(*cluster.NewShardData)(0xc20a3684d0)}, SpaceName:""}. open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).CreateShards:749) RAFT: CreateShards: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit:626) Couldn't write data for continuous query: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files)

[dgnorton]

@XANi does it create them in a loop or is a client continually writing to it and it creates a new file every time the client attempts to write? Either is bad...just want to make sure I understand what you're seeing.

[XANi]

From what I've debugged:

• clients are continusly writing (we pipe our monitoring data into it)
• at midnight it tries to create new shard
• shard creation failed because of FD limit
• tries to create shard again
• gets into loop creating thousands of shards