Influxdb 0.13 runs out of memory

[johnjelinek]

Bug report

System info: [0.13.0, Ubuntu 16.04, 32GB RAM]

Board conversation: https://groups.google.com/forum/#!topic/influxdb/fiQzEy1aWP8

Steps to reproduce:

1. Start with a large CSV (it's free) (1.6GB in size)
2. Unzip and add headers to the CSV: echo '"Symbol","Date","Open","High","Low","Close","Volume","Ex-Dividend","Split Ratio","Adj. Open","Adj. High","Adj. Low","Adj. Close","Adj. Volume"' | cat - WIKI_20160816.csv | sponge WIKI_20160816.csv

CSV Data sample:

"Symbol","Date","Open","High","Low","Close","Volume","Ex-Dividend","Split Ratio","Adj. Open","Adj. High","Adj. Low","Adj. Close","Adj. Volume"
A,1999-11-18,45.5,50.0,40.0,44.0,44739900.0,0.0,1.0,43.471809559155,47.771219295775,38.21697543662,42.038672980282,44739900.0
A,1999-11-19,42.94,43.0,39.81,40.38,10897100.0,0.0,1.0,41.025923131212,41.083248594367,38.035444803296,38.580036703268,10897100.0
A,1999-11-22,41.31,44.0,40.06,44.0,4705200.0,0.0,1.0,39.468581382169,42.038672980282,38.274300899775,42.038672980282,4705200.0
A,1999-11-23,42.5,43.63,40.25,40.25,4274400.0,0.0,1.0,40.605536401409,41.685165957493,38.455831533099,38.455831533099,4274400.0
A,1999-11-24,40.13,41.94,40.0,41.06,3464400.0,0.0,1.0,38.341180606789,40.070498745296,38.21697543662,39.22972528569,3464400.0
A,1999-11-26,40.88,41.5,40.75,41.19,1237100.0,0.0,1.0,39.057748896226,39.650112015493,38.933543726057,39.353930455859,1237100.0
A,1999-11-29,41.0,42.44,40.56,42.13,2914700.0,0.0,1.0,39.172399822536,40.548210938254,38.752013092733,40.25202937862,2914700.0
A,1999-11-30,42.0,42.94,40.94,42.19,3083000.0,0.0,1.0,40.127824208451,41.025923131212,39.115074359381,40.309354841775,3083000.0
A,1999-12-01,42.19,43.44,41.88,42.94,2115400.0,0.0,1.0,40.309354841775,41.503635324169,40.013173282141,41.025923131212,2115400.0

14288591 points in total

1. Import the CSV into influxdb (example is using csv-to-influxdb): csv-to-influxdb -m bars -t Symbol -ts Date -tf 2006-01-02 -d eodbars WIKI_20160816.csv
2. Try some queries:
3. SELECT MAX(Open) AS Open, MAX(Close) AS Close, MAX(Volume) AS Volume, (MAX(Close) - MAX(Open)) / MAX(Open) * 100 AS PctChg INTO newBars FROM bars WHERE time < now() GROUP BY time(1d), Symbol
4. SELECT Close / Open INTO pctChg FROM bars
5. SELECT * FROM bars LIMIT 1

Expected behavior: The queries should work or tell me that I can't because it's too much.

Actual behavior: Out of Memory and influxdb restarts

[pnovotnak]

Also seeing this issue

[jwilder]

You likely need to alter your retention policy (before inserting) with a larger shard group duration. The sample data your showed is creating very sparse data and lots of shards. The 1.0 nightlies have some fixes that reduce memory in those cases, but increasing your shard group duration to 10y should help.