save data in temporary file instead of memory #106
Conversation
fippo
force-pushed
the
tempfile
branch
5 times, most recently
from
7331f4e
to
aca9781
Compare
fippo
changed the title
fippo
force-pushed
the
tempfile
branch
2 times, most recently
from
def669c
to
539bb13
Compare
|
deployed, seems to survive longer now... |
fippo
force-pushed
the
tempfile
branch
2 times, most recently
from
9824499
to
003493d
Compare
since snoop crashes with OOM ~150 times a day for me right I am writing things to disk for now.
|
after deleting stale data snoop-server just survived for at least nine hours without dying. This looks good.. |
fippo
changed the title
| generateFeatures(referer, client, clientid); | ||
| delete db[referer][clientid]; | ||
| tempStream.on('finish', function() { | ||
| fs.readFile(tempStream.path, {encoding: 'utf-8'}, function(err, data) { |
There was a problem hiding this comment.
At some point I think we will have to put this reading/processing part in a different process reading the collected files in background and extracting features.
There was a problem hiding this comment.
yeah. I noticed that feature generation can take quite long which skews serverside timestamps.
I'm considering doing the accept in the cluster-master and the feature generation in a couple of workers. Also splitting up the big feature loop such that each feature is a setTimeout.
|
LGTM. Not sure if it is worth starting to use EC2 instances with SSD disk (not sure which ones have it now). Just an idea. |
|
yeah, I am currently running into disk-full issues with writing this to the root disk ;-) |
since snoop crashes with OOM ~150 times a day for me right now I am writing things to disk for now.