Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate the performance of the disk serializer #76

Closed
marpaia opened this issue Aug 28, 2014 · 0 comments
Closed

Investigate the performance of the disk serializer #76

marpaia opened this issue Aug 28, 2014 · 0 comments
Labels
bug ready for review Pull requests that are ready to be reviewed by a maintainer

Comments

@marpaia
Copy link
Contributor

marpaia commented Aug 28, 2014

Serializing into a HistroicalQurtyResult type is slower than it should be. Make sure the scheme is correct. If necessary, break it into multiple keys in the column family and use thrift's serialization to store it (http://techblog.ridewithvia.com/post/37118173720/take-care-when-serializing-and-deserializing-thrift).
@marpaia marpaia added the bug label Aug 28, 2014
@marpaia marpaia changed the title Investigate the performance of disk serializer Investigate the performance of the disk serializer Aug 28, 2014
@marpaia marpaia modified the milestone: Deploy on OS X Sep 2, 2014
@marpaia marpaia self-assigned this Sep 2, 2014
marpaia added a commit that referenced this issue Sep 2, 2014
@marpaia
Fix performance issue with the disk serializer
66a2a6f 
This is the issue noted in #76. Keeping all historical results of
queries in the HistoricalQueryResults struct makes serializing and
deserializing those structs very, very slow as time goes on. By only
storing the last execution of the query, we keep the performance
constant, but we kill the feature where osquery can rebuild timelines
without accessing logs. After talking it over, we decided that this
isn't actually that big of a deal because, if you really wanted to
rebuild the old data, you should be able to process the logs, similarly
to bin log replication in MySQL.
@marpaia marpaia mentioned this issue Sep 2, 2014
Merged
@marpaia marpaia closed this as completed Sep 2, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug ready for review Pull requests that are ready to be reviewed by a maintainer
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@marpaia