Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cassandra/Scylla export plugin #857

Closed
nicolargo opened this issue Apr 29, 2016 · 2 comments
Closed

Add Cassandra/Scylla export plugin #857

nicolargo opened this issue Apr 29, 2016 · 2 comments
Milestone

Comments

@nicolargo
Copy link
Owner

nicolargo commented Apr 29, 2016

Description

The goal of this feature request is to export stats to a Cassandra (or Scylla) database.

Data model

Cassandra can store up to 2 billion columns per row

Have a look on the second pattern: https://academy.datastax.com/resources/getting-started-time-series-data-modeling

Resources

Python lib: https://pypi.python.org/pypi/cassandra-driver
Python lib sample: https://github.com/nicolargo/pythonarena/blob/master/cassandra/client01.py

Data model:

=> Datamodel TBD: primary key ?

For test bed:

@nicolargo nicolargo added this to the 2.7 milestone Apr 29, 2016
@nicolargo
Copy link
Owner Author

nicolargo commented Jun 9, 2016

Testbed: https://github.com/nicolargo/pythonarena/blob/master/cassandra/testbd.ipynb

Implementation started in a dedicated branch.

Pb: 2016-06-09 14:03:51,667 -- ERROR -- Cannot export docker stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value '857c72aeaada691aa6060d27e4b083380b10b33ddc30df5b1af0d4554fd08865' is not of type float")
2016-06-09 14:03:51,670 -- ERROR -- Cannot export ip stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value '80.15.130.7' is not of type float")
2016-06-09 14:03:51,675 -- ERROR -- Cannot export network stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value 'docker0' is not of type float")
2016-06-09 14:03:51,676 -- ERROR -- Cannot export percpu stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value 'cpu_number' is not of type float")
2016-06-09 14:03:51,679 -- ERROR -- Cannot export system stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value 'Linux' is not of type float")
2016-06-09 14:03:51,686 -- ERROR -- Cannot export diskio stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value 'dm-1' is not of type float")
2016-06-09 14:03:51,687 -- ERROR -- Cannot export fs stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value 'mnt_point' is not of type float")
2016-06-09 14:03:51,691 -- ERROR -- Cannot export sensors stats to Cassandra (Error from server: code=2200 [Invalid query] message="Invalid map literal for stat: value 'Core 1' is not of type float")

The data model is the following:

session.execute("CREATE TABLE %s (plugin text, time timeuuid, stat map<text,float>, PRIMARY KEY (plugin, time))" % self.table)

==>

If you want untyped data, store blobs. Or store in a different column
(they're free when empty, after all). Type safety is considered a good
thing by many.

==>

String value will not be stored in the database. All numbers (including Boolean) will be stored as float.

@nicolargo
Copy link
Owner Author

Done in the latest commit (DEVELOP BRANCH)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant