Visualizing Chef Data
Thursday, Kirkland, 10:30
- Collecting data out of the Chef Server and visualizing it
- Requirements
- See systems in the context of Chef
- Cloud infrastructure - how many servers deployed
- List of tools
- Traditionally point at monitoring tools
- High level - this is what your stack looks like today, this is where it is going
@Brian Scott API endpoint that runs against a small Sinatra app Stored in MongoDB generated from data bags People who have access to Chef org can see data Visualized in a Django web app Pie charts/Graph goodness Will be open source Uses high charts for graphing, will probably switch to D3 Node data is around 16MB, does not work well with MongoDB - looking at Cassandra Is it necessary to store data? Chef has Chef Analytics want to try that 230 workstations against Chef Server Cluster of 3 nodes handling all that data Hosted solution/SASS
The Foreman does similar stuff for Puppet Will probably go for more multi-provisioner stuff Lifecycle management tool with core reporting in it
OK sending node data off to third parties? Hosted?
Performance data Data that can be visualized from node attributes This unicon instance has 15 instances Visualization of the infrastructure itself Click org, show diagram of topology - all interconnected Issue figuring out what switches are used, using OpenStack OpenVZ/GraphViz/ Display graph what nodes connected to which port Cookbooks list what ports need to be open, display diagram
How are relationships built? Neo4J Want to open source enough by changing it on the side so not Brian Scott proprietery Search built into tool as well - replace entire Chef UI because it does not work By the time someone logs into the Chef UI, can generate whole report in the Brisn Scott system (Chef 12, UI is not installed by default out of the box) Sensu is in there as well with Sensu checks CLA problems with Chef open sourcing
How do you determine database in Chef changed? Define core set of cookbooks to set set stuff up Chef handler from knife plugin, does post to an endpoint, massages the data and writes to MongoDB When you destroy node, how do you get data? Deleting a node triggers a handler to database to write event
How do you trigger the delete? On the back end, use ridley
Oculus rift - geotagging
Does having UI like this creation tension using Chef itself? No Big customer uses chargeback system - all teams Use that as a tool for that What about the technical people? They love it because it is a lot faster than the Chef API
Other people say don't understand Chef because they built an alternative UI. Skinning everything, make it easy and point and click But don't understand what is happening under the hood Chef APIs can edit stuff, can't capture in source control "Unless I can click it, I don't know how it works" chef-guard can solve problem with capturing data in source control Sensitive about how pretty it is preventing command line usage Some parts of the business require GUI
Brian Scott tool, don't use it to manage the Chef infrastructure
chef-browser - another tool Just displays data and shows level of attributes without UI Chef UI is terrible - needs an overhaul UI constant spinning Chef logo waiting Have 230 orgs, really, really slow But Brian Scott does not need a UI, can see why it is not a focus point In a workflow perspective, the UI is useless Go direction of Sensu and drop dashboard completely, let community develop dashboards Security an issue - no back end API filtered by security, turns into n+1 query
To get nodes, must use search - performing more queries than it should at once Shoveling around too much data - need partials and caching
Chef Analytics intentionally not in manage. It's a separate API. Releasing Actions first, will build on top of that Building real language for Actions to utilize message bus for external system integration Integration for Splunk and so forth Next step after that venturing into fields of compliance Reporting dashboard in manage will be ripped out Reporting goes into Analytics You will be able to signal events when they happen Need to use at least Chef 11 and need OCID for analytics Have plans to improve utilization, but overall approach is API driven
What about community analytics? What about analytics on supermarket? Old star rating and downloads was lie - mysql four star cookbook, only 27 ratings Badges in supermarket - passes food critic, has tests, etc. Brian Scott - every time cookbook is submitted, runs stuff against cookbooks and displays table
Methods for visualization No telemetry for EC Would people be interested in getting telemetry for EC as an opt-in feature?
Wants to integrate with Chef Actions Chef Actions - instruments erchef and drops things into RabbitMQ Will post raw messages into RabbitMQ, so could consume that Chef doesn't want you to use the raw messages, as Chef will add more stuff around that, but you could https://github.com/bscott will open source
chef-zero great for doing analytics in isolated environments without good internet connectivity that is still secure What security? Handler pushes data to chef metrics, replace sensitive data like passwords with hashes and 1's, clean up IP addresses as well bison gem in ruby doesn't like periods, so have to massage data anyway ACL system for people to view things granular down to particular items Can share items with easy hyperlink with others to assemble dashboards and customer reports as well
Do password auth against Chef Server? No, use Active Directory. Every five minutes poll network share as a hack with all data have access for speed Plan to integrate with OCID in future OCID should have groups as well Devise authentication gem in ruby might be handy Can report on 20k nodes and more In Chef, query is in Rails, then ingest point is in erlang because it is fast Want to write knife plugin to export data c3 is a wrapper for d3, "framework" for d3, has graphs pre-assembled Diffing two Chef nodes, ScriptRock Guardrail like
Do better at attribute history What changed when and how - identify subset of attributes tree that changed Feature that should be coming in 2015 in Chef Actions Want to also know Chef run failure rate and so forth
What about Chef Server monitoring? Statistics, numbers and graphs Perhaps just a statsd export Released first version of opscode-monitoring with a reflector Doesn't do active checking, just passive Omnibus package drop on chef server, get statsd output - internally Chef Server uses this, but data is going nowhere Sensu will already have a statsd running Everybody has their own monitoring system, just want to forward the statsd data somewhere else Report on which boxes running chef as a daemon
Puppet looks at JMS metrics on server
knife-diff output has changed substantially, would like tool that has standard data format Brian Scott solution for chef metrics runs on chef server host? No runs on separate machine cluster Does this use a queue? Pushes directly to endpoint, endpoint will queue only when stressed to make it is as fast as possible
Idea: Piggyback sensu's rabbitmq implementation, already encrypted
Metrics cluster with MongoDB, memcache cluster and redis cluster Redis-sentinel lets you cluster redis - highly-available redis
Does anyone know about statistically accurate libraries for metrics? Statistical models that are valuable?
Interally Chef has a feed of all actions happening within Chef Server as part of the replication feature This will be rewritten to use the same message bus of Chef Actions
Also chef-graphite
Fastly has visualization library called epic - used for showing graphs of streaming data