Fluent Dashboard

Fluent Dashboard is a Google Spreadsheet that receives Fluentd event logs and display charts from them in real-time.

Features

• It's a Google Spreadsheet: hosted by Google at free, easy to customize and integrate with your business process even for non-programmers. Copy the spreadsheet, click some buttons as in Getting Started, and it's ready to use
• It's real-time: If you hit your nginx from browser, Fluent Dashboard will draw a chart for it within 10 sec. Useful for watching how the system stats and service KPIs are moving when you have game or campaign events etc
• Easy integration with Norikra, a real-time CEP: Demo video and instruction shows it takes only 7 min to build a cluster of Apache Bench > nginx > Norikra > Fluent Dashboard from scratch by using Docker, to draw charts of req/s, cpu and net stats of the nginx server
• Suited for building a Lambda Architecture (coming soon): the dashboard could be used for forming a lambda architecture to get a merged view from 1) real-time continuous query on streaming data and 2) batch-based query on historical and large data set. No more worries for CPU consumption of your Elasticsearch. Stay tuned for future release for the lambda support!

Getting Started

To use the spreadsheet, you need to copy and configure it for receiving Fluentd event by its endpoint URL with the following steps.

1. Open this spreadsheet and select File - Make a copy menu to make a copy of it
2. Copy the URL of the copied spreadsheet to clipboard
3. Select Tools - Script editor... menu
4. On the Script editor, open fluent_listener.gs. Paste the copied URL on the place of <<PLEASE PUT YOUR SPREADSHEET URL HERE>>. Select File - Save menu to save the file
5. Select Publish - Deploy as web app...
6. On the Deploy as web app dialog, enter 1 in the Project version field and click Save New Version, select Anyone, even anonymous on the Who has access to the app menu, and click Deploy button
7. Select the Current web app URL. This is the endpoint URL for receiving event logs from Fluentd. Copy and paste the URL to clipboard or anywhere to use it later
8. Select Run - doPost menu, click Continue button of the Authorization Required dialog and click Accept button on the Request for Permission dialog
9. Confirm that there are a line chart appeared on the spreadsheet. Now it's ready to accept event logs from Fluentd

You can choose one from the two options to use the spreadsheet.

Option A: Use with fluent-plugin-https-json:

In this option, you can use the spreadsheet as a dashboard for any event log collected by Fluentd.

1. Configure a host for Fluentd installation
2. Install fluent-plugin-https-json
3. Edit td-agent.conf to add the following match element. Replace the <<ENDPOINT URL>> with your endpoint URL and edit the ** pattern if needed. Save the file and restart td-agent

    <match **>
        type            https_json
        use_https       true
        buffer_path     /tmp/buffer
        buffer_chunk_limit 256m
        buffer_queue_limit 128
        flush_interval  3s
        endpoint        <<ENDPOINT URL>>
    </match>

Option B: Use with fluentd-norikra-gas Docker image:

In this option, you can use the spreadsheet as a dashboard for Norikra, a Complex Event Processing (CEP) tool that let you use SQL for fast and scalable event log aggregation.

1. Configure a host for Fluentd installation
2. Prepare a Docker environment
3. Execute the following docker command with putting the endpoint URL at the place of <<ENDPOINT URL>>

$ sudo docker run -p 26578:26578 -p 26571:26571 -p 24224:24224 -p 24224:24224/udp -e GAS_URL=<<ENDPOINT URL>> -t -i -d kazunori279/fluentd-norikra-gas

Now the host works as a Fluentd + Norikra server. Configure your Fluentd clients to forward logs to the host, and add Norikra queries by using its Web UI. The query result will be displayed as a new sheet on this spreadsheet. See this site for details of Norikra.

Usage:

• The spreadsheet can only receive one event log per a few seconds for each sheet. You can use it for receiving aggregated statistics (such as req/s, average CPU/memory usage etc) every few seconds, rather than using it for receiving raw streaming events. Recommended event rate is 1 event per 3 seconds for each sheet.
• When the spreadsheet receive an event log with a new tag name, it creates a new sheet with the Fluentd tag name (or Norikra query name)
• If the tag name has a suffix _AREA, _BAR, _COLUMN, _LINE, _SCATTER, or _TABLE, it will also create a new sheet with a specified chart
• If the tag name has a suffix _AREA_STACKED, _BAR_STACKED or _COLUMN_STACKED, it will create a stacked chart
• The endpoint URL does not support authentication. Please make sure to keep the URL secret and not to make it public