InfluxDB support #2208

Merged
merged 331 commits into from Jan 6, 2016

Projects

None yet
@laf
Member
laf commented Oct 26, 2015

This is extremely alpha at present - see the doc for more details.

I've not added the library used to Makefile as it needs compiling with composer to create the necessary files used here.

wdalmut and others added some commits Mar 2, 2015
@wdalmut wdalmut Merge branch 'release/0.3.0' f8f33bd
@wdalmut wdalmut Merge tag '0.3.0' into develop
Added HTTP Adapter thanks to @lekz
6d9bd14
@wdalmut wdalmut Fixes #14 - guzzle http json_decode -> assoc arrays 06ae666
@wdalmut wdalmut Merge pull request #15 from corley/hotfix/14
Fixes #14 - guzzle http json_decode -> assoc arrays
4737d2a
@wdalmut wdalmut Bumped version 0.3.1 704ea74
@wdalmut wdalmut Merge branch 'release/0.3.1' b690eb9
@wdalmut wdalmut Merge tag '0.3.1' into develop
Fixes ColumnsPointsFilter json decode error
c6d4ee4
@gianarb gianarb phpunit configuration improvement
This feature implement env parameters for phpUnit configuration.
This upgrade is a first step for multi influxdb version testing
e125b29
@wdalmut wdalmut Contributing doc file 16197e2
@wdalmut wdalmut Files should end with a newline character 4140300
@wdalmut wdalmut Bumped version 0.3.2 4fa3ad0
@wdalmut wdalmut Merge branch 'release/0.3.2' 2aec8b4
@wdalmut wdalmut Merge tag '0.3.2' into develop
Fixes file end with newline
9da406a
@gianarb gianarb Remove filters e369530
@gianarb gianarb Composer improvement Guzzle 5 8481e78
Gianluca Arbezzano Merge pull request #25 from gianarb/feature/guzzle-5
Composer improvement Guzzle 5
8fb012f
@gianarb gianarb Dump release 0.3.3 d677fa9
@gianarb gianarb Merge branch 'release/0.3.3' 85783dd
@gianarb gianarb Merge branch 'release/0.3.3' into develop 731498e
Gianluca Arbezzano Merge pull request #21 from corley/feature/remove-filters
Remove filters
00b634b
@gianarb gianarb Add explicit PHP version into composer.json 4633336
Gianluca Arbezzano Merge pull request #20 from corley/feature/composer-php-version
Add explicit PHP version into composer.json
27811cb
@wdalmut wdalmut Restore HTTP adapter for InfluxDB 0.9 37da860
@gianarb @wdalmut gianarb Update travis configuration to install InfluxDB 0.9rc30
Conflicts:
	.travis.yml
a5e5da5
@wdalmut wdalmut Restore base behavior fd28135
@wdalmut wdalmut Dropped time precision feature 4f3fdcb
@wdalmut wdalmut Added test for direct messsages with HTTP client 72b0856
@wdalmut wdalmut Added test UDP/IP supports direct messages 63207b4
@wdalmut wdalmut Updated docs 311411d
@wdalmut wdalmut Removed filter benchmarks b65eb99
@wdalmut wdalmut Updated benchmarks and bench results 70c0f57
@wdalmut wdalmut Prepare new travis-ci runner 9ed7fe6
@wdalmut wdalmut Fixes install file name d43f449
@wdalmut wdalmut Added tags test case 35b8dd0
@wdalmut wdalmut Use prepared default options
In order to use user prepared options the message is alse merged with
all user preapared options.
f8bb3fc
@wdalmut wdalmut Added retention policy as option 4fa0393
@wdalmut wdalmut Complete refactor in order to support RC-32
The influxdb RC-32 is quite strange, uses a UDP/IP line protocol with
batch updates.

This commit will cover the base UDP/IP protocol as first implementation
94fa4e6
@wdalmut wdalmut Updated docs 270154f
@wdalmut wdalmut Rename data provider with a more valid name 4f94172
@wdalmut wdalmut Added tests for global tags and retention policies 5a162a9
@wdalmut wdalmut Updated docs 654b2c7
@wdalmut wdalmut Removed exceptions 6f80454
@wdalmut wdalmut Moved to PSR-4
Autoload from PSR-0 to PSR-4
9db3509
@wdalmut wdalmut License to 2015 8dffa27
@wdalmut wdalmut Removed bootstrap from phpunit.xml
It was not a bootstrap file
f95e32e
@wdalmut wdalmut Bench inline protocol conversion f185935
@wdalmut wdalmut Test also multiple measurement with UDP/IP 048e308
@wdalmut wdalmut Version 0.4.0 f5a71a5
@wdalmut wdalmut Client dependency with construction injection
Removed the adapter setter injection in favor or constructor injection
38f65a7
@wdalmut wdalmut Test refactoring
Removed UDP/IP adapter construction (duplication)
ff6f865
@wdalmut wdalmut Fixes invalid saved timestamp for UDP/IP adece73
@wdalmut wdalmut Moved to InfluxDB RC33 245a911
@wdalmut wdalmut Merge pull request #29 from corley/feature/influxdb-0.9
Support InfluxDB v0.9

Actually Release Candidate 33 is supported
8112a60
@wdalmut wdalmut Minor fixes on docs 7c72de4
@wdalmut wdalmut Travis updated with vv0.9 b02a6ef
@wdalmut wdalmut Bumped version 0.4.1 ba4dc0a
@wdalmut wdalmut Merge branch 'release/0.4.1' ca2a8f3
@wdalmut wdalmut Merge tag '0.4.1' into develop
Support version 0.4.1
be237b9
@wdalmut wdalmut Fixes minor violations
 * Removed unsed `use` statemment
 * Moved helper http url composer to `GuzzleAdapter` (responsibility)
 * Removed unused `clearTimePrecision` method
 * Updated composer
63b8bb4
@wdalmut wdalmut Bumped version 0.4.2 473d21e
@wdalmut wdalmut Merge branch 'release/0.4.2' b35c4d1
@wdalmut wdalmut Merge tag '0.4.2' into develop
Minor violations fixes
770cc96
@wdalmut wdalmut Fixes time generation for udp/ip packets
Uses microtimes instead of Datetime
c162760
@wdalmut wdalmut Bumped version 0.4.3 b2f044f
@wdalmut wdalmut Merge branch 'release/0.4.3' 799b523
@wdalmut wdalmut Merge tag '0.4.3' into develop
Fixes UDP/IP send multiple metrics (time overlaps)
Tries to compute nanoseconds in order to fixes time serie overlaps.
0626452
@wdalmut wdalmut Allows non write adapters
 * Renamed `AdapterInterface` to `WritableInterface`
 * Constructor now allows different adapter implementations
 * Check that adapter implements `WritableInterface` during data send
c52377a
@wdalmut wdalmut Removed useless `return $this` during constructor 452b50f
@wdalmut wdalmut Removed deprecated from GuzzleAdapter dd255ad
@wdalmut wdalmut Merge pull request #30 from corley/feature/allows-non-write-adapters
Allows non write adapters
88b9e9c
@wdalmut wdalmut Fixes benchmark procedures 18442a6
@wdalmut wdalmut Bumped version 0.4.4 5d48787
@wdalmut wdalmut Merge branch 'release/0.4.4' 8a0c4ea
@wdalmut wdalmut Merge tag '0.4.4' into develop
Minor on adapter names
7a9bfa2
@wdalmut wdalmut Minor on docs b676703
@wdalmut wdalmut Removed senso insight (shows no medal when kudos) e4c3888
@wdalmut wdalmut Moved sockets compile section to bottom 06fe10b
@wdalmut wdalmut Bumped version 0.4.5 c04de6d
@wdalmut wdalmut Merge branch 'release/0.4.5' d238c49
@wdalmut wdalmut Merge tag '0.4.5' into develop
Just docs recap
07e5ec3
@thecodeassassin thecodeassassin Intitial commit 93ed031
@thecodeassassin thecodeassassin Added license and readme files d9c2ebd
@beckettsean beckettsean Update README.md acf8429
Daniel Martinez Adding unit tests 0986f43
Daniel Martinez ResultSet - implementing getPoints d6dc5d5
Daniel Martinez Testing ResultSet 413cbf6
Daniel Martinez ResultSet - getSeries 016c5ab
@thecodeassassin thecodeassassin Intermediate commit; added querybuilder; retention policy object 87121d2
@danibrutal danibrutal Changing measurement to name in result set 7d91bb0
@danibrutal danibrutal ResultSet::getPoints empty parameters 033d593
@danibrutal danibrutal Throws InfluxDBClientError if there was an error after quering d55af50
@danibrutal danibrutal Throws Exception in constructor for error in first serie 72cd8b8
@thecodeassassin thecodeassassin Added list methods, cleaned up API 43c89a1
@danibrutal danibrutal Testing database 7421989
@danibrutal danibrutal Adding Point class to handle Points aa25eae
@thecodeassassin thecodeassassin Fixed query builder last method, added median method; fixed adding re…
…tention policies
d625efd
@danibrutal danibrutal Changing Client Exception, testing Point 20e2527
@danibrutal danibrutal ResultSet bugfix if tags key is missing in the result e4f5bb9
@danibrutal danibrutal Writing Points tests 90277f1
@danibrutal danibrutal Client - write method 6b8d037
@paszczus

WOW !!!

@laf
Member
laf commented Oct 26, 2015

@f0o mind getting me ssh access to this CI or running a poller debug to see why it's stopped running.

@laf
Member
laf commented Nov 1, 2015

Bump

@f0o
Member
f0o commented Nov 1, 2015

Looking very nice on the CI

@f0o
Member
f0o commented Nov 4, 2015

👍 from here, Only thing I dislike is the subtree'ing/squashing of the libs :S

@f0o f0o added the Core label Nov 4, 2015
@laf
Member
laf commented Nov 4, 2015

I hate subtreeing in general, 1,000's of files when only a handful are needed :(

@f0o
Member
f0o commented Nov 6, 2015

CI running for 3 days now, all good 👍

I'd say merge and tag as Beta feature :)
opinions?

@laf
Member
laf commented Nov 8, 2015

Bump

@laf
Member
laf commented Nov 12, 2015

Bump

@f0o
Member
f0o commented Nov 15, 2015

Bump :)

@SaaldjorMike
Member

I briefly tried installing InfluxDB to test it out, but seems like I have some issues which may just be me not setting it up properly.

Running the poller returns the following error:

PHP Fatal error:  Uncaught exception 'InfluxDB\Exception' with message 'Error creating resource: [message] fopen(http://127.0.0.1:8086/write?db=librenms&precision=n): failed to open stream: HTTP request failed! 
[file] /opt/librenms/lib/influxdb-php/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php
[line] 244' in /opt/librenms/lib/influxdb-php/src/InfluxDB/Database.php:149
Stack trace:
#0 /opt/librenms/includes/influxdb.inc.php(70): InfluxDB\Database->writePoints(Array)
#1 /opt/librenms/includes/polling/functions.inc.php(271): influx_update(Array, 'poller-perf', Array, Array)
#2 /opt/librenms/poller.php(122): poll_device(Array, Array)
#3 {main}
  thrown in /opt/librenms/lib/influxdb-php/src/InfluxDB/Database.php on line 149

Config.php:

$config['influxdb']['enable'] = true;
$config['influxdb']['transport'] = 'http';
$config['influxdb']['host'] = '127.0.0.1';
$config['influxdb']['port'] = '8086';
$config['influxdb']['db'] = 'librenms';
$config['influxdb']['username'] = 'admin';
$config['influxdb']['password'] = 'admin';

Opening the CLI to InfluxDB shows me that at least something was inserted:

Connected to http://localhost:8086 version 0.9.4.2
InfluxDB shell 0.9.4.2

> use librenms
Using database librenms

> show measurements
name: measurements
------------------
name
poller-perf
processors
uptime

> select * from uptime limit 1
name: uptime
------------
time                            hostname                        uptime
2015-11-12T18:40:02.75397075Z   switch01.example.org  2.497232e+07

Any suggestions?

@laf
Member
laf commented Nov 24, 2015

I've wiped my LibreNMS install and still can't replicate your issue :(

What distro + php version are you on @SaaldjorMike?

@SaaldjorMike
Member

Currently I can't seem to reproduce it. My guess would be that my inexperience with InfluxDB is the issue here. I am currently running the following which is the same system as I tested on previously:

[root@ubuntu-librenms librenms (master ✗)]$ php --version     
PHP 5.5.9-1ubuntu4.14 (cli) (built: Oct 28 2015 01:34:46) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
[root@ubuntu-librenms librenms (master ✗)]$ lsb_release -a    
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.2 LTS
Release:        14.04
Codename:       trusty
@laf
Member
laf commented Nov 25, 2015

Still can't replicate the issue @SaaldjorMike. Tested on Ubuntu with the following versions:

root@ubuntu-librenms:/opt/librenms# php --version
PHP 5.5.9-1ubuntu4.11 (cli) (built: Jul  2 2015 15:23:08)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
root@ubuntu-librenms:/opt/librenms# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:    14.04
Codename:   trusty

Anyone else care to comment on this @librenms/reviewers

@Rosiak
Contributor
Rosiak commented Nov 26, 2015

I might have time in the weekend to give this a spin.

@laf laf Removed influxdb-php library
05e3c58
@laf laf Added influxdb library
8e5d619
@f0o
Member
f0o commented Dec 7, 2015

b..u..m..p..?..

@laf
Member
laf commented Dec 8, 2015

It's good to go from my side.

@paulgear
Member

We seem to be missing influx from the recently-merged Ceph support.

@paulgear
Member

Sorry - file for the last comment is ./includes/polling/applications/ceph.inc.php; same goes for ./includes/polling/applications/proxmox.inc.php.

@paulgear
Member

./includes/polling/os/fortigate.inc.php has 'fortigate)sessions' - probably should be 'fortigate_sessions'

@paulgear
Member

Other places which seem to be missing influx:
./includes/polling/os/ciscowlc.inc.php
./includes/polling/os/pulse.inc.php
./includes/polling/functions.inc.php (line containing $ping_rrd)
I think this demonstrates why we should be replacing rrdtool_update (and rrd_create_update) with a function that handles both rrdtool and influxdb. That way we will know that we have 100% coverage of polling data.

@laf laf closed this Dec 12, 2015
@Rosiak
Contributor
Rosiak commented Dec 12, 2015

@librenms/core-team
I really think Influx is the way we need to go, I think we need to discuss the future of this one next time we speak together.

@laf laf Fixed fortigate sessions
ee226c0
@laf laf reopened this Dec 13, 2015
@laf
Member
laf commented Dec 13, 2015

I've re-opened this with the fix for fortigate_sessions.

The missing influx calls can be added at a later stage. Some of these are missing purely because of the length of time this PR has been open for.

Merging influx and rrd calls I consider outside of the scope of this PR. The core of this PR doesn't tread on the toes of the existing code so should be relatively safe, centralising the code would make this a risky merge as it could affect existing installs.

@Rosiak
Contributor
Rosiak commented Dec 16, 2015

Working fine on my dev machine.
👍

laf added some commits Dec 18, 2015
@laf laf Influx update 3b976ba
@laf laf More influx changes
d48aa27
@f0o
Member
f0o commented Dec 20, 2015

bump :)

@f0o f0o added the Rebase Needed label Dec 20, 2015
@laf
Member
laf commented Dec 20, 2015

Needs a rebase again now, will do so when this is going to be merged.

@Rosiak
Contributor
Rosiak commented Dec 20, 2015

I suppose we are waiting on @paulgear's go.

@paulgear
Member

I don't think this needs to block on me. I've given my opinions above. @laf are you still concerned about the size of the influxdb-php library? If so, we should work that out before we merge.

@laf
Member
laf commented Dec 20, 2015

Not the size but more using git subtree but that's now a mute point as trying to tidy up this PR by cherry picking the commits I need just lead to lots of merge conflicts so as it stands then this PR is presented now as is aside from the last rebase that will be needed.

@f0o
Member
f0o commented Dec 21, 2015

With the new daily we can start using submodules, that reduces the size greatly

@laf
Member
laf commented Dec 21, 2015

The issue is picking this PR apart now to get what I need from it.

@paulgear
Member

If you're happy for me to do it, I'll have a try over the holidays to get just the code we need from influxdb-php. If we don't really need it, I'd rather not bloat the repository.

@laf
Member
laf commented Dec 23, 2015

Feel free. Running this at work now so my needs are filled I guess.

@f0o
Member
f0o commented Dec 27, 2015

I'd like to see this merged. It's really useful and in future we can think of a generic feeder function that will multiplex and/or route datasources (influx/mysql/rrd/...)

@Rosiak
Contributor
Rosiak commented Dec 27, 2015

I suggest that we set a date, if no objections are made then we merge at that date.

@laf
Member
laf commented Dec 27, 2015

You have no complaints from me for a merge. Running this currently at work now:

image

@paulgear
Member

Give me until new year's day - I'm going to have a shot at reducing the size of the influxdb-php part.

@paulgear
Member
paulgear commented Jan 1, 2016

After looking at this, I don't think it's worthwhile to try to reduce the size of the influxdb-php install. I did find one other minor issue, which is that tags with spaces are not escaped. I worked around it like this:

diff --git a/includes/influxdb.inc.php b/includes/influxdb.inc.php
index 2f844e2..4970d7f 100644
--- a/includes/influxdb.inc.php
+++ b/includes/influxdb.inc.php
@@ -46,6 +46,8 @@ function influx_update($device,$measurement,$tags=array(),$fields) {
     $tmp_fields = array();
     $tmp_tags['hostname'] = $device['hostname'];
     foreach ($tags as $k => $v) {
+        // collapse all whitespace in tags to a single escaped space
+        $v = preg_replace("/(\s+)/", '\\ ', $v);
         $tmp_tags[$k] = $v;
     }
     foreach ($fields as $k => $v) {

I'm very keen to

  1. collapse influxdb & rrd support into a single function
  2. allow graphing from influxdb
  3. include MIB poller data in influxdb

but these can all happen after this is merged. 👍 from me!

@f0o
Member
f0o commented Jan 1, 2016

Personally I'd like to see a generic data store/get function that then dispatches on the reapective backend. Howeven backends must be multiplexed so they are AND.

I'll happily contribute this after merge :)

@laf
Member
laf commented Jan 4, 2016

You may have checked an older commit, a replace was added in this commit:

laf@32f5980

To take care of the special chars that influx says you can't have in a tag.

@laf laf Rebased
0f33f6d
@laf laf removed the Rebase Needed label Jan 4, 2016
@laf
Member
laf commented Jan 4, 2016

Rebased

@Rosiak
Contributor
Rosiak commented Jan 4, 2016

All good here 👍

@laf laf merged commit 1744529 into librenms:master Jan 6, 2016

2 checks passed

Auto-Deploy Build finished. No test results found.
Details
Scrutinizer 1633 Issues, 19 Patches
Details
@laf laf deleted the laf:influx branch Jan 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment