Input Plugins OS Support #167

pnovotnak · 2017-01-31T23:11:38Z

It would be nice if some of the unsupported inputs were supported on these platforms;

in_cpu
in_mem
in_kmsg
in_tail
in_proc

I'm not a C developer, or a BSD developer, but I'm willing to start work on the in_tail plugin--I need that one for a work project (collecting logs from a PfSense router). I'm not sure where to begin with this though, so it would be awesome if someone could get me pointed in the right direction.

P.S: I do know Golang fairly well, so I could probably be more useful there if that is a viable route.

The text was updated successfully, but these errors were encountered:

edsiper · 2017-02-01T05:05:10Z

hi @pnovotnak,

thanks for reaching out. If I am not wrong, the in_tail plugin should work on BSD, if you want to test now (I can do it later this week), you can try to remove the plugin from the "Linux" conditional here:

https://github.com/fluent/fluent-bit/blob/master/plugins/CMakeLists.txt#L66

I mean, just move this part outside of the conditional:

REGISTER_IN_PLUGIN("in_tail")

pnovotnak · 2017-02-01T05:07:30Z

Nice! I'll test tonight / tomorrow and post results.

…

On Jan 31, 2017 21:05, "Eduardo Silva" ***@***.***> wrote: hi @pnovotnak <https://github.com/pnovotnak>, thanks for reaching out. If I am not wrong, the in_tail plugin should work on BSD, if you want to test now (I can do it later this week), you can try to remove the plugin from the "Linux" conditional here: https://github.com/fluent/fluent-bit/blob/master/ plugins/CMakeLists.txt#L66 I mean, just move this part outside of the conditional: REGISTER_IN_PLUGIN("in_tail") — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#167 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFctXWMWElLo0GrfXolaJCuSk-SflFWaks5rYBKGgaJpZM4LzS4g> .

pnovotnak · 2017-02-01T21:11:21Z

Seems like wildcards don't work

[2.3.3-DEVELOPMENT][admin@pfSense.localdomain]/root: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p path=/var/log/*.log -p db=~/nginx.log.db -o stdout
/root/fluent-bit-0.10.1/build/bin/fluent-bit: No match.

When specifying a single file by full path, it reads the entire contents of the file in 1-minute intervals;

[2.3.3-DEVELOPMENT][admin@pfSense.localdomain]/var/log: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p path=/var/log/nginx.log -o stdout
Fluent-Bit v0.10.1
Copyright (C) Treasure Data

[2017/02/01 20:51:52] [ info] [engine] started
[0] tail.0: [1485982312, {"log"=>"Feb  1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485982312, {"log"=>"Feb  1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485982312, {"log"=>"Feb  1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485982312, {"log"=>"Feb  1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485982312, {"log"=>"Feb  1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485982312, {"log"=>"Feb  1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485982312, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485982312, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485982312, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485982312, {"log"=>"Feb  1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485982312, {"log"=>"Feb  1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[0] tail.0: [1485982372, {"log"=>"Feb  1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485982372, {"log"=>"Feb  1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485982372, {"log"=>"Feb  1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485982372, {"log"=>"Feb  1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485982372, {"log"=>"Feb  1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485982372, {"log"=>"Feb  1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485982372, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485982372, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485982372, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485982372, {"log"=>"Feb  1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485982372, {"log"=>"Feb  1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[0] tail.0: [1485982432, {"log"=>"Feb  1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485982432, {"log"=>"Feb  1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485982432, {"log"=>"Feb  1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485982432, {"log"=>"Feb  1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485982432, {"log"=>"Feb  1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485982432, {"log"=>"Feb  1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485982432, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485982432, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485982432, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485982432, {"log"=>"Feb  1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485982432, {"log"=>"Feb  1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]

However, when using an sqlite3 database it offsets properly after each read 👍 though it still seems not to actively watch the file, polling it at 1m intervals instead.

[2.3.3-DEVELOPMENT][admin@pfSense.localdomain]/root: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p path=/var/log/nginx.log -p db=/root/nginx.log.db -o stdout
Fluent-Bit v0.10.1
Copyright (C) Treasure Data

[2017/02/01 21:05:12] [ info] [engine] started
[0] tail.0: [1485983112, {"log"=>"Feb  1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485983112, {"log"=>"Feb  1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485983112, {"log"=>"Feb  1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485983112, {"log"=>"Feb  1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485983112, {"log"=>"Feb  1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485983112, {"log"=>"Feb  1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485983112, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485983112, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485983112, {"log"=>"Feb  1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485983112, {"log"=>"Feb  1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485983112, {"log"=>"Feb  1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[0] tail.0: [1485983292, {"log"=>"Feb  1 21:07:58 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:21:07:58 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485983292, {"log"=>"Feb  1 21:08:06 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:21:08:06 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]

edsiper · 2017-02-01T21:17:34Z

For wildcard, modify this:

-p path=/var/log/*.log

to

-p 'path=/var/log/*.log'

on that way we can avoid wildcard expansion in the shell.

In order to check the build flags and internals, would you please run fluent-bit -h and paste the output of the last section called Internal ?

pnovotnak · 2017-02-01T21:22:10Z

Oops! Here they are;

[2.3.3-DEVELOPMENT][admin@pfSense.localdomain]/root: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p 'path=/var/log/*.log' -o stdout
Fluent-Bit v0.10.1
Copyright (C) Treasure Data

[2017/02/01 21:14:27] [ info] [engine] started
[2017/02/01 21:14:29] [error] [:328 errno=2] No such file or directory
[2017/02/01 21:14:29] [error] [:328 errno=2] No such file or directory
[0] tail.0: [1485983667, {"log"=>"Feb  1 20:30:10 pfSense dhclient: PREINIT"}]
[1] tail.0: [1485983667, {"log"=>"Feb  1 20:30:10 pfSense dhclient[5178]: DHCPDISCOVER on em0 to 255.255.255.255 port 67 interval 1"}]
...

[2.3.3-DEVELOPMENT][admin@pfSense.localdomain]/root: fluent-bit-0.10.1/build/bin/fluent-bit -h
Usage: fluent-bit [OPTION]

Available Options
  -c  --config=FILE	specify an optional configuration file
  -d, --daemon		run Fluent Bit in background mode
  -f, --flush=SECONDS	flush timeout in seconds (default: 5)
  -i, --input=INPUT	set an input
  -m, --match=MATCH	set plugin match, same as '-p match=abc'
  -o, --output=OUTPUT	set an output
  -p, --prop="A=B"	set plugin configuration property
  -e, --plugin=FILE	load an external plugin (shared lib)
  -l, --log_file=FILE	write log info to a file
  -t, --tag=TAG		set plugin tag, same as '-p tag=abc'
  -v, --verbose		enable verbose mode
  -q, --quiet		quiet mode
  -V, --version		show version number
  -h, --help		print this help

Inputs
  tail                  Tail files
  head                  Head Input
  health                Check TCP server health
  serial                Serial input
  stdin                 Standard Input
  tcp                   TCP
  mqtt                  MQTT, listen for Publish messages
  forward               Fluentd in-forward
  random                Random

Outputs
  counter               Records counter
  es                    Elasticsearch
  forward               Forward (Fluentd protocol)
  http                  HTTP Output
  influxdb              InfluxDB Time Series
  null                  Throws away events
  plot                  Generate data file for GNU Plot
  stdout                Prints events to STDOUT
  td                    Treasure Data
  flowcounter           FlowCounter

Internal
 Event Loop  = select
 Build Flags =  JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB FLB_HAVE_FLUSH_LIBCO FLB_HAVE_FORK FLB_HAVE_PROXY_GO FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4

edsiper · 2017-02-01T21:26:08Z

thanks.

So if I understand correctly, in_tail after a long period is not longer watching the files ?

notes: BSD system using select(2) backend to watch logs.

pnovotnak · 2017-02-01T21:47:59Z

Ah, yes, someday I will learn how to read. I suppose supporting every kernel starts to become a rabbithole. Looks like the in_tail plugin works! Thanks @edsiper!

edsiper · 2017-02-01T21:50:04Z

awesome!. Before to close the issue, would you please describe your use case, is it for production ?

pnovotnak · 2017-02-01T22:35:47Z

Sure. We want to collect logs from our pfSense box and aggregate them in Stackdriver so that we can have offsite log backups, do alerting, etc. Target is production, but "production" here means 5 users on a wlan + 2 headless workstations, so it's a pretty small deployment.

edsiper · 2018-02-19T20:17:13Z

Closing this ticket as "help wanted", we need more contributors to implement specific OS stuff.

edsiper added the help wanted label Feb 19, 2018

edsiper closed this as completed Feb 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Input Plugins OS Support #167

Input Plugins OS Support #167

pnovotnak commented Jan 31, 2017

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017 via email

pnovotnak commented Feb 1, 2017 •

edited

Loading

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017 via email •

edited

Loading

edsiper commented Feb 19, 2018

Input Plugins OS Support #167

Input Plugins OS Support #167

Comments

pnovotnak commented Jan 31, 2017

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017 via email

pnovotnak commented Feb 1, 2017 • edited Loading

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017

edsiper commented Feb 1, 2017

pnovotnak commented Feb 1, 2017 via email • edited Loading

edsiper commented Feb 19, 2018

pnovotnak commented Feb 1, 2017 •

edited

Loading

pnovotnak commented Feb 1, 2017 via email •

edited

Loading