Running SyslogMonitor

[MrRJ45]

What is the best way to run the Syslog Monitor? I have tried this:

[program:oxidized-syslog]
command=/var/lib/gems/1.9.1/gems/oxidized-0.4.1/extra/syslog.rb
autostart=true
autorestart=true
user=oxidized
stderr_logfile=/var/log/oxidized-syslog.err.log
stdout_logfile=/var/log/oxidized-syslog.out.log

It fails with this:

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require': cannot load such file -- ./rest_client (LoadError) from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /var/lib/gems/1.9.1/gems/oxidized-0.4.1/extra/syslog.rb:17:in `

'

Hard coding the full path to rest_client.rb works fine, but all of this is version specific.

What would be nice would be to have /usr/local/bin/oxidized-syslog which can be ran and doesnt depend on the version number or hard coding paths :)

[MrRJ45]

The ability to also pass arguments would be great too, for example, running using port 1514 doesn't require extra privileges.

[ytti]

I've not tried the example more less since initial commit, so it's not exactly user-friendly right now, it more explains how you can benefit from some oxidized features.
Perhaps it should be rolled in and be more 'productized', so it would work generally.

So first question, does it work, at all, today? I understand that as you're doing init script for it, you've been able to succesfully run it, which makes me quite impressed already. You're getting actual user in git (git blame) and all? I'm amazed if we've not changed the API enough to break it.

Perhaps ea7503d fixes your issue?

[MrRJ45]

ea7503d does fix that, thank you :) I've changed syslog.rb and it starts, and listens on the port.

I'll test it shortly and let you know what happens!

[ytti]

How about 02b3b0f does it fix your UDP port issue?

[MrRJ45]

02b3b0f works fine, if you pass a port number. It should default to 514 if no argument is given.

[ytti]

Fixed in fb0e4e1

[MrRJ45]

I have set up a 2960 to log the config changes, and SyslogMonitor is getting this as I saw an error connecting to port 8888 (I have it set to bind to the public IP).

I now have it binding on 0.0.0.0:8888 but I see nothing in the logs - what is the best way to start debugging this?

[ytti]

I'd first check in 'def ios' what are values you're sending to the rest client.

What it should do is get syslog entry from far-end, get out information who made the config change and to which node, then it should ask oxidized via HTTP to fetch this node next and would pass the username as argument.

Then when the node has finished your output model can consume this information, and git output will, git output will mark the user seen in syslog as the committer of the change.

[MrRJ45]

Ah, I see the issue. getname is not resolving the name, because we have no PTR for these IP's.

I guess this relates to Issue #78

[ytti]

Right, and you now actually have name there as 'primary key', and it can't find your node? If you had IP as primary key, it would still working.

Maybe there could be API call for /next_by_ip/XYZ so it wouldn't rely on finding by name.

[MrRJ45]

Perhaps /nodes/next/ could accept both a name and IP address? If it is a valid IP then use that, otherwise assume it is a name.

That I think would help stop any future changes breaking it :)

[ytti]

Here is where the matching happens:
https://github.com/ytti/oxidized/blob/master/lib/oxidized/nodes.rb#L114

So I suppose something like

index { |e| e.name == node or e.ip == node}

might fix it, and insha'allah doesn't break anything else...

[MrRJ45]

That seems to work fine, I can pass "/node/next/" either the IP, or the name and it queues it for the next device.

Syslog doesnt work correctly though, I dont see it queued. It is posting to the correct URL, the data it is sending is:

{"user":"<username>","from":"<ipv4>","model":"ios","ip":"<ipv4>","name":"<ipv4>"}

Where can we start to debug this? :)

[ytti]

So what should happen, and has worked, at least loong time ago is:

1. we receive syslog message via udp - https://github.com/ytti/oxidized/blob/master/extra/syslog.rb#L88
2. we determine if sender is ios or junos - https://github.com/ytti/oxidized/blob/master/extra/syslog.rb#L76
3. we call rest http library with data - https://github.com/ytti/oxidized/blob/master/extra/syslog.rb#L61
4. we do HTTP PUT and send the data as JSON - https://github.com/ytti/oxidized/blob/master/extra/rest_client.rb#L21
5. we receive the HTTP PUT and call Nodes#next with the data - https://github.com/ytti/oxidized-web/blob/master/lib/oxidized/web/webapp.rb#L76
6. we move the node to head of queue and set the data - https://github.com/ytti/oxidized/blob/master/lib/oxidized/nodes.rb#L68
7. after work is finished, we send the data to output (git) - https://github.com/ytti/oxidized/blob/master/lib/oxidized/worker.rb#L40
8. we set the committer name in git - https://github.com/ytti/oxidized/blob/master/lib/oxidized/output/git.rb#L94

Where are you failing? Is it not moving the node to the head of the queue? Or is it moving, but the data, like committer does not appear in git?

[MrRJ45]

Its failing on either point 5, or 6. Is there a way of adding more logging for the data being sent?

What I do notice is that nodes.next references opt['msg'] which isnt sent as part of the data, could that be the issue here?

[ytti]

'msg' missing shouldn't be a problem, it'll be used in 'commit message' if set, but not required.

It's more curious if 'node' is set to something which works.

I'd add 'p node' and 'p opt' before 'nodes.next. line in webapp.rb, to see how those variables are set, prior to being called.
I guess we know 'nodes.next node' works, as GET /next/node works. So question is, what are we actually sending to it.

[MrRJ45]

It looks like its failing on line 75 of webapp.rb - with this error:

NameError - uninitialized constant Oxidized::API::WebApp::JSON:
/var/lib/gems/1.9.1/gems/oxidized-web-0.2.0/lib/oxidized/web/webapp.rb:75:in `block in class:WebApp'

Adding "require 'json'" to the top of that file fixes this.

The git repo now has the correct user as authored, and the IP the change was made from

[MrRJ45]

Some of our devices have reverse DNS, and that will resolve to a full host name including our domain.

However, the "name" in Oxidized is a shorter version. I'm not sure of a sensible way of handling this.

Perhaps we could have a config option to disable DNS resolution, if somebody is happy to do everything based on IP's alone?

[ytti]

Fixed oxidized-web, thank you for all the hard work you've done to get the syslog working

Option to disable DNS resolution should be somewhat easy to implement.

[MrRJ45]

You're welcome :) Its working very well at the moment - we see the user and their IP. Using GitList as it appears to be a simple web based git browser.

I've changed the function getname in syslog.rb so it will return back the IP and that works fine. Perhaps we should just have a config variable for this?

The next thing would be to have a wrapper for this, so we can run "/usr/local/bin/oxidized-syslog". Is that something that is easy to do so it isnt version dependent? :)

[MrRJ45]

Should this syslog code be a separate gem?

I guess that way it will have a wrapper and can be demonized in the same way that oxidized itself is :)

[ytti]

I really like 'gitlab' for hosting my git repos. (you could run 'git push' in crontab every 5min or 10min or what not, to push your oxidized repo periodically to 'upstream').
Seeing 'git blame' in 'gitlab' is really cool if you use this syslog feature, because then you very nicely see which person added which line in the config, and when.

I think gemifying the syslog.rb to generic trigger which allows multiple types of triggers for config pull. The currently it already almost supports reading syslog file from filesystem in addition to udp/network. But I'm not personally going to contribute any time for it in foreseeable future, even though this feature probably would be quite interesting to many, if it were more userfriendly.

[ytti]

I guess this can be closed, seeing MrRJ45 fixed this