visualize config graph

[furiel]

This patchset intends add supporting code so that users could visualize configuration graph. The results could be used for debugging.

Usage:

• Generate config graph from a running syslog-ng instance using syslog-ng-ctl

./syslog-ng-ctl export-config-graph

Optional:

• Use a newly added script in contrib to convert json to dot format. Then you can create a png, pdf or your preference (paths may differ for you):

rm tmp.png; ./syslog-ng-ctl export-config-graph | ../../syslog-ng/contrib/scripts/config-graph-json-to-dot.py | dot -Tpng -o tmp.png && xdg-open tmp.png

---

The idea is simple: logpipes will be nodes, arcs will be connections either through multiplexer (next_hop) or pipe_next. The walk starts from cfg->initialized_pipes. Anything that does something with message processing should be accessible from those.

During graph walk, I needed to use sets when collecting nodes, because some pipes can occur multiple times during walk. For example multiplexers are added to initialized pipes, but also linked inside the graph.

Some metadata data needed to be added that made easier to identify the logpipes. The metadata of choice were based on my needs in #2989, so it is surely incomplete. I do not plan to make this complete in the scope of this PR (I do not have any usecase that I could follow).

Here is an example for config:

@version: 3.24

options {
  time-reopen(5);
};

destination d_network {
    network("127.0.0.1" port(4447));
    network("127.0.0.1" port(4448));
};

log {
	source { example-msg-generator(); };
	source { example-msg-generator(); };
        destination(d_network);
        destination { file(/tmp/a.txt); };
        destination { file(/tmp/b.txt); };
        destination { file(/tmp/c.txt); };
	flags(flow-control);
};

And here is the output:

PS:
Me neither know what is that logpipe on the top right :)

[kira-syslogng]

Build FAILURE

[kira-syslogng]

Build SUCCESS

[Kokan]

[optional] I would consider instead of appending descriptions something like LogPiPe:to_json virtual function, or a function that provides a key-value pair list/hashmap with things to know about a pipe.

[furiel]

I agree the representation and walk logic was interleaving, leaving a lot of mess for the reviewers. I improved that part, now string manipulation is separated from the data representation. Please check if it is good enough now.

[kira-syslogng]

Build SUCCESS

[kira-syslogng]

Build SUCCESS

[kira-syslogng]

Build FAILURE

[furiel]

Am I allowed to link core to json-c? I am thinking about that cfg-tree could generate the config graph as json. The json object could be serialized as string, and passed through the syslog-ng-ctl socket/api.

The problem is for now core does not link to json-c.
We should have json-c on every system, either from system or from the internal source. What do you think?

[furiel]

I have rewritten the PR from scratch.

• The functionality moved from syslog-ng option to syslog-ng-ctl command
• The graph walk are based on arcs instead of neighbors. This means we can add information about the arc type too.
• Printing the arc type too: next_hop or pipe_next
• The string representation is delayed as much as possible, resulting in a cleaner code.
• I still manually assemble the json output. Core is not linked to json-c, and I did not want to add that dependency for this feature.

I updated the pr description with the new functionality. Removing wip flag.

[kira-syslogng]

Build SUCCESS

[czanik]

@furiel all packages maintained by me (openSUSE/SLES, fedora/RHEL, FreeBSD) have JSON support in the core syslog-ng package. There was a bug in syslog-ng that it did not start with SCL-s included from syslog-ng.conf and JSON support missing. The easiest workaround for me was to make JSON a mandatory dependency.
In FreeBSD ports it's still possible to disable it, but that breaks compilation with the following message:

.if !${PORT_OPTIONS:MJSON}
BROKEN=         Default configuration does not currently support non-JSON builds
.endif

So, from my part (covering probably 80%+ of our users) making JSON a mandatory dependency would not be any problem.

Of course, for embedded guys it might be still a pain.

[furiel]

@czanik thanks for the feedback. In the end I decided not to add json-c, with similar reasoning. Thinking about embedded systems, those could ship syslog-ng without json-c. They would most certainly do not need this feature, and they would prefer smaller deployment complexity.

If some other feature pulls json-c in, then we can rewrite this part though.

[Kokan]

thanks for the modifications.

[lbudai]

@furiel : I think, sooner or later, we need to add json as a hard dependency (just think of EWMM...).
But I think it is okay to postpone this decision :)

[lbudai]

@furiel : I have three notes.

1. it would be useful if this feature would be also available 'offline' (like preprocess-into)
2. why json? (I know: why not? :) this question is just for out of curiosity)
3. the output can be really huge (just try to run syslog-ng with the default Debian config ;)
   22859 x 3243 PNG generated from the dot file)
   Do you have a plan for implement additional feature on top of this? (subgraphs, etc...?)

[furiel]

1. it would be useful if this feature would be also available 'offline' (like preprocess-into)

We had a similar discussion with Peter: https://github.com/syslog-ng/syslog-ng/pull/2990/files/e79eaf1929f4d6518355d04c48f2f40d52c2a830#diff-66cacb90ca599fbe12b333a3ea155664
Unfortunately, some of the pipes are attached during init. That means I need to pass the init phase, so syslog-ng needs to be started. After syslog-ng has started, I was afraid to stop syslog-ng automatically. That's why the patchset does not look like a one shot function.

Another idea was that later syslog-ng might change configuration graph after init. In that case it would be nice if we could dump the configuration whenever we want. This time of writing, the logpipes do not change after init (except maybe reload) so this is not necessarily a good argument.

2. why json? (I know: why not? :) this question is just for out of curiosity)

I was aware that I am not familiar with visualization techniques, also there must be many ways/programs that could be used for visualizing, I wanted to make one thing for sure: to dump the graph some data in a popular format. So that if anyone has interest to work with the graph, they could have as easy job as much to transform to their data format of preference. To help them a little bit, I add an very simple example in python that could be used as a starting point.

Why json? I was aware that it could be easily parsed with python. I also got a little hyped that with json-c, I can easily transform my graph representation into json, and serialize as string, maybe I could also support pretty printing. It turned out later that core does not link to json-c, and I did not want to be the first one that add such dependency. Then shortly considered xml, but in the end I was thinking it is somewhat easier to parse json than xml (this may be a prejudice from my side). That would have meant that I need to pull another dependency: libxml or glib-xml parser. So I dropped the xml idea. From that I went to the most obvious path, hand formatted json.

3. the output can be really huge (just try to run syslog-ng with the default Debian config ;)
   Do you have a plan for implement additional feature on top of this? (subgraphs, etc...?)

My original intention was just develop some tool that could help me understand #2989. Actually, graph visualization as feature was not my goal, just to pull things together enough to deal with that issue. Also, I wanted to make sure if someone else runs into a similar scenario, they could have some code that they could fiddle with, according to their needs. The target users were more the maintainers than the end users. It is visible from the patches, that the focus was more like grabbing as much info about the config, and a little portion is that deals with that we could end up with a screenshot.

Where to go next? We can improve the graph walk. If we can collect enough information, we might add a post compile phase that optimizes the graph. Or we improve the the graph visualization. We could choose a better tool, maybe add information about the layouts, or groups. Nice coloring etc.

I do not plan to continue either directions in the near future, though. For now I am happy that we could finally catch #2989.