Config objects preceded by an inline destination are ignored

[MrAnno]

syslog-ng

Version of syslog-ng

syslog-ng v3.22.1

Issue

The following configuration will not send any logs to d_asd:

@version: 3.22
destination d_asd {
	network("127.0.0.1" port(4448));
};

log {
	source { network(port(4444)); };
	destination {
		network("127.0.0.1" port(4447) );
	};
	destination(d_asd);
	flags(flow-control);
};

The cfg graph built from this config is incorrect. Changing the order of the 2 destinations (listing the inline destination last) fixes the issue.

Root cause

Inline destinations are compiled as sequences (cfg_tree_compile_sequence()), where no multiplexer is created for other objects. In case of general/non-inline destinations (cfg_tree_compile_reference()), the multiplexer is there:

https://github.com/balabit/syslog-ng/blob/1742b11e5cfa6544ece38aaecde96e9b423d61c5/lib/cfg-tree.c#L842-L851

Unfortunately, it seems syslog-ng creates a new multiplexer for every non-inline destination instead of adding a hop to the previous one. I don't know the reason, but we could improve on this too.

[bazsi]

Hi, The reason for multiplexers is that the next pointer for a non-inline destination cannot be used. So the multiplexer is always inline, its list of next hops should contain a reference to the destination group. I think that the inline destinations being a sequence should not cause problems. We could discuss this via phone or slack or similar to find a good enough solution. László Várady <notifications@github.com> ezt írta (időpont: 2019. júl. 5., P 14:24):

…

syslog-ng Version of syslog-ng syslog-ng v3.22.1 Issue The following configuration will not send any logs to d_asd: @Version: 3.22 destination d_asd { network("127.0.0.1" port(4448)); }; log { source { network(port(4444)); }; destination { network("127.0.0.1" port(4447) ); }; destination(d_asd); flags(flow-control); }; The cfg graph built from this config is incorrect. Changing the order of the 2 destinations (listing the inline destination last) fixes the issue. Root cause Inline destinations are compiled as sequences (cfg_tree_compile_sequence()), where no multiplexer is created for other destinations. In case of general/non-inline destinations (cfg_tree_compile_reference()), the multiplexer is there: https://github.com/balabit/syslog-ng/blob/1742b11e5cfa6544ece38aaecde96e9b423d61c5/lib/cfg-tree.c#L842-L851 Unfortunately, it seems syslog-ng creates a new multiplexer for *every* non-inline destination instead of adding a hop to the previous one. I don't know the reason, but we could improve on this too. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2820>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFOK5W52JWZ65DE6TPRJZDP544QTANCNFSM4H6K3FQA> .

[furiel]

Is it possible that the problem depends on the types of the config elements? I checked with an inline file destination and a network destination, and it works for me:

$ cat ../etc/tmp.conf 
@version: 3.24

options {
  time-reopen(5);
};

destination d_network {
    network("127.0.0.1" port(4447) template("NETWORK: $MESSAGE\n"));
};

log {
	source { example-msg-generator(); };
	destination { file(/dev/stdout template("FILE: $MESSAGE\n")); };
        destination(d_network);
	flags(flow-control);
};
$ nc -k -l 4447 &
[1] 8261
$ timeout 5 ./syslog-ng -Fe -f ../etc/tmp.conf 
[2019-10-22T08:34:25.696765] syslog-ng starting up; version='3.24.1.11.g126a19f'
[2019-10-22T08:34:25.696968] Syslog connection established; fd='11', server='AF_INET(127.0.0.1:4447)', local='AF_INET(0.0.0.0:0)'
FILE: -- Generated message. --
NETWORK: -- Generated message. --
FILE: -- Generated message. --
NETWORK: -- Generated message. --
FILE: -- Generated message. --
NETWORK: -- Generated message. --
FILE: -- Generated message. --
NETWORK: -- Generated message. --
FILE: -- Generated message. --
NETWORK: -- Generated message. --
[2019-10-22T08:34:30.671485] syslog-ng shutting down; version='3.24.1.11.g126a19f'
FILE: -- Generated message. --
NETWORK: -- Generated message. --

However, with the config example in the ticket, I also experience the problem.

[furiel]

After looking into the matter deeper here is what I found out.

If I swap the order of an inline and normal destination, the config graph looks substantially different.

Showing a few examples:

• inline file, referenced network version (the config I used int the previous comment)
  

The file destination and the network destination are linked into the same chain serially. One would think how this can work at all. And the trick is that the affile_dd_queue function ends with a log_dest_driver_queue_method(s, msg, path_options);, that pushes forward the message in the chain.

• referenced network - inline file
  If I swap the order, the two destinations are put to the two tips of a multiplexer:

• inline network - referenced network
  So what is happening in case of the network-network version (the config that is posted in the issue)? The graph looks like this:
  

Technically it should look like one long chain (similar to the file-network graph), but the lower part is disjoint. It is disjoint at:

syslog-ng/modules/afsocket/afsocket-dest.c

Line 546 in 4b22d46

log_pipe_append(&self->super.super.super, (LogPipe *) self->writer);

That code expects self->next_pipe to be null, but it points to the lower part of the chain. After overwriting the link with the pointer of the logwriter, the chain is broken.

I was thinking how this should be resolved but I do not have any plan yet.

The analogous solution would be that instead of log_pipe_append, I would join the logwriter into the chain. But that would mean the message should flow through log_writer_queue. So that to work I need to add a log_dest_driver_queue_method-ish call to the end of log_writer_queue. But log_writer_queue does kind of strange things to the message (possibly drops), so I do not think that is a good idea.

Another thing that feels a little off, that now no destinations attach their workers to themselves, because they might not be a terminal point in the graph. Conceptually, me feels natural that a destination should be a terminal point in the graph.

It also feels strange that if I change the order of two destinations, the config graph looks so differently (single chain vs fork).

@bazsi could you give me some hint how to fix this?

[MrAnno]

I also feel it is right that destinations are terminal nodes in the graph. We obviously don't chain sources, but we should not chain destinations either in my opinion.

[I removed the misleading part of this comment, so I have now 0 real reason why we should use a multiplexer 😄.]

[MrAnno]

Sorry, I was not right about the flow control part, add_ack() is working properly without the multiplexer between destinations.