`<worker>` section to enable a set of configuration in a specified worker #1392

tagomoris opened this Issue Dec 27, 2016 · 1 comment


None yet

1 participant

tagomoris commented Dec 27, 2016 edited

Symmetric multi worker processing #1386 doesn't permit to execute any plugins and configurations in just one process.
But there are requirements and feature requests for such use cases. For example, executing plugins for monitoring (should be unique in a server).

This is an idea to keep symmetric semantics for multi process workers, but to realize such workloads.

  • add <worker N> section in configuration top-level
  • enable that section only in the specified worker (specified by worker id)
  • allow to run plugins only in <worker> sections, which doesn't support multi worker feature
  workers 8
  root_dir /path/fluentd/root
<source>  # top-level sections works on all workers in parallel
  @id edge_input
  @type forward
  @label @traffic
  port 24224
<label @traffic>  # this section works on all workers too
  <match **>
    @id backend_output
    @type forward
      @type file
      flush_interval 8m
<worker 0> # this section works only on first worker process
    @id system_log_reader
    @type tail
    tag monitoring
    path /path/now/watching/...
  <match monitoring>
    @id output_for_monitoring
    @type elasticsearch
    # ...

Points to be considered:

  • configuration dump is now done in worker_id=0 under a premise that "all workers are working with just same configuration" - this feature should update logging.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment