`<worker>` section to enable a set of configuration in a specified worker #1392

Open
tagomoris opened this Issue Dec 27, 2016 · 1 comment

Projects

None yet

1 participant

@tagomoris
Member
tagomoris commented Dec 27, 2016 edited

Symmetric multi worker processing #1386 doesn't permit to execute any plugins and configurations in just one process.
But there are requirements and feature requests for such use cases. For example, executing plugins for monitoring (should be unique in a server).

This is an idea to keep symmetric semantics for multi process workers, but to realize such workloads.

  • add <worker N> section in configuration top-level
  • enable that section only in the specified worker (specified by worker id)
  • allow to run plugins only in <worker> sections, which doesn't support multi worker feature
<system>
  workers 8
  root_dir /path/fluentd/root
</system>
<source>  # top-level sections works on all workers in parallel
  @id edge_input
  @type forward
  @label @traffic
  port 24224
</source>
<label @traffic>  # this section works on all workers too
  <match **>
    @id backend_output
    @type forward
    <buffer>
      @type file
      flush_interval 8m
    </buffer>
  </match>
</label>
<worker 0> # this section works only on first worker process
  <source>
    @id system_log_reader
    @type tail
    tag monitoring
    path /path/now/watching/...
  </source>
  <match monitoring>
    @id output_for_monitoring
    @type elasticsearch
    # ...
  </match>
</worker>
@tagomoris
Member

Points to be considered:

  • configuration dump is now done in worker_id=0 under a premise that "all workers are working with just same configuration" - this feature should update logging.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment