Add <worker n> section to set a configuration for a specific worker #1507

mururu · 2017-03-17T07:55:20Z

Closes #1392.

This PR introduce <worker n> section. Configurations under this section is enabled only in specified worker and act as if they work in single worker environment. More precisely, config elements in worker section get worker_target_id recursively. Prior to configure phase of corresponding plugins, system_config.workers for the plugins is overwritten as 1 if target_worker_id equals the worker's worker_id.
It also means that plugins(including owned plugins) which doesn't support multi worker feature can run only in worker section.
A worker section can have <label>, <source>, <filter> and <match> as a child element.

Example from #1392

<system>
  workers 8
  root_dir /path/fluentd/root
</system>
<source>  # top-level sections works on all workers in parallel
  @id edge_input
  @type forward
  @label @traffic
  port 24224
</source>
<label @traffic>  # this section works on all workers too
  <match **>
    @id backend_output
    @type forward
    <buffer>
      @type file
      flush_interval 8m
    </buffer>
  </match>
</label>
<worker 0> # this section works only on first worker process
  <source>
    @id system_log_reader
    @type tail
    tag monitoring
    path /path/now/watching/...
  </source>
  <match monitoring>
    @id output_for_monitoring
    @type elasticsearch
    # ...
  </match>
</worker>

mururu · 2017-03-23T07:16:59Z

Updated.
@repeatedly @tagomoris Could you review this?

tagomoris · 2017-03-23T07:23:45Z

Please add an example configuration which use this feature under example/.

mururu · 2017-03-23T07:47:06Z

Done.

mururu · 2017-03-23T08:11:53Z

Updated the example to try this feature easily.

tagomoris · 2017-03-23T08:51:10Z

example/worker_section.conf

+  </inject>
+</match>
+
+<worker 0> # this section works only on first worker process


This configuration doesn't use the <worker> section feature to execute plugins which doesn't support multi process workers.

tagomoris · 2017-03-23T08:52:36Z

lib/fluent/config/element.rb

+        }
+      end
+
+      def has_target?


#has_target doesn't show the correct meaning of this method. What's the target?
#has_target_worker_id looks better.

repeatedly · 2017-04-06T20:11:10Z

lib/fluent/engine.rb

-    private
+    def worker_id
+      return @_worker_id if @_worker_id
+      @_worker_id = (ENV['SERVERENGINE_WORKER_ID'] || 0).to_i


Above deleted comment should be here.

repeatedly · 2017-04-06T20:23:12Z

lib/fluent/root_agent.rb

+
+        target_worker_id = target_worker_id_str.to_i
+        if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
+          raise ConfigError, "worker#{target_worker_id} specified by <worker> directive doesn't exist. Specify id between 0 and #{(Fluent::Engine.system_config.workers - 1)}"


If specify -1, log message is worker-1. This is unclear, worker - 1 or worker -1.

"worker id #{target_worker_id} specified in directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"

repeatedly · 2017-04-06T20:24:29Z

lib/fluent/agent.rb

@@ -62,6 +62,7 @@ def configure(conf)

      # initialize <match> and <filter> elements
      conf.elements('filter', 'match').each { |e|
+        next if e.has_target_worker_id? && e.target_worker_id != Fluent::Engine.worker_id


Extract to method is better?

repeatedly · 2017-04-06T20:24:43Z

lib/fluent/agent.rb

@@ -121,7 +122,8 @@ def lifecycle(desc: false)
    end

    def add_match(type, pattern, conf)
-      log.info :worker0, "adding match#{@context.nil? ? '' : " in #{@context}"}", pattern: pattern, type: type
+      log_type = conf.target_worker_id == Fluent::Engine.worker_id ? :default : :worker0


Extract to method is better too?

repeatedly · 2017-04-06T20:25:32Z

test/command/test_fluentd.rb

@@ -614,5 +614,205 @@ def assert_fluentd_fails_to_start(cmdline, *pattern_list, timeout: 10)
        "config error file=\"#{conf_path}\" error_class=Fluent::ConfigError error=\"Plugin 'file' does not support multi workers configuration (Fluent::Plugin::FileBuffer)\"",
      )
    end
+
+    test 'failed to start workers when configured plugins as chidren of MultiOutput do not support multi worker configuration' do
+      script =  "require 'fluent/plugin/output'\n"


Here document is better.

repeatedly · 2017-04-06T20:25:50Z

test/command/test_fluentd.rb

+    end
+
+    test 'success to start workers when configured plugins only for specific worker do not support multi worker configuration' do
+      script =  "require 'fluent/plugin/input'\n"


repeatedly · 2017-04-06T20:25:58Z

test/command/test_fluentd.rb

+    end
+
+    test 'success to start workers when configured plugins as a chidren of MultiOutput only for specific worker do not support multi worker configuration' do
+      script =  "require 'fluent/plugin/output'\n"


…nd #for_another_worker? instead of #has_target_worker_id?

mururu · 2017-04-10T10:18:13Z

I added #for_every_workers?, #for_this_worker? and #for_another_worker? instead of #has_target_worker_id? and #target_worker_id ==/!= Fluent::Engine.worker_id. How about that?

repeatedly · 2017-04-12T12:56:06Z

LGTM :)

repeatedly · 2017-04-13T13:06:20Z

Merged. Thanks for hard work!

mururu · 2017-04-13T17:36:58Z

Thanks!

mururu force-pushed the worker-section branch from e039a24 to 2de6e20 Compare March 23, 2017 05:51

mururu changed the title ~~[WIP] Add <worker n> section to set a configuration for a specific worker~~ Add <worker n> section to set a configuration for a specific worker Mar 23, 2017

mururu force-pushed the worker-section branch from 2de6e20 to a23d1ea Compare March 23, 2017 07:46

Add <worker n> section to set a configuration for a specific worker

08ab6ec

mururu force-pushed the worker-section branch from a23d1ea to 08ab6ec Compare March 23, 2017 08:09

tagomoris reviewed Mar 23, 2017

View reviewed changes

Yuki Ito added 2 commits March 23, 2017 18:16

Rename Fluent::Config::Element#has_target? to #has_target_id?

6cf3d67

Use in_tail as a example for worker section

9ec9f1c

repeatedly reviewed Apr 6, 2017

View reviewed changes

Yuki Ito added 4 commits April 10, 2017 01:50

Fix error message

66b14da

Use heredoc instead of concatenation

de73789

Rewrite deleted comment

0077226

Add Fluent::Config::element#for_evenry_workers?, #for_this_workers? a…

b328e59

…nd #for_another_worker? instead of #has_target_worker_id?

repeatedly merged commit 751d130 into fluent:master Apr 13, 2017

mururu deleted the worker-section branch April 13, 2017 17:37

daipom mentioned this pull request Feb 21, 2023

Fix value of system_config.workers at run_configure #4066

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add <worker n> section to set a configuration for a specific worker #1507

Add <worker n> section to set a configuration for a specific worker #1507

mururu commented Mar 17, 2017 •

edited

mururu commented Mar 23, 2017

tagomoris commented Mar 23, 2017

mururu commented Mar 23, 2017

mururu commented Mar 23, 2017

tagomoris Mar 23, 2017

tagomoris Mar 23, 2017

repeatedly Apr 6, 2017

repeatedly Apr 6, 2017

repeatedly Apr 6, 2017

repeatedly Apr 6, 2017

repeatedly Apr 6, 2017

repeatedly Apr 6, 2017

repeatedly Apr 6, 2017

mururu commented Apr 10, 2017

repeatedly commented Apr 12, 2017

repeatedly commented Apr 13, 2017

mururu commented Apr 13, 2017

Add <worker n> section to set a configuration for a specific worker #1507

Add <worker n> section to set a configuration for a specific worker #1507

Conversation

mururu commented Mar 17, 2017 • edited

mururu commented Mar 23, 2017

tagomoris commented Mar 23, 2017

mururu commented Mar 23, 2017

mururu commented Mar 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mururu commented Apr 10, 2017

repeatedly commented Apr 12, 2017

repeatedly commented Apr 13, 2017

mururu commented Apr 13, 2017

mururu commented Mar 17, 2017 •

edited