Skip to content

RPC gracefulReload fails for mongo_replset plugin #179

@jeevamanivel

Description

@jeevamanivel

We are using mongo_replset plugin to forward logs to Mongo DB. When we reload fluentd config using fluentd HTTP RPC api, instead of restarting fluentd, it fails with below error

Reload URL : http://127.0.0.1:24444/api/config.gracefulReload

Config:

<system>
rpc_endpoint 127.0.0.1:24444
</system>
       <source>
	  @type forward
	  @id mongo_forward
	  port 24224 
	  bind 0.0.0.0
	  @label @LOGS
	</source>
	<label @LOGS>
		<match mongo_forward>
			@type mongo_replset
			nodes HOST_PORT
			database DB_NAME
			collection COLLECTION_NAME
			replica_set REPLICA_NAME
			include_time_key false
			date_keys datekey
			<buffer tag>
				@type file
				path /fluentd/work/buf
				chunk_limit_size 256m
				total_limit_size 500g
				compress gzip
				queued_chunks_limit_size 10
				flush_thread_count 10
				flush_mode interval
				flush_interval 2s
				flush_thread_burst_interval 1.0
				flush_at_shutdown true
				overflow_action throw_exception
				retry_max_times 3
				retry_exponential_backoff_base 2
				retry_type exponential_backoff
				retry_timeout 24h
				retry_wait 2s
				retry_forever false
			</buffer>
		</match>
	</label>

Error

2023-03-23 09:48:34 +0530 [debug]: fluent/log.rb:309:debug: preparing shutdown output plugin type=:mongo_replset plugin_id="object:a50"
2023-03-23 09:48:34 +0530 [info]: fluent/log.rb:330:info: shutting down output plugin type=:mongo_replset plugin_id="object:a50"
2023-03-23 09:48:34 +0530 [warn]: fluent/log.rb:351:warn: unexpected error while shutting down on output plugin plugin=Fluent::Plugin::MongoOutputReplset plugin_id="object:a50" error_class=NoMethodError error="undefined method `close' for nil:NilClass\n\n      @client.close\n             ^^^^^^"
#<Thread:0x0000029a74c9d590 C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/root_agent.rb:264 run> terminated with exception (report_on_exception is true):
C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/log.rb:566:in `caller_line': undefined method `[]' for nil:NilClass (NoMethodError)

        line = caller(depth+1)[0]
                              ^^^
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/log.rb:467:in `dump_stacktrace'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/log.rb:357:in `warn_backtrace'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/root_agent.rb:287:in `rescue in block (3 levels) in shutdown'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/root_agent.rb:266:in `block (3 levels) in shutdown'
C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluent-plugin-mongo-1.6.0/lib/fluent/plugin/out_mongo.rb:189:in `shutdown': undefined method `close' for nil:NilClass (NoMethodError)

      @client.close
             ^^^^^^
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/root_agent.rb:280:in `block (3 levels) in shutdown'
2023-03-23 09:48:34 +0530 [debug]: fluent/log.rb:309:debug: Got Win32 event "fluentd_21228_STOP_EVENT_THREAD"
2023-03-23 09:48:34 +0530 [error]: fluent/log.rb:372:error: Failed to reload config file: undefined method `[]' for nil:NilClass

        line = caller(depth+1)[0]
                              ^^^
Unexpected error undefined method `[]' for nil:NilClass

        line = caller(depth+1)[0]
                              ^^^
  C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/log.rb:566:in `caller_line'
  C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/log.rb:467:in `dump_stacktrace'
  C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/log.rb:357:in `warn_backtrace'
  C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/root_agent.rb:287:in `rescue in block (3 levels) in shutdown'
  C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3-x64-mingw-ucrt/lib/fluent/root_agent.rb:266:in `block (3 levels) in shutdown'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions