New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep receiving messages even after recieveing SIGTERM and lost data #1985

Open
snowhork opened this Issue May 16, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@snowhork
Copy link

snowhork commented May 16, 2018

Even after fluentd receives SIGTERM and closes, fluentd seems to keep recieving messages for a while.
So, even if flush_at_shutdown is true,

  • buffer remains after shutdown
  • some messages seem to be lost

Environment:

commit 9738a421c2fe9f001b723ce86e4a6e47412c4baf
ruby 2.3.1p112
Mac OS 10.13.4

Preparation:

gem install fluent-logger

vi fluentd.conf
vi post.rb  # source is as follows

fluentd.conf is

<source>
  @type forward
</source>

<match debug.**>
  @type file
  path tmp/file
  flush_at_shutdown true
</match>

post.rb is

require 'fluent-logger'
log = Fluent::Logger::FluentLogger.new

cnt = 0
while log.post('debug', { message: cnt+1 })
  cnt += 1
  sleep 0.00001
end
puts "finished at #{Time.now}, post-count: #{cnt}"

Run commands below:

bundle exec ./bin/fluentd -c ./fluent.conf &
# => [1] 61688
ruby post.rb &
kill 61688
# => ....
# => finished at 2018-05-16 13:39:10 +0900, post-count: 111296

After that, you can find a buffer file.

tail -n 1 tmp/file.20180516_0.log
# => 2018-05-16T13:39:07+09:00	debug	{"message":79760}
tail -n 1 tmp/file/buffer.b56c4b4cd48555a7da3dd7abe659bf157.log
# => 2018-05-16T13:39:07+09:00	debug	{"message":79763}

In this case, messages between 79763 and 111296 seem to be lost.

@repeatedly

This comment has been minimized.

Copy link
Member

repeatedly commented May 21, 2018

Will check with fluent-logger.

@repeatedly repeatedly self-assigned this May 21, 2018

@repeatedly repeatedly added the v1 label May 21, 2018

@abicky

This comment has been minimized.

Copy link
Contributor

abicky commented May 21, 2018

I've investigated the cause before. I think tarcieri/cool.io#61 is related although I familiar with neither cool.io nor socket programming.

@snowhork snowhork changed the title Keep recieving messages even after recieveing SIGTERM and lost data Keep receiving messages even after recieveing SIGTERM and lost data May 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment