New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal error: concurrent map writes #2102

Closed
zrice57 opened this Issue Nov 29, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@zrice57

zrice57 commented Nov 29, 2016

Bug report

Telegraf crashes with error "fatal error: concurrent map writes"

Relevant telegraf.conf:

See gist.

Relevant Input

I have two statsd inputs configured and a few other inputs. I've only included the statsd input config because it seems the most relevent.

This has been happening on multiple servers regularly for at least a week.

System info:

Steps to reproduce:

  1. Run telegraf
  2. telegraf will run without issues for many hours
  3. kapacitor will tell me one of my servers has stopped sending data
  4. kapacitor will tell me "JK it's all good" about 30 seconds later
  5. Check telegraf logs (in /varr/log/messages)
  6. Learn that telegraf has crashed and there is a huge crash log

Expected behavior:

Telegraf will run until the heat death of the universe.

Actual behavior:

Telegraf runs for a few hours.

Additional info:

Here is the link to the crash log and configs: Link

I will provide anything else you need.
Thank you for your time!

@sparrc

This comment has been minimized.

Show comment
Hide comment
@sparrc

sparrc Nov 29, 2016

Contributor

thanks for the crash log, might be related to this: #1975

Contributor

sparrc commented Nov 29, 2016

thanks for the crash log, might be related to this: #1975

@sparrc sparrc added this to the 1.2.0 milestone Nov 29, 2016

@sparrc sparrc added bug panic labels Nov 29, 2016

@zrice57

This comment has been minimized.

Show comment
Hide comment
@zrice57

zrice57 Dec 2, 2016

I've disabled one of the statsd inputs and have not seen a crash since.

zrice57 commented Dec 2, 2016

I've disabled one of the statsd inputs and have not seen a crash since.

sparrc added a commit that referenced this issue Dec 20, 2016

Do not create a global statsd "previous instance"
this basically reverts #887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes #1975
fixes #2102

@sparrc sparrc closed this in #2185 Dec 20, 2016

@121watts 121watts removed the in progress label Dec 20, 2016

njwhite pushed a commit to njwhite/telegraf that referenced this issue Jan 31, 2017

Do not create a global statsd "previous instance"
this basically reverts #887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes #1975
fixes #2102

bcaudesaygues added a commit to viareport/telegraf that referenced this issue Feb 6, 2017

Do not create a global statsd "previous instance"
this basically reverts #887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes #1975
fixes #2102

maxunt pushed a commit that referenced this issue Jun 26, 2018

Do not create a global statsd "previous instance"
this basically reverts #887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes #1975
fixes #2102
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment