Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Massive red shutdown of nodes #1214

Closed
kpp opened this issue Sep 28, 2018 · 2 comments
Closed

Massive red shutdown of nodes #1214

kpp opened this issue Sep 28, 2018 · 2 comments
Labels
P0 Critical priority
Milestone

Comments

@kpp
Copy link

kpp commented Sep 28, 2018

Happened on 2018-09-25.

Seems like there is a memleak in the code.

<LittleVulpix> nurupo: Oh? Let's see...
 toxme.io is running so
 oh wiw
 wow*
 [2152112.531883] Out of memory: Kill process 3782 (tox-bootstrapd) score 893 or sacrifice child
 [2152112.643896] oom_reaper: reaped process 3782 (tox-bootstrapd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

 If so many went offline at that time I would imagine someone caused a leak to manifest in tox bootstrap daemon took down several nodes with it that way
 nurupo: node started but yeah I'm pretty sure this wouldn't have been random. I doubt we all started our nodes exactly at the same time so it must have been someone doing something with the network that caused bootstrap nodes to die
<nurupo> are you sure it was tox-bootstrapd that filled up your ram and not some other program?
<LittleVulpix> nurupo: not minding the fact the same happened to several completely unrelated machines, yeah I'm pretty sure
 [2152112.531817] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
 [2152112.531846] [ 3782]   999  3782   467797   455738     903       4        0             0 tox-bootstrapd
  467MB ram
  reserved
  455 actually used
  normal resource usage is about 10-15MB RAM from what I remember
  soooo that's like 45x as much
<iphy> LittleVulpix: interesting
<nurupo> if there is a memory leak then it's likely to be in toxcore itself so every tox node (i.e. even a client), i don't think there is anything in tox-bootstrapd that can leak memory, so everything that uses toxcore is likely affected, even clients. also, a lot of nodes were using old toxcore, like irungentoo old, so if there is a memory leak issue - it must be long present
<iphy> yes

nodes8

@anthonybilinski
Copy link

I have a core of my still running bootstrap node, which is up to 3.2GB res. Willing to work with someone to dig into it to root cause, ping me on Tox or IRC when a toxcore dev has time.

@kpp
Copy link
Author

kpp commented Oct 7, 2018

@kpp kpp closed this as completed Oct 7, 2018
@iphydf iphydf added this to the v0.2.8 milestone Oct 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0 Critical priority
Projects
None yet
Development

No branches or pull requests

3 participants