Skip to content

Massive red shutdown of nodes #1214

Description

@kpp

Happened on 2018-09-25.

Seems like there is a memleak in the code.

<LittleVulpix> nurupo: Oh? Let's see...
 toxme.io is running so
 oh wiw
 wow*
 [2152112.531883] Out of memory: Kill process 3782 (tox-bootstrapd) score 893 or sacrifice child
 [2152112.643896] oom_reaper: reaped process 3782 (tox-bootstrapd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

 If so many went offline at that time I would imagine someone caused a leak to manifest in tox bootstrap daemon took down several nodes with it that way
 nurupo: node started but yeah I'm pretty sure this wouldn't have been random. I doubt we all started our nodes exactly at the same time so it must have been someone doing something with the network that caused bootstrap nodes to die
<nurupo> are you sure it was tox-bootstrapd that filled up your ram and not some other program?
<LittleVulpix> nurupo: not minding the fact the same happened to several completely unrelated machines, yeah I'm pretty sure
 [2152112.531817] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
 [2152112.531846] [ 3782]   999  3782   467797   455738     903       4        0             0 tox-bootstrapd
  467MB ram
  reserved
  455 actually used
  normal resource usage is about 10-15MB RAM from what I remember
  soooo that's like 45x as much
<iphy> LittleVulpix: interesting
<nurupo> if there is a memory leak then it's likely to be in toxcore itself so every tox node (i.e. even a client), i don't think there is anything in tox-bootstrapd that can leak memory, so everything that uses toxcore is likely affected, even clients. also, a lot of nodes were using old toxcore, like irungentoo old, so if there is a memory leak issue - it must be long present
<iphy> yes

nodes8

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical priority

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions