Memory leak with set

[dom96]

I have been investigating the high memory usage of my application for some days now. One cause that I've just found is set. The memory grows unbounded in my program if I use it.

I've tried to reproduce this, but the best I could do is to get an application that uses a lot of memory (the memory usage doesn't grow). The example below uses 75MB of memory on my MacBook, and a simple s/set/HashSet/ makes that decrease to 1.5MB so there is surely something wrong here.

As far as my application goes, switching to HashSet also fixes the unbounded growth.

Example

import tables, sets

type
  BotEventKind* = enum
    DirectionChanged, SpeedChanged, SizeChanged, NearbyFoods, NearbyPlayers,
    PlayerDeath, FoodEaten

  Degrees = float
  PlayerID = distinct int16
  ParticleID = (PlayerID, int)
  Point[T] = object
    x, y: T

  NearbyPlayerTuple* = tuple[id: PlayerID, pos: Point[float], size: float, gap: float]
  NearbyFoodTuple* = tuple[id: ParticleID, pos: Point[float], size: uint8, dist: float]
  BotEvent* = object
    case kind*: BotEventKind
    of DirectionChanged: # Triggered on direction change as set/sent to server.
      direction*: Degrees
    of SpeedChanged: # Triggered when the speed changes.
      speed*: int # How fast the player is moving per second.
      burst*: bool
    of SizeChanged: # Triggered when the bot size changes.
      size*: float
      delta*: float
    of NearbyFoods:
      foods*: seq[NearbyFoodTuple]
    of NearbyPlayers:
      players*: seq[NearbyPlayerTuple]
    of PlayerDeath:
      deadPlayers*: set[PlayerID]
    of FoodEaten:
      foodEaten*: HashSet[ParticleID]
  
  Bot = ref object
    queuedEvents: Table[BotEventKind, BotEvent]
 

proc queueEvent(bot: Bot, event: BotEvent) =
  bot.queuedEvents[event.kind] = event

proc getEvents(bot: Bot): Table[BotEventKind, BotEvent] =
  return bot.queuedEvents

proc onTick(bot: Bot) =
  let events = bot.getEvents()
  if SpeedChanged in events:
    discard

proc eventLoop(bot: Bot) =
  bot.queueEvent(
    BotEvent(
      kind: SpeedChanged,
      speed: 12,
      burst: true
    )
  )
  bot.queueEvent(
    BotEvent(
      kind: DirectionChanged,
      direction: 4.5,
    )
  )
  
  bot.queueEvent(
    BotEvent(
      kind: PlayerDeath,
      deadPlayers: {PlayerID(12)}
    )
  )
  
  bot.onTick()
  bot.queuedEvents.clear()

proc runBots() =
  var bots: seq[Bot]
  for i in 0 .. 100:
    bots.add(Bot())
  
  while true:
    for b in bots:
      b.eventLoop()

runBots()

Current Output

Memory usage is 75MB

Expected Output

Memory usage should be lower, or alternatively we should disallow usage of set this way to prevent memory leaks (or perceived memory leaks).

Possible Solution

Use HashSet instead

Additional Information

$ nim -v
Nim Compiler Version 0.20.99 [MacOSX: amd64]
Compiled at 2019-08-06
Copyright (c) 2006-2019 by Andreas Rumpf

git hash: 8407a574992ebd6bccec647a902cf54a4de8db18
active boot switches: -d:release

[Araq]

Well set is implemented as a bitset and only allows for uint16 for this reason. I don't see how further restricting it would help.

[dom96]

Well, from what I could tell the allocator doesn't handle this very well. Do you have any ideas why that might be?

[Araq]

Memory usage with Boehm is even higher for me, that rules out fragmentation. You have 101 bots each has a table with 64 entries, each entry takes roughly 8KB, total memory usage about 50.648MiB. Floating garbage and other stuff makes it 82MiB on my machine. Perfectly acceptable except that tables's defaultInitialSize* = 64 is a bit high.

[dom96]

Is it acceptable? This is quite a gotcha that might make some feel that there is a bug in Nim causing a mem leak. We should at least document this well.

…

On Wed, 18 Sep 2019 at 11:43, Andreas Rumpf ***@***.***> wrote: Closed #12052 <#12052>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12052?email_source=notifications&email_token=AAB4G67CCFKLRKG2QDQPLKLQKIA4PA5CNFSM4IPO7722YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTWEW6QY#event-2643029827>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB4G64OKE3TGPYQC4JZAE3QKIA4PANCNFSM4IPO772Q> .

[Araq]

There is no leak and here is another workaround:

proc runBots() =
  var bots: seq[Bot]
  for i in 0 .. 100:
    bots.add(Bot(queuedEvents: initTable[BotEventKind, BotEvent](8)))