Erratic failure to exit when using thread pools dependent on setMaxPoolSize

[Benjamin-Lee]

When using multithreading, I'm getting weird issues that, when using certain values of setMaxPoolSize, the code won't exit. It never seems to exit when using a single thread and sometimes hits weird values where it also won't exit.

Example

import threadpool
import locks
import os
import strutils

var stdoutLock: Lock
initLock(stdoutLock)

proc checkLine(line: string) = 
  if line.len > 50:
    withLock stdoutLock:
      echo(line)

setMaxPoolSize(paramStr(2).parseInt)
for line in paramStr(1).lines:
  spawn(checkLine(line))
sync()

Current Output

Correct output, just no exiting.

For example:

~/D/P/R/viroid-search ❯❯❯ nim c --threads:on -d:danger src/test
Hint: used config file '/usr/local/Cellar/nim/1.4.0/nim/config/nim.cfg' [Conf]
Hint: used config file '/usr/local/Cellar/nim/1.4.0/nim/config/config.nims' [Conf]
.....................CC: stdlib_assertions.nim
CC: stdlib_locks.nim
CC: stdlib_sharedlist.nim
CC: stdlib_io.nim
CC: stdlib_system.nim
CC: stdlib_posix.nim
CC: stdlib_cpuinfo.nim
CC: stdlib_cpuload.nim
CC: stdlib_parseutils.nim
CC: stdlib_strutils.nim
CC: stdlib_times.nim
CC: stdlib_os.nim
CC: stdlib_threadpool.nim
CC: test.nim

Hint:  [Link]
Hint: 47002 lines; 1.360s; 61.414MiB peakmem; Dangerous Release build; proj: src/test; out: /Users/BenjaminLee/Desktop/Python/Research/viroid-search/src/test [SuccessX]
~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 2 | wc -l
^CSIGINT: Interrupted by Ctrl-C.

________________________________________________________
Executed in    6.35 secs   fish           external
   usr time   30.62 millis  220.00 micros   30.40 millis
   sys time   55.59 millis  879.00 micros   54.71 millis

~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 3 | wc -l
    2494

________________________________________________________
Executed in   54.68 millis    fish           external
   usr time   27.55 millis  170.00 micros   27.38 millis
   sys time   46.69 millis  756.00 micros   45.94 millis

~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 4 | wc -l
    2494

________________________________________________________
Executed in   54.68 millis    fish           external
   usr time   27.76 millis  186.00 micros   27.58 millis
   sys time   46.89 millis  810.00 micros   46.08 millis

~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 5 | wc -l
    2494

________________________________________________________
Executed in   57.76 millis    fish           external
   usr time   28.35 millis  151.00 micros   28.20 millis
   sys time   49.17 millis  715.00 micros   48.46 millis

~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 2 | wc -l
^CSIGINT: Interrupted by Ctrl-C.

________________________________________________________
Executed in    1.46 secs   fish           external
   usr time   28.28 millis  214.00 micros   28.07 millis
   sys time   48.85 millis  929.00 micros   47.92 millis

~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 1 | wc -l
^CSIGINT: Interrupted by Ctrl-C.

________________________________________________________
Executed in    2.25 secs   fish           external
   usr time   29.53 millis  263.00 micros   29.27 millis
   sys time   51.33 millis  1109.00 micros   50.22 millis

Then, when I recompile, it works:

~/D/P/R/viroid-search ❯❯❯ nim c --threads:on -d:danger --gc:refc src/test
Hint: used config file '/usr/local/Cellar/nim/1.4.0/nim/config/nim.cfg' [Conf]
Hint: used config file '/usr/local/Cellar/nim/1.4.0/nim/config/config.nims' [Conf]
.....................CC: stdlib_assertions.nim
CC: stdlib_locks.nim
CC: stdlib_io.nim
CC: stdlib_system.nim
CC: stdlib_cpuinfo.nim
CC: stdlib_parseutils.nim
CC: stdlib_strutils.nim
CC: stdlib_times.nim
CC: stdlib_os.nim
CC: stdlib_threadpool.nim
CC: test.nim

Hint:  [Link]
Hint: 47002 lines; 1.387s; 61.461MiB peakmem; Dangerous Release build; proj: /Users/BenjaminLee/Desktop/Python/Research/viroid-search/src/test; out: /Users/BenjaminLee/Desktop/Python/Research/viroid-search/src/test [SuccessX]
~/D/P/R/viroid-search ❯❯❯ time ./src/test test.fasta 2 | wc -l
    2494

________________________________________________________
Executed in   56.40 millis    fish           external
   usr time   28.06 millis  197.00 micros   27.86 millis
   sys time   47.58 millis  850.00 micros   46.73 millis

Expected Output

Same as current output, just with exiting.

Possible Solution

Additional Information

ORC seems to work with two threads but still doesn't exit with one thread.

$ nim -v
Nim Compiler Version 1.4.0 [MacOSX: amd64]
Compiled at 2020-10-17
Copyright (c) 2006-2020 by Andreas Rumpf

active boot switches: -d:release -d:nimUseLinenoise

[Benjamin-Lee]

By the way, this may be the same issue as #11875 but I'm not 100% sure. It doesn't always fail and it doesn't always fail on the same sizes.

[ruqianl]

Having the same issue but not always fail on the same sexMaxPoolSize(). Just hangs like 1 out of five runs.

Nim Compiler Version 1.4.0 [Linux: amd64]
Compiled at 2020-10-18
Copyright (c) 2006-2020 by Andreas Rumpf

git hash: bdcd87afca238a0a7b2c70971827cf9172817b12
active boot switches: -d:release