occasional SIGSEGV when using threadpool and channels #5626

vegansk · 2017-03-29T10:32:06Z

import threadpool

var ch: Channel[int]
ch.open
var pch = ch.addr

proc run(f: proc(): int {.gcsafe.}): proc() =
  let r = spawn f()
  return proc() = await(r)

var working = false

proc handler(): int =
  while true:
    let (h, v) = pch[].tryRecv()
    if not h:
      discard cas(working.addr, true, false)
      break
  1

proc send(x: int) =
  ch.send(x)
  if cas(working.addr, false, true):
    discard run(handler)

for x in 0..1000000:
  send(x)

crashes with error:

Traceback (most recent call last)
threadpool.nim(330)      slave
tp.nim(8)                :tmpWrapper
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

The text was updated successfully, but these errors were encountered:

Araq · 2018-04-22T22:53:50Z

I suppose spawn does not yet support function pointers. Workaround:

import threadpool

var ch: Channel[int]
ch.open
var pch = ch.addr

var working = false

proc handler(): int =
  while true:
    let (h, v) = pch[].tryRecv()
    if not h:
      discard cas(working.addr, true, false)
      break
  1


proc run(): proc() =
  let r = spawn handler()
  return proc() = await(r)

proc send(x: int) =
  ch.send(x)
  if cas(working.addr, false, true):
    discard run()

for x in 0..1000000:
  send(x)

genotrance · 2018-07-02T19:49:31Z

No longer crashes with #head.

-------- SNIPPET --------
import threadpool

var ch: Channel[int]
ch.open
var pch = ch.addr

proc run(f: proc(): int {.gcsafe.}): proc() =
  let r = spawn f()
  return proc() = await(r)

var working = false

proc handler(): int =
  while true:
    let (h, v) = pch[].tryRecv()
    if not h:
      discard cas(working.addr, true, false)
      break
  1

proc send(x: int) =
  ch.send(x)
  if cas(working.addr, false, true):
    discard run(handler)

for x in 0..1000000:
  send(x)
-------------------------

-------- OUTPUT --------
Hint: used config file 'c:\users\gt\Desktop\DL\programming\nimdevel\config\nim.cfg' [Conf]
Hint: system [Processing]
Hint: temp [Processing]
Hint: threadpool [Processing]
Hint: cpuinfo [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: bitops [Processing]
Hint: algorithm [Processing]
Hint: unicode [Processing]
Hint: os [Processing]
Hint: times [Processing]
Hint: winlean [Processing]
Hint: dynlib [Processing]
Hint: ospaths [Processing]
Hint: cpuload [Processing]
CC: temp
CC: stdlib_system
CC: stdlib_sharedlist
CC: stdlib_locks
CC: stdlib_threadpool
CC: stdlib_cpuinfo
CC: stdlib_strutils
CC: stdlib_parseutils
CC: stdlib_math
CC: stdlib_bitops
CC: stdlib_algorithm
CC: stdlib_unicode
CC: stdlib_os
CC: stdlib_times
CC: stdlib_winlean
CC: stdlib_dynlib
CC: stdlib_ospaths
CC: stdlib_cpuload
Hint:  [Link]
Hint: operation successful (27677 lines compiled; 2.859 sec total; 31.848MiB peakmem; Debug Build) [SuccessX]

Ran successfully, returned -1073741819
------------------------

-------- NIMTEMP --------
Hint: used config file 'c:\users\gt\Desktop\DL\programming\nimdevel\config\nim.cfg' [Conf]
Hint: system [Processing]
Hint: temp [Processing]
Hint: threadpool [Processing]
Hint: cpuinfo [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: bitops [Processing]
Hint: algorithm [Processing]
Hint: unicode [Processing]
Hint: os [Processing]
Hint: times [Processing]
Hint: winlean [Processing]
Hint: dynlib [Processing]
Hint: ospaths [Processing]
Hint: cpuload [Processing]
CC: temp
CC: stdlib_system
CC: stdlib_sharedlist
CC: stdlib_locks
CC: stdlib_threadpool
CC: stdlib_cpuinfo
CC: stdlib_strutils
CC: stdlib_parseutils
CC: stdlib_math
CC: stdlib_bitops
CC: stdlib_algorithm
CC: stdlib_unicode
CC: stdlib_os
CC: stdlib_times
CC: stdlib_winlean
CC: stdlib_dynlib
CC: stdlib_ospaths
CC: stdlib_cpuload
Hint:  [Link]
Hint: operation successful (27677 lines compiled; 5.860 sec total; 31.711MiB peakmem; Debug Build) [SuccessX]

Ran successfully, returned -1073741819
-------------------------

-------- VERSION --------
Nim Compiler Version 0.18.1 [Windows: i386]
Compiled at 2018-07-02
Copyright (c) 2006-2018 by Andreas Rumpf

git hash: d1f983b37cf1b0aa03b1424c3cca9d4212558db9
active boot switches: -d:release
-------------------------

skilchen · 2018-07-04T07:57:26Z

On my Ubuntu 16.04 nim c --threads:on -r tp.nim still produces mostly the same runtime crash as before (linenumber in threadpool changed) :

Traceback (most recent call last)
threadpool.nim(339)      slave
tp.nim(8)                :tmpWrapper
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

If compiled with -d:release it doesn't crash.

But what should this program actually do?

Silently stop after some random number of tryRecv()calls?
Receive every x from the channel that was sent to it?

It effectively stops after a random number of tryRecv() calls, even with the workaround proposed by @Araq.

The test case from #8204 only tests if the program can be compiled. It doesn't run it. With a

discard """
action: run
"""

block at the top it would probably fail on the CI platforms.

Araq · 2018-07-04T08:47:40Z

threadpool simply doesn't work well and either needs to be rewritten or deprecated.

genotrance · 2018-07-04T19:59:48Z

Please reopen this, it is broken on Linux.

genotrance · 2018-07-04T21:40:53Z

How come testament doesn't run this?

Nim/tests/testament/categories.nim

Lines 200 to 208 in c7cc934

    
           # ------------------------- threading tests ----------------------------------- 
        
           proc threadTests(r: var TResults, cat: Category, options: string) = 
        
             template test(filename: untyped) = 
        
               testSpec r, makeTest(filename, options, cat, actionRun) 
        
               testSpec r, makeTest(filename, options & " -d:release", cat, actionRun) 
        
               testSpec r, makeTest(filename, options & " --tlsEmulation:on", cat, actionRun) 
        
             for t in os.walkFiles("tests/threads/t*.nim"): 
        
               test(t)

Per the code, all threading test cases are executed with actionRun.

Araq · 2018-07-04T22:01:10Z

Good catch, mayb it did run it and yet it was green.

genotrance · 2018-07-04T23:28:06Z

Nope, it doesn't run - setting actionRun in makeTest() is completely ignored in testSpec().

Nim/tests/testament/tester.nim

Lines 298 to 302 in c7cc934

    
           if test.action != actionRunNoSpec: 
        
             expected = parseSpec(tname) 
        
           else: 
        
             specDefaults expected 
        
             expected.action = actionRunNoSpec

The default value of action ends up being actionCompile but can change to actionRun if output, exitCode or other such options are defined in the test. I see some test categories which could possibly not run the tests at all if there's nothing in the test that flips to actionRun. For example: the gc/cyclecollector.nim.

I think this code should be changed to:

  if test.action != actionRunNoSpec:
    expected = parseSpec(tname)
    if test.action == actionRun and expected.action == actionCompile:
      expected.action = actionRun
  else:
    specDefaults expected
    expected.action = actionRunNoSpec

Feedback appreciated.

Araq · 2018-07-05T10:22:08Z

Sounds good.

This reverts commit c7cc934.

FedericoCeratto · 2019-03-22T22:32:33Z

This happens with Nim 0.19.4 on Linux:

All receiver block after receiving 2 or 202, 3 and 303 and later are never received.

import os
import threadpool
import times

type EventChannel = Channel[int]

proc sender(c: ptr EventChannel, starting_from:int) {.thread.} =
  for i in 0..5:
    doAssert c[].trysend(i + starting_from) == true
    echo "sent " & $(i + starting_from)
    sleep 10

proc sender_broken(c: ptr EventChannel, starting_from: int) {.thread.} =
  var chan = c[]
  for i in 0..5:
    doAssert chan.trysend(i + starting_from) == true
    echo "sent " & $(i + starting_from)
    sleep 10

proc receiver(pc: ptr EventChannel, n:int) {.thread.} =
  var chan = pc[]
  chan.open(0)
  while true:
    echo $n & " blocks"
    let x = recv(chan)
    echo($n & " received from chan: " & $x)

var chan: EventChannel
chan.open(0)
spawn sender(addr chan, 0)
spawn sender_broken(addr chan, 100)
spawn sender(addr chan, 200)

spawn receiver(addr chan, 1)
sleep 10
spawn receiver(addr chan, 2)
sleep 10
spawn receiver(addr chan, 3)

sleep 500
echo "Finished listening"

close(chan)

Occasional SIGSEGV:

sent 0
sent 200
sent 100
1 blocks
1 received from chan: 0
1 blocks
1 received from chan: 100
1 blocks
sent 1
sent 201
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

Araq · 2019-03-25T13:05:13Z

It's invalid to copy a channel like this var chan = pc[].

narimiran · 2020-07-09T09:52:06Z

This happens with Nim 0.19.4 on Linux:
All receiver block after receiving 2 or 202, 3 and 303 and later are never received.

@FedericoCeratto, I've tried your code example with Nim devel v1.3.5 and i couldn't reproduce the SIGSEGV. Can you confirm?

narimiran · 2021-01-28T07:55:55Z

I couldn't reproduce the SIGSEGV.

The missed keyword was occasional SIGSEGV — I was writing the test case for this, everything was passing at first, and then I noticed it fails sometimes, in ~1/4 of the runs.

Araq · 2023-08-27T21:03:20Z

Use Malebolgia instead and avoid channels with thread pools.

andreaferretti added the Threads label Jun 22, 2017

vegansk closed this as completed Jul 3, 2018

Varriount pushed a commit that referenced this issue Jul 4, 2018

Test case for #5626 (#8204)

c7cc934

Araq reopened this Jul 4, 2018

Araq pushed a commit that referenced this issue Jul 5, 2018

Revert "Test case for #5626 (#8204)" (#8206)

f7b76c9

This reverts commit c7cc934.

genotrance mentioned this issue Jul 7, 2018

Testament should run tests with no action #8232

Merged

narimiran added a commit to narimiran/Nim that referenced this issue Jan 20, 2021

close nim-lang#5626 by adding a test case

1aa8026

timotheecour changed the title ~~Crash when using threadpool and channels~~ occasional SIGSEGV when using threadpool and channels Jan 29, 2021

Araq closed this as completed Aug 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

occasional SIGSEGV when using threadpool and channels #5626

occasional SIGSEGV when using threadpool and channels #5626

vegansk commented Mar 29, 2017

Araq commented Apr 22, 2018 •

edited

genotrance commented Jul 2, 2018

skilchen commented Jul 4, 2018

Araq commented Jul 4, 2018

genotrance commented Jul 4, 2018

genotrance commented Jul 4, 2018

Araq commented Jul 4, 2018

genotrance commented Jul 4, 2018

Araq commented Jul 5, 2018

FedericoCeratto commented Mar 22, 2019 •

edited

Araq commented Mar 25, 2019

narimiran commented Jul 9, 2020

narimiran commented Jan 28, 2021

Araq commented Aug 27, 2023

occasional SIGSEGV when using threadpool and channels #5626

occasional SIGSEGV when using threadpool and channels #5626

Comments

vegansk commented Mar 29, 2017

Araq commented Apr 22, 2018 • edited

genotrance commented Jul 2, 2018

skilchen commented Jul 4, 2018

Araq commented Jul 4, 2018

genotrance commented Jul 4, 2018

genotrance commented Jul 4, 2018

Araq commented Jul 4, 2018

genotrance commented Jul 4, 2018

Araq commented Jul 5, 2018

FedericoCeratto commented Mar 22, 2019 • edited

Araq commented Mar 25, 2019

narimiran commented Jul 9, 2020

narimiran commented Jan 28, 2021

Araq commented Aug 27, 2023

Araq commented Apr 22, 2018 •

edited

FedericoCeratto commented Mar 22, 2019 •

edited