SIGSEGV in a very specific setting

[mbollmann]

When running

normalize -c test.cfg -f test.txt --train

with the attached sample files, Norma (0.5.0 with ICU strings) randomly segfaults (SIGSEGV) about 80% of the time.

This...

• happens only when the parameter files are not yet created
• happens also when the lines in test.txt are followed by 50000+ more tokens
• only seems to happen when the lines in test.txt are at the beginning of a file
• is the only segfault I experienced in almost 1.2 millions tokens of training

There must be something special going on here, though I don't know what it is.

test.cfg.txt
test.txt

[fpetran]

If the segfault does not occur 100% of the times, it may be due to race conditions in the multithreading. Can you send a backtrace?

[mbollmann]

This line appears to be the culprit:

https://github.com/comphist/norma/blob/master/src/normalizer/rulebased/rule.cpp#L224

Maybe the array index can go out of bounds?

Full backtrace:
backtrace.txt