Segmentation fault during trainning

[mattiadg]

Hi,

We've written a new model file using a LSTM-based encoder and the Transformer decoder. Most of the code was a copy-paste of existing classes, especially for the transformer.

The training works, but in every training run, we get at random moments a segmentation fault with no additional errors and no backtrace. We have also tried to run the training inside gdb, following some suggestions we found on stackoverflow, but despite the occurrence of the "segmentation fault" text, the program exited normally according to gdb.

What is really strange is that it can occur at any moment, it occured during the third epoch, but also during the 137th.

Has anything like this ever happened to you?

Thank you.