libprotobuf error causes crash in the middle of training

[Feynman27]

I'm running into a strange error. During training (after several thousand iterations), my training script crashes with the error

libprotobuf FATAL google/protobuf/wire_format.cc:830] CHECK failed: (output->ByteCount()) == (expected_endpoint): : Protocol message serialized to a size different from what was originally expected.  Perhaps it was modified by another thread during serialization?
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (output->ByteCount()) == (expected_endpoint): : Protocol message serialized to a size different from what was originally expected.  Perhaps it was modified by another thread during serialization?
Command terminated by signal 6

It doesn't look like I'm running out of memory on my gpu. Could this be a result of writing too much information to tensorboard?

[ruotianluo]

I met this too. It doensn't always happen for me. But it does happen occasionally. Currently I just ignore this. But sure it's good to investigate it sometime

[Feynman27]

It's happening to me ~ every 10k iterations after 130k iterations. Before this, it only happened every 70k iterations or so. Do you have any idea what may be causing it. I'm also getting a bad_alloc error, so something funny is happening with the memory.

[ruotianluo]

For now, just replace https://github.com/ruotianluo/pytorch-faster-rcnn/blob/master/lib/model/train_val.py#L252
with
if False:

[Feynman27]

Yeah, I just commented it out. I'll post if I discover a more permanent solution.