(nil) (unknown) Aborted (core dumped)

[GuoxingYan]

When I try to run you training script to train a resnet101 cascade faster rcnn, I encounter a error like this:

F0530 22:30:27.787274 1077 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7ff7618205cd google::LogMessage::Fail()
.............................
Aborted (core dumped)

i know it because memory is not enough.but When I try to run you training script to train a resnet50 cascade faster rcnn, I encounter the other error like this:

@     0x7f999f9855d5  (unknown)

I0530 22:31:45.417707 1167 layer_factory.hpp:77] Creating layer scale2a_branch2b
I0530 22:31:45.418323 1167 net.cpp:122] Setting up scale2a_branch2b
I0530 22:31:45.418344 1167 net.cpp:129] Top shape: 1 64 250 150 (2400000)
I0530 22:31:45.418352 1167 net.cpp:137] Memory required for data: 276001080
I0530 22:31:45.418380 1167 layer_factory.hpp:77] Creating layer res2a_branch2b_relu
I0530 22:31:45.418406 1167 net.cpp:84] Creating Layer res2a_branch2b_relu
I0530 22:31:45.418423 1167 net.cpp:406] res2a_branch2b_relu <- res2a_branch2b
I0530 22:31:45.418457 1167 net.cpp:367] res2a_branch2b_relu -> res2a_branch2b (in-place)
@ 0x7f999a5966ba start_thread
I0530 22:31:45.418839 1167 net.cpp:122] Setting up res2a_branch2b_relu
I0530 22:31:45.418860 1167 net.cpp:129] Top shape: 1 64 250 150 (2400000)
I0530 22:31:45.418870 1167 net.cpp:137] Memory required for data: 285601080
I0530 22:31:45.418885 1167 layer_factory.hpp:77] Creating layer res2a_branch2c
I0530 22:31:45.418934 1167 net.cpp:84] Creating Layer res2a_branch2c
I0530 22:31:45.418954 1167 net.cpp:406] res2a_branch2c <- res2a_branch2b
I0530 22:31:45.418989 1167 net.cpp:380] res2a_branch2c -> res2a_branch2c
@ 0x7f99aa19641d clone
@ (nil) (unknown)
Aborted (core dumped)

22/5000
Why are these two errors different? Can you give me some advice about the latter?

[pyupcgithub]

I got the same question. Have you solved it ? @GuoxingYan

[chanyixialex]

@GuoxingYan you solve it? I encounted the same problem. thanks!

[zhaowujie]

Becasue you didn't change root_folder in train.prototxt and test.prototxt