Semantic segmentation work-flow

[gheinrich]

No description provided.

[lukeyeager]

This link doesn't work. I guess it needs the "http://"?

[gheinrich]

Thanks, updated in latest commit.

[lukeyeager]

• I can't view the labels DB for some reason
• You should make a note that this requires NVcaffe 0.15 to avoid errors like Message type "caffe.LayerParameter" has no field named "crop_param".

[gheinrich]

I can't view the labels DB for some reason

did you set the label encoding to 'png'?

you should make a note that this requires NVcaffe 0.15

Well spotted, thanks! (BVLC/master is also fine)

[lukeyeager]

did you set the label encoding to 'png'?

Ah sorry, sloppy reading.

Got it to work! Cool stuff!

[gheinrich]

I have added a note about the need for NVcaffe 0.15.

[jjmata]

Not sure this is supposed to actually work yet (assume so?) but I've followed the steps and keep hitting this error after a few hours of training:

Running from a fork of your repo, @gheinrich on the tip of the dev/semantic-segmentation branch.

Sorry, should have probably checked this first ... the Caffe logs seem to indicate and out-of-memory situation in the GPU?

*** SIGSEGV (@0xffffffff) received by PID 21336 (TID 0x7fe23d125a40) from PID 18446744073709551615; stack trace: ***
@     0x7fe23af54cb0 (unknown)
@     0x7fe21f30604a (unknown)
@     0x7fe21f314ae4 (unknown)
@     0x7fe21f276921 (unknown)
@     0x7fe21f26124d (unknown)
@     0x7fe23bfc05ce (unknown)
@     0x7fe23bfedc0c (unknown)
@     0x7fe23ca19715 caffe::GPUMemory::Manager::deallocate()
@     0x7fe23c9e3129 caffe::SyncedMemory::~SyncedMemory()
@     0x7fe23c998082 boost::detail::sp_counted_impl_p<>::dispose()
@     0x7fe23c98eda9 caffe::Blob<>::~Blob()
@     0x7fe23ca2c1b0 caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer()
@     0x7fe23ca7abf9 caffe::DataLayer<>::~DataLayer()
@           0x40df96 std::vector<>::~vector()
@           0x40ec51 caffe::Net<>::~Net()
@           0x40ee19 caffe::Net<>::~Net()
@     0x7fe23c9ac310 caffe::SGDSolver<>::~SGDSolver()
@           0x40d05e boost::detail::sp_counted_base::release()
@           0x40aaf4 train()
@           0x4086bc main
@     0x7fe23af3ff45 (unknown)
@           0x408e8d (unknown)
@                0x0 (unknown)

[gheinrich]

Hi @jjmata, this should work if you use dev/semantic-segmentation on my fork. Which version of Caffe are you using?

[lukeyeager]

Yeah, deallocating an invalid pointer sounds like a caffe bug.

[lukeyeager]

To match the above desc, should this be "single-channel images (paletted or grayscale)"?

[gheinrich]

Sure, I will make the two tool tips the same.

[jjmata]

I'm on NVIDIA's 0.15.11 (and using your branch) ... sure 4GB should be enough? This is one of those dreadful AWS grid GPUs, FWIW!

[gheinrich]

I'm on NVIDIA's 0.15.11 (and using your branch) ... sure 4GB should be enough?

I presume so. When training on VOC with the default batch size of 1, GPU memory utilization is under 2GB. I am using NV-Caffe 0.15.9 (the version that got released through .deb packages).

[gheinrich]

@lukeyeager thanks for the review comments. I have updated the patch.

[lukeyeager]

Thanks! Great stuff.