[WIP] Tracing / Scripting

[t-vi]

With this patch you can get a traced/scripted MaskRCNN model. To facilitate discussion - maybe also to have a test-case for the remaining features to be wanted in the jit - I put this out in it's current state, rather than working in private towards a mergeable patch.
I appreciate your feedback but note that it's not quite ready yet.

We have:

• demo/trace_model.py produces a torch script model by a mix of tracing/scripting,
• demo/traced_model.cpp uses the model in C++.

So all of this is very raw, and there are classes of hacks described in the issue #27, in particular

• call forward directly all the time to avoid errors about tracing tensor/non-tuple functions, but my understanding is that there might be a way to have more structured types with the jit soonish,
• some script functions with funny indirections to make jit work better (though some might be unneeded by the awesome work of the PyTorch team),
• paths with alternative calculations, in particular to avoid inplace operations,
• a few custom ops, e.g. to avoid untraceable variable length loops,

Lots of Todos:

• there are warnings to be investigated, in particularly for loops and boxlist sizes
• it needs some PyTorch JIT fixes that are in master (all there, thanks!)
• clean up model, including reverting unneeded changes,
• clean up cmake, e.g. proper detection of OpenCV,
• invoke cmake / build the custom ops from setup.py,
• try a different model/config,
• maybe script more to be more flexible with sizes,
• have a leaner pretrained model.

As for my use case: It also would work on Android if PyTorch was there yet.

[fmassa]

I think the indentation in here is off

[facebook-github-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[Eric-Zhang1990]

@t-vi Can you tell me why this error shows when I am running your demo trace_model.py ? Thanks. The error is: OSError: /home/eric/Disk100G/githubProject/maskrcnn-benchmark/maskrcnn_benchmark/csrc/custom_ops/libmaskrcnn_benchmark_customops.so: undefined symbol: _ZN5torch3jit8ListType9ofTensorsEv

[t-vi]

@Eric-Zhang1990 that symbol (aka torch::jit::ListType::ofTensors()) looks like it might be in libtorch and the libmaskrcnn_benchmark_customops.so should be linked to libtorch if all goes OK. maybe the libtorch you built with isn't the one you use? With the lightning speed of JIT development necessitates my experience is that having matching versions during build and invocation often helps.

[xxradon]

@t-vi Wondelful work you had done,can you give a tutorial to reproduce your work?I met nearly same error as @Eric-Zhang1990 did. Thanks.

OSError: /home/shining/Projects/github-projects/pytorch-project/maskrcnn-benchmark/maskrcnn_benchmark/libmaskrcnn_benchmark_customops.so: undefined symbol: _ZN3c108demangleEPKc

[t-vi]

@xxradon I think it's premature to expect this to work without bumps, unfortunately. But of course, I appreciate that you are trying and I'll be very happy for suggestions how to improve the build process.

One key aspect is that you have to make 100% sure that your libtorch exactly matches the one you used to build and in Python and the headers need to be the right ones, too. Try pointing to it with LD_LIBRARY_PATH to torch/lib or so.
In your specific case: For me
objdump -T maskrcnn_benchmark/csrc/custom_ops/build/libmaskrcnn_benchmark_customops.so | c++filt | grep demang
seems to indicate that the lib doesn't require it directly, so apparently whichever libtorch is used doesn't find a matching libc10/libcaffe2...

[xxradon]

@t-vi Thanks , your suggestion is right,I can run trace_model.py without mistake and got the end_to_end_model.pt file.And when I tested trace_model.cpp,built was ok,but when I ran the demo there are some mistakes :
terminate called after throwing an instance of ' c10::Error ' what(): read_bytes == 8 ASSERT FAILED at ../caffe2/serialize/inline_container.h:182, please report a bug to PyTorch. Expected to read 8 bytes but got %llu bytes0 (read64BitIntegerLittleEndian at ../caffe2/serialize/inline_container.h:182) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6a (0x7fb099312aaa in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libc10.so) frame #1: <unknown function> + 0x5b1a86 (0x7fb0aefada86 in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libtorch.so.1) frame #2: <unknown function> + 0x5b3aef (0x7fb0aefafaef in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libtorch.so.1) frame #3: <unknown function> + 0x5b490f (0x7fb0aefb090f in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libtorch.so.1) frame #4: <unknown function> + 0x5ad9cc (0x7fb0aefa99cc in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libtorch.so.1) frame #5: torch::jit::load(std::istream&) + 0x2f4 (0x7fb0aefad2d4 in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libtorch.so.1) frame #6: torch::jit::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x3a (0x7fb0aefad49a in /home/shining/Projects/github-projects/pytorch-project/pytorch-build/torch/lib/tmp_install/lib/libtorch.so.1) frame #7: main + 0x2d6 (0x409546 in /home/shining/Projects/github-projects/pytorch-project/cpp-pytorch/Release/traced_model) frame #8: __libc_start_main + 0xf0 (0x7fb095d4a830 in /lib/x86_64-linux-gnu/libc.so.6) frame #9: _start + 0x29 (0x409c39 in /home/shining/Projects/github-projects/pytorch-project/cpp-pytorch/Release/traced_model)

And I know why this error happen,because the libmaskrcnn_benchmark_customops.so is not added into link file,but I added LINK_DIRECTORIES and target_link_libraries in CMakelist.txt,it not worked either.
So can you give me a tutorial of your CMakelist.txt to run trace_model.cpp?
Thanks a lot!!!

[t-vi]

Oh dear. I forgot to include that. I didn't actually use cmake, just

g++ traced_model.cpp -lopencv_core -lopencv_highgui -lopencv_imgcodecs -I /usr/local/lib/python3.6/dist-packages/torch/lib/include/  -L /usr/local/lib/python3.6/dist-packages/torch/lib/ -ltorch -lcaffe2 -lc10 -lopencv_imgproc -L ../maskrcnn_benchmark/csrc/custom_ops/build/ -lmaskrcnn_benchmark_customops

and then I called it with LD_LIBRARY_PATH=/usr/local/lib/python3.6/dist-packages/torch/lib/:../maskrcnn_benchmark/csrc/custom_ops/build/ ./a.out.

It less than fancy, sorry about that.

[xxradon]

Oh dear. I forgot to include that. I didn't actually use cmake, just

g++ traced_model.cpp -lopencv_core -lopencv_highgui -lopencv_imgcodecs -I /usr/local/lib/python3.6/dist-packages/torch/lib/include/  -L /usr/local/lib/python3.6/dist-packages/torch/lib/ -ltorch -lcaffe2 -lc10 -lopencv_imgproc -L ../maskrcnn_benchmark/csrc/custom_ops/build/ -lmaskrcnn_benchmark_customops

and then I called it with LD_LIBRARY_PATH=/usr/local/lib/python3.6/dist-packages/torch/lib/:../maskrcnn_benchmark/csrc/custom_ops/build/ ./a.out.

It less than fancy, sorry about that.

Thanks for your reply,but I got the same mistake...Can you try use CMakelist.txt instead,or give a more specific tutorial?
Thanks a lot.

[t-vi]

Can you actually read the .pt from torch?
(This is the exact error message you get when libtorch cannot open the file/it doesn't exist - I'll put up a PR for a better message.)

[Eric-Zhang1990]

@t-vi Now I can trace and save model, but when I run your trace_model.cpp file, it shows an error about nms, which is:
Starting /home/eric/Disk100G/githubProject/trace_model_cpp_test/build/traced_model...
terminate called after throwing an instance of 'load model ok
torch::jit::script::ErrorReport'
what():
Schema not found for node. File a bug report.
Node: %3556 : Dynamic = maskrcnn_benchmark::nms(%3550, %3554, %3555)

Input types:Dynamic, Dynamic, float
candidates were:
.
Can you tell me how I can fix it?
Thanks a lot.

[t-vi]

@Eric-Zhang1990 Did you find and link to the libmaskrcnn_benchmark_customops.so (see the g++ above - I'm looking to provide cmake)...

[t-vi]

With the latest set of changes you should get the custom ops built for you. The C++ demo now has a CMakeLists.txt, but it isn't built for now (for lack of a good place to put the binary).

[fmassa]

question: do we need to register the backwards for the op in C++ then so that we can perform training properly using the same codepath?

[t-vi]

The current obstacle here is that we cannot trace through the Python autograd function.
I'm not aware of a way to register the derivative so we could go through the op directly during training as well.

[fmassa]

Thanks! @goldsborough do you know if we can register derivatives for custom ops?

[t-vi]

Peter responded elsewhere, and "not quite yet", but it is a to-do on his list.

[t-vi]

So I have a cleanup todo in `box_head/inference.py. Done.

For the .forward elimination. We'd have to wait on @zdevito for advice whether we can patch torch.nn.Module tracing to allow our boxlist's by using _get_tensors on inputs and outputs (or we could do our own subclass).

The other changes to the main modules I saw are in the detailed comments above.

[xxradon]

@t-vi Thanks for your contribution,I can run demo/traced_model.cpp and get the right result.But everything is cpu mode,I know the current method to build libmaskrcnn_benchmark_customops only with gcc ,so is there a way to run our serialized .pt file in GPU mode.
I have tried to add nvcc compilation to build libmaskrcnn_benchmark_customops.so,but failed with some mistakes,and I am working on it.

[t-vi]

There isn't currently, but it should be not too hard. We'd need GPU support for unifying the code paths, so if you have it, I'd be happy to use it, if you don't, I'll try to get to it todayish.

[t-vi]

Just so noone is disappointed too much: tracing on GPU won't work currently due to #13969 (at least).
I think we might just wait for that to be fixed tracing on GPU is supported. The obvious workarounds are scripting, but I'm sure it's good to fix PyTorch here and there if we can.

[kleisauke]

Added some code suggestions to solve the compatibility of tracing/scripting on Python 2.

[kleisauke]

Suggested change Beta

Give us feedback

	def my_paste_mask(mask, bbox, height: int, width: int, threshold: float=0.5, padding: int=1, contour: bool=True, rectangle: bool=False):
	def my_paste_mask(mask, bbox, height, width, threshold=0.5, padding=1, contour=True, rectangle=False):
	# type: (Tensor, Tensor, int, int, float, int, bool, bool) -> Tensor

[kleisauke]

Suggested change Beta

Give us feedback

	def add_annotations(image, labels, scores, bboxes, class_names: str=','.join(coco_demo.CATEGORIES), color=torch.tensor([255, 255, 255], dtype=torch.long)):
	def add_annotations(image, labels, scores, bboxes, class_names=','.join(coco_demo.CATEGORIES), color=torch.tensor([255, 255, 255], dtype=torch.long)):
	# type: (Tensor, Tensor, Tensor, Tensor, str, Tensor) -> Tensor

[kleisauke]

Suggested change Beta

Give us feedback

	def combine_masks(image, labels, masks, scores, bboxes, threshold: float=0.5, padding: int=1, contour: bool=True, rectangle: bool=False, palette=torch.tensor([33554431, 32767, 2097151])):
	def combine_masks(image, labels, masks, scores, bboxes, threshold=0.5, padding=1, contour=True, rectangle=False, palette=torch.tensor([33554431, 32767, 2097151])):
	# type: (Tensor, Tensor, Tensor, Tensor, Tensor, float, int, bool, bool, Tensor) -> Tensor

[kleisauke]

Suggested change Beta

Give us feedback

	def merge_levels(levels, unmerged_results: List[torch.Tensor]):
	def merge_levels(levels, unmerged_results):
	# type: (Tensor, List[Tensor]) -> Tensor

[kleisauke]

Suggested change Beta

Give us feedback

	import numpy
	from __future__ import division
	import numpy

[t-vi]

Thanks for the reminder that there still is Python 2. :) I'll update the PR in due course.

[t-vi]

pytorch/pytorch#14992 is keeping the scripted version of the merge_levels from working with 1.0. The obvious work-around would be to move it into a custom op (as it was in the initial version).

On the up side, we might be closer to tracing with GPU, as the creation tracing is fixed now.

I you think it makes sense to split out the move to custom ops and also the "uncontroversial" parts of the model changes into separate PRs, I'd do that.

[fmassa]

I think there are a few parts of this PR that can be sent separately and could be merged straightaway:

• PR1: change signature of the functions so that they can be custom ops (float -> double etc)
• PR2: out-of-place ops via torch.stack / torch.cat instead of in-place modification (although in-place modifications are now supposed to work on the JIT I believe)
• PR3: minor refactoring detections_to_keep into a function for easier JIT
• PR4: misc improvements to make the jit happier (add tuple for the return argument, etc).

Then we will be almost there for the JIT, and the misc changes can be discussed afterwards.
But I'd like to keep this PR open for now.

What do you think?

[t-vi]

Sounds good, Thanks Francisco! I'll get those parts going and also check with inplace ops (even though I'm a bit sceptical as my repro for that pytorch/pytorch#14992 seems to also work with indexed assignments - while the ops probably work, there seems to be some optimization going on that might not be desirable for us).
Would you also take the switch to custom ops itself?

[fmassa]

Would you also take the switch to custom ops itself?
Not for the next few PRs, given that it's not really clear to me that it is working 100% for backwards computation, and I'd like to be cautious about it