libtorch exports protobuf symbols #14573
Assignees
Labels
high priority
module: abi
libtorch C++ ABI related problems
module: build
Build system issues
module: cpp
Related to C++ API
module: protobuf
triaged
This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Comments
|
I'm pretty sure that we aren't exporting the symbols in the Python extension. In any case, this is definitely a mistake. |
|
What is the source of the libtorch that you're using? Are you using the nightly libtorch or libtorch from a different nightly package? If you're using the nightly libtorch, which variation of it are you using (shared-with-deps, static-without-deps, etc.) ? EDIT: I found protobuf symbols in the nightly libtorch packages. We're investigating now |
|
Yes, it was the nightly. You are awesome! In case it helps, you can use my minimal example from the issue linked above together with this cmake script to reproduce: |
|
I've been able to link on Torch's protobuf. The cmake protobuf-config.cmake file is in pytorch/torch/lib/tmp_install/cmake. You can use it directly with and by adding the cmake argument |
|
If you really need to link on protobuf (with the solution above I got unresolved symbols and build type mismatch [/MTd vs /MDd on Windows]), you can simply copy the third_party/protobuf somewhere else and compile it yourself. You'll get the exact same version. |
ezyang
added
high priority
module: cpp
|
Is there any update on this? I was able to compile a project with both libtorch and protobuf by compiling libtorch from source with the desired version of protobuf, but it would be nice if it weren't necessary to do so. Thanks! |
|
This was never completed; it fell into the backlog behind the library unification. @kostmo is this on your roadmap? |
|
any updates, please? |
|
There is no one currently working on this, but @ljk53 might end up accomplishing this while stripping symbols out for the mobile builds. IIUC his approach using linker scripts could work here as well for linux builds |
|
I believe mobile is not linking protobuf, so the symbols probably aren't there but the approach won't fix non-mobile builds. We should at least verify this is still an issue. |
|
We have hit this same issue. It's really bizarre. Do you have any idea why libtorch is exhibiting this behavior? |
|
My current understanding is as follows: Lines 174 to 190 in d8c66c1 cc. @dzhulgakov @soumith to confirm whether this understanding is correct. If it is correct, it means we should either not expose protobuf symbols in any of Caffe2 and ONNX public APIs, or provide instructions on how to compile other libraries using the protobuf version shipped with libtorch. |
|
Has anything changed in this regard since October? |
|
I've been scratching my head on this, maybe @malfet can fill me in so I can better understand the state of it. From what I do know:
I probably wrote a bunch of nonsense, and that's what I've concluded after digging through this for a day. For context, I started to try and pull the latest version of protobuf (3.25.1 / 25.1 ) and while my downloaded pytorch 2.1.0 works on MacOSX, I get linker errors on Linux: My build on mac OSX and Linux both share this cmake script I added: Looking through the release code, I see that but not sure what to make of that either. If there could be some comment here that explains the state of protobuf, and the nuances/caveats we need to know about using it with relation to PyTorch, that would be super helpful. |
|
Following up here -- if I have the following in my build: which is needed for some other libraries/integrations, how do I navigate this protobuf install with PyTorch? |
|
Lots of things changed since 2018, one of them is that we no longer build PyTorch with caffe2 by default, and I have not heard of anyone who want to use libtorch and at the same time be able to export model to ONNX. (@BowenBao have you heard of such usecases) @anthonyalayo just to confirm, are you seeing this in shared or in static library? |
|
@malfet here's the exposed symbols from the shared library: Checking out one of the nightly builds: I can see the following: If disabling Caffe2 and Onnx is an option for the nightly, that would be great. Otherwise we need to build PyTorch from source in order to change the version of protobuf, which is the case I am hitting. Of course another option would be to upgrade Protobuf to a more mainstream version (v22 and later, which has a breaking change with dependency on abseil). |
|
I can make the PR, just wanted to confirm this means we can turn off Onnx and/or Caffe2 @malfet ? |
|
Caffe2 integration has been disabled 2+ years ago, and removing ONNX integration from libtorch builds sounds fine to me, but I would defer to @BowenBao opinion here |
|
@BowenBao friendly bump |
|
@anthonyalayo please do not ping people all the time, just propose a PR and we can discuss it there (Or I can try to find time next week to write one) |
|
Hi @anthonyalayo feel free to go ahead and give it a shot. I'm not aware of any existing use cases of ONNX in libtorch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
libtorch, the amazing c++ interface for Pytorch, exports the symbols of it's linked libprotobuf.
This apparently requires me to use the same version of libprotobuf and protoc for my project when depending on libtorch.
Attempting to link another version of protobuf caused memory corruption in my case.
Proposed Solutions
A) Don't export the symbols, if that's possible.
B) Document the libprotobuf version used by libtorch. This way, I could at least fall back to that version for now. However, I was not able to find the used version.
The text was updated successfully, but these errors were encountered: