Libtorch segmentation fault with gaga

[tbaudier]

From Hermann:

I am facing facing some challenges with gaga in GATE.

Using the current gate-development branch, I managed to compile it using
Ubuntu 20.04 LTS and the cuda and torch libraries:

• cudnn-11.2-linux-x64-v8.1.1.33
• libtorch-cxx11-abi-shared-with-deps-1.8.1+cu102

However, when starting a simulation it crashes with a segmentation fault somewhere in the pytorch initialization function.
To avoid that it is from my trained models, I downloaded the one from https://github.com/OpenGATE/GateContrib/blob/master/dosimetry/gaga-phsp/pth/001.pt but the segmentation fault is still there.

[tbaudier]

• usually the problem of "pre-cxx11 ABI" or "cxx11 ABI" versions is visible during the compliation of Gate. So I think that if you succeed to compile Gate with torch, that does it mean you have the correct ABI version of torch
• For benchmarks (https://github.com/OpenGATE/GateBenchmarks) I used libtorch v1.4.0 cpu version. I do not need to train the network, this is why I used cpu version (it's easier to install on remote machines).
  So maybe 2 directions to help you, try an older version of libtorch (1.7 or 1.4 for example) and try cpu version.

[tbaudier]

From Herman

I liked the idea to use only the cpu version of libtorch to reduce the complexity as it removes cuda from the picture. Unfortunately, the error still persists.
I have added some details below.
Using the newest libtorch cpu in version 1.8.1 (downloaded directly, without pip/conda from https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcpu.zip), compilation and linking works without errors.

For testing I used the example provided in GateContrib gaga-phsp. Using the example with a conventional root file as source, everything works fine.
However using gaga as source, a segmentation fault occurs:

Gate mac/main.mac 
[G4] 
[G4] **************************************************************
[G4]  Geant4 version Name: geant4-10-07-patch-01    (5-February-2021)
[G4]                        Copyright : Geant4 Collaboration
[G4]                       References : NIM A 506 (2003), 250-303
[G4]                                  : IEEE-TNS 53 (2006), 270-278
[G4]                                  : NIM A 835 (2016), 186-225
[G4]                              WWW : http://geant4.org/
[G4] **************************************************************
[G4] 
[Core-0] Initialization of geometry
[Core-0] Initialization of physics
[Core-0] Initialization of actors
[Core-0] 
[Core-0] *************************************************
[Core-0]  GATE version 9.1 (2021)
[Core-0]  Copyright : OpenGATE Collaboration
[Core-0]  Reference : Phys. Med. Biol. 49 (2004) 4543-4561
[Core-0]  Reference : Phys. Med. Biol. 56 (2011) 881-901
[Core-0]  Reference : Med. Phys. 41(6)    (2014)
[Core-0]  http://www.opengatecollaboration.org        
[Core-0] *************************************************
[Core-0] 
[Core-0] You are using Geant4 version 10.7.1
[Core-0] Starting macro mac/main.mac
-----------------------------------------------

Nb of moves:       1

	GATE object:        'phsp_vol/placement'
	Is enabled?         Yes
	Move type:     placement
	Translation:       0 0 0 fm 
	Rotation axis:     (0,0,0)
	Rotation angle:    0 deg

Nb of repeaters:       0
-----------------------------------------------

[Physic-0] The following Geant4's physic-list is enabled :emstandard_opt1
[Core-0] Initialization of geometry
[Physic-1]   
[Physic-1]  ----------------------------------------------------------
[Physic-1]                      Ionization potential 
[Physic-1]   - worldDefaultAir	 defaut value: I = 85.7031 eV 
[Physic-1]   - Air	 defaut value: I = 85.6694 eV 
[Physic-1]   - Vacuum	 defaut value: I = 19.2 eV 
[Physic-1]   - Water	 defaut value: I = 68.9984 eV 
[Physic-1]  ----------------------------------------------------------
[Core-0] Initialization of physics
[Core-0] Initialization of actors
[Actor-1]  Dose DoseActor    = 'dose_3d'
	Dose image        = 0
	Dose squared      = 0
	Dose uncertainty  = 0
	Dose to water image        = 0
	Dose to water squared      = 0
	Dose to water uncertainty  = 0
	Edep image        = 1
	Edep squared      = 0
	Edep uncertainty  = 1
	Number of hit     = 0
	     (last hit)   = 1
	Dose algorithm    = VolumeWeighting
	Mass image (import) = 
	Mass image (export) = 
	EdepFilename      = output/dose-Edep.mhd
	DoseFilename      = output/dose-Dose.mhd
	Dose by regions           = 0
	DoseByRegionsInput        = 
	DoseByRegionsOutput       = 
	Number of regions         = 0
	Nb Hits filename  = output/dose-NbOfHits.mhd
GateSourcePhaseSpace::AddFile Add data/pth/001.pt
[Acquisition-0]   
[Acquisition-0] ============= Source initialization =============
[Actor-1]  GateSourcePhaseSpace InitializePyTorch
[Actor-1]  GateSourcePhaseSpace reading data/pth/001.pt

 *** Break *** segmentation violation

I am using gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04), geant4.10.7p01, root-6.22.06 and the current Gate development branch from github (commit 32db66b from David Sarrut on Fri Apr 9 09:04:33 2021 +0200).

I also tried tried to compile using the binary of libtorch cpu in version 1.4 (downloaded directly, without pip/conda from https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.4.0%2Bcpu.zip)

Unfortunately I get some errors during linking with some G4RunManager and geometry errors such as
/usr/bin/ld: CMakeFiles/GateLib.dir/source/geometry/src/GateCone.cc.o: in function G4Cons::CheckDPhiAngle(double)': GateCone.cc:(.text._ZN6G4Cons14CheckDPhiAngleEd[_ZN6G4Cons14CheckDPhiAngleEd]+0x294): undefined reference to G4Exception(char const*, char const*, G4ExceptionSeverity, std::basic_ostringstream<char, std::char_traits, std::allocator >&)'

[tbaudier]

I investigate your problem today but for the moment I do not have a solution :/

You described 2 errors. The second error is a link error and it's because you do not choose the correct libtorch version due to your gcc version. You downloaded: https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.4.0%2Bcpu.zip but you need https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcpu.zip (ABI version).

For the first error, I succeed to reproduce your segmentation fault in a docker version of ubuntu 20.04. It's very strange because I do not have any problem with a centos or opensuse docker image with version 1.8.1 of libtorch, this is why I did not detect the bug. By the way, with version 1.4.0 of libtorch, it worked. I tried with version 1.7.1 of libtorch and it segfault too.

[tbaudier]

From Hermann

I tried the other 1.4 version which you recommended and it works! :-)))))))))
For my application version 1.4 is sufficient, but it may help to include the link to both 1.4 versions on the Gate homepage, just in case.
I have found no easy way to determine the download path for these old versions. It was a guessing game as only the links for pip/conda are advertised.

Concerning the newer versions I would be curious to know what gcc on Ubuntu does differently...

[tbaudier]

For the moment, I did not find a solution for the problem. But I found similar issue on Github (pytorch/pytorch#49460) and the same behavior. The problem occurs with libtorch >= v1.6.0.

I will add the link of libtorch in the documentation.