Memory leak in ImageNet.LoadImageAndResize224 #24
Comments
|
Hi @Claus1 , Thank you for letting me know. I have reworked to fix the issue and tested on both CPU and GPU. On GPU, memory stayed absolutely still. On CPU, it was volatile up and down in a range of dozen MB, so I hope it is fixed. Please test it on branch Please let me know how you go with the fix. If everything okay, I will merge for a new version tag. |
|
Hi @sugarme ! Can not compile directly @image on my Radeon computer: So I run setup.sh from the created by go folder go/pkg/mod/github.com/sugarme/gotch@v0.3.9-0.20210410094112-93dc63424aef which has updated files. After that I run my program and got the same leak. Possibly Go uses the previous library. |
|
Yep, It used old code. After 800 calls the topic code |
|
Hi @Claus1 , Executing In the next version, there will be a seperate
In your machine,
# github.com/sugarme/gotch/libtch
/usr/bin/ld: cannot find -lcuda
/usr/bin/ld: cannot find -lcudart
/usr/bin/ld: cannot find -lcublas
/usr/bin/ld: cannot find -lcudnn
/usr/bin/ld: cannot find -lcaffe2_nvrtc
/usr/bin/ld: cannot find -lnvrtc-builtins
/usr/bin/ld: cannot find -lnvrtc
/usr/bin/ld: cannot find -lnvToolsExt
/usr/bin/ld: cannot find -lc10_cuda
/usr/bin/ld: cannot find -ltorch_cuda
collect2: error: ld returned 1 exit statusJust ignore them. Make sure you add/update these lines to your export GOTCH_LIBTORCH="/usr/local/lib/libtorch"
export LIBRARY_PATH="$LIBRARY_PATH:$GOTCH_LIBTORCH/lib"
export CPATH="$CPATH:$GOTCH_LIBTORCH/lib:$GOTCH_LIBTORCH/include:$GOTCH_LIBTORCH/include/torch/csrc/api/include"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GOTCH_LIBTORCH/lib"Hope that helps. |
|
I did it in another way. The last error I got from @image branch. |
|
@Claus1 , Last error: fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c1 pc=0x1c1]
runtime stack:
runtime.throw(0x12fa241, 0x2a)
/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:726 +0x269
goroutine 1 [syscall]:
runtime.cgocall(0x101fee0, 0xc0002b9ab8, 0x12f8d7f)
/usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc0002b9a70 sp=0xc0002b9a38 pc=0x461e7b
github.com/sugarme/gotch/libtch._Cfunc_atg_totype(0x7f3eec2fc660, 0x7f3eec2fc680, 0x6)
_cgo_gotypes.go:22434 +0x45 fp=0xc0002b9ab8 sp=0xc0002b9a70 pc=0xe6b905
github.com/sugarme/gotch/libtch.AtgTotype.func1(0x7f3eec2fc660, 0x7f3eec2fc680, 0xc000000006)
/home/george/go/pkg/mod/github.com/sugarme/gotch@v0.3.8/libtch/c-generated.go:6146 +0xac fp=0xc0002b9b10 sp=0xc0002b9ab8 pc=0xee030c
github.com/sugarme/gotch/libtch.AtgTotype(0x7f3eec2fc660, 0x7f3eec2fc680, 0xc000000006)
/home/george/go/pkg/mod/github.com/sugarme/gotch@v0.3.8/libtch/c-generated.go:6146 +0x4c fp=0xc0002b9b60 sp=0xc0002b9b10 pc=0xe958cc
github.com/sugarme/gotch/tensor.(*Tensor).Totype(0xc0002dad88, 0x13be3c0, 0x1164a40, 0x0, 0x0, 0x0, 0x0)
/home/george/go/pkg/mod/github.com/sugarme/gotch@v0.3.8/tensor/tensor-generated.go:17916 +0x106 fp=0xc0002b9c00 sp=0xc0002b9b60 pc=0xfedd46
github.com/sugarme/gotch/vision.(*ImageNet).Normalize(0xc00000f700, 0xc0002dad88, 0x0, 0x0, 0x0)
/home/george/go/pkg/mod/github.com/sugarme/gotch@v0.3.8/vision/imagenet.go:36 +0xfb fp=0xc0002b9d08 sp=0xc0002b9c00 pc=0x1006fdb
github.com/sugarme/gotch/vision.(*ImageNet).LoadImageAndResize224(0xc00000f700, 0xc0002d0cd0, 0x46, 0x0, 0x0, 0x0)
/home/george/go/pkg/mod/github.com/sugarme/gotch@v0.3.8/vision/imagenet.go:149 +0x24d fp=0xc0002b9dd8 sp=0xc0002b9d08 pc=0x10083ed
main.ProbsImage(0xc0002d0cd0, 0x46, 0xc000010218, 0x0, 0x0, 0x0)
/home/george/Projects/unigui-go/examples/engine/main.go:220 +0xae fp=0xc0002b9e90 sp=0xc0002b9dd8 pc=0x100c28e
main.main()
/home/george/Projects/unigui-go/examples/engine/main.go:200 +0x20d fp=0xc0002b9f88 sp=0xc0002b9e90 pc=0x100c10d
runtime.main()
/usr/local/go/src/runtime/proc.go:204 +0x1cf fp=0xc0002b9fe0 sp=0xc0002b9f88 pc=0x49df6f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0002b9fe8 sp=0xc0002b9fe0 pc=0x4d4421
goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00007e7d0)
/home/george/go/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:154 +0x19e
created by go.opencensus.io/stats/view.init.0
/home/george/go/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:32 +0x5bShowed that you haven't installed new version and your program was still using
I have tested on CPU and GPU and fresh Google Colab machines and all worked well. |
|
To be able to reproduce the issue, please do a fresh installation of |
|
@surgame, I followed the instructions and got an error on Step2: |
|
@Claus1 , I thought you have aready had
|
|
@sugarme , I removed 0.3.9-rc1 repeat all steps even removed and reinstalled libtorch and got the same error. |
|
If I make the omitted files manually I get such error: Install script removes my addition files and after reports about omitting. |
|
@Claus1 , If you look at the script sudo mv $GOTCH_PATH/libtch/dummy_cuda_dependency.cpp $GOTCH_PATH/libtch/dummy_cuda_dependency.cpp.gpu
sudo mv $GOTCH_PATH/libtch/fake_cuda_dependency.cpp.cpu $GOTCH_PATH/libtch/fake_cuda_dependency.cppJust two swap default I saw from your provided log that your machine user |
|
@Claus1 , Please delete |
|
@sugarme How I can delete during installing setup-gotch? I removed the package and libtorch before installing, work from go directory as you advised and get this state with the error |
|
@Claus1 , Those file names look okay for me. Can you just go to your project and update to v0.3.9-rc1 by executing |
|
|
@Claus1 , Please do:
|
|
@sugarme, I removed all gotch export variables reinstall libtorch and gotch again from scratch. I switched to vs code and try to compile code (what else I can do). |
|
@Claus1 , Your If not, put those lines to then do |
|
@sugarme Yes, GOTCH_LIBTORCH pointed to libtorch inside gotch pkg. Fixed. After I do And open Vs and see the same error in Problems. |
|
@Claus1 , The error is because of package libtch
// #cgo CFLAGS: -I${SRCDIR} -O3 -Wall -Wno-unused-variable -Wno-deprecated-declarations -Wno-c++11-narrowing -g -Wno-sign-compare -Wno-unused-function
// #cgo CFLAGS: -I/usr/local/include
// #cgo CFLAGS: -D_GLIBCXX_USE_CXX11_ABI=1
// #cgo LDFLAGS: -lstdc++ -ltorch -lc10 -ltorch_cpu -L/lib64
// #cgo CXXFLAGS: -std=c++17 -I${SRCDIR} -g -O3
// #cgo CFLAGS: -I${SRCDIR}/libtorch/lib -I${SRCDIR}/libtorch/include -I${SRCDIR}/libtorch/include/torch/csrc/api/include -I${SRCDIR}/libtorch/include/torch/csrc
// #cgo LDFLAGS: -L${SRCDIR}/libtorch/lib
// #cgo CXXFLAGS: -I${SRCDIR}/libtorch/lib -I${SRCDIR}/libtorch/include -I${SRCDIR}/libtorch/include/torch/csrc/api/include -I${SRCDIR}/libtorch/include/torch/csrc
import "C"then The current setup seems to make it very hard for installation/updating and I am preparing a different way to improve it. |
|
@Claus1 , If you still struggle with getting gotch up and running, please try this new shell script:
|
|
@sugarme Ignore pls the previous message. Now it compiles ok with 0.3.9 cpu as desired. I try to figure out a reason. |
|
My jpg file causes this error although it looks normal. So the last setup-gotch.sh works ok. It fixes the building process. Memory leak fixed. Thanks a lot! |
The code
causes memory leak about 1-1.5 MB on every call.
The text was updated successfully, but these errors were encountered: