EXCEPTION_ACCESS_VIOLATION from Flux.gradient() using Threads.@threads and Flux.gpu() #36752
Description
I tried my absolute best looking for equivalent issues. I saw similar errors, but do not know enough to understand if they are the same problem. But the output said to post this as an issue, so here I am.
The idea is very simple, train some Flux models on the gpu, but have the models generated and trained from different cpu threads. As near as I can tell, this should be thread-safe to do, because everything that would get mutated in training is thread-local. The data should not be mutated by training.
https://gist.github.com/MacKenzieHnC/7596910b83f7351e92a5fd5c5dcdc94d
# simple_double_threaded.jl
using Flux
using Statistics
import CuArrays
CuArrays.allowscalar(false)
# generate a bunch of data
X = rand(100)
Y = 0.5X + rand(100)
Xd = reduce(hcat,X)
Yd = reduce(hcat,Y)
# global gpu data
data = gpu([(Xd,Yd)])
# cpu threads
Threads.@threads for i in 1:100 # lower values only fail sometimes???
# gpu model
model = gpu(Dense(1,1))
loss(x, y) = mean((model(x).-y).^2)
opt = ADAM()
par = params(model);
for j in 1:100 # lower values only fail sometimes???
# Training crashes gloriously
Flux.train!(loss,par,data,opt)
end
end
So why does it fail so dramatically? And so inconsistently when using real-world values??
Version Info:
julia> using InteractiveUtils
julia> versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
JULIA_DEPOT_PATH = C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1;C:\Users\Lepre\AppData\Local\JuliaPro 1.4.2-1\Julia-1.4.2\local\share\julia;C:\Users\Lepre\AppData\Local\JuliaPro 1.4.2-1\Julia-1.4.2\share\julia
JULIA_EDITOR = "C:\Users\Lepre\AppData\Local\JuliaPro 1.4.2-1\app-1.47.0\atom.exe" -a
JULIA_NUM_THREADS = 6
JULIA_PKG_SERVER = pkg.juliacomputing.com
And then this is the output:
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- g report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception:
Please submit a bug repor at 0x668432c5 -- at 0x668432c5 -- g report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- OLATION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- OLATION ve/d/buildbotin expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
trampoline_alloc at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined]
jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
in expression starting at C:\Users\Leprein expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
ety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATIONtrampoline_alloc atrampoline_alloc at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined]
jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
ety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- OLATION ve/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined]
jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
trampoline_alloc at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined]
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threadjl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
#launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupjl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
#launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 [inlined]
launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006536812A)
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlined]
lined]
launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006536812A)
e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
]
lined]
launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006536812A)
e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupan#launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 [inlined]
launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006536812A)
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlinmacro expansion at C:\Users\Lepre\.juliamacro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlined]
#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:83
mapreducedim!##kw at C:\Users\Lepre\.julmapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81
#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:83
mapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81
unknown function (ip: 000000006535BC85)
apro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highljl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18
#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highljl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
el.jl:83
unknown function (ip: 000000006535BC85)
unknown function (ip: 000000006535BC85)
re\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreducejl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329
#launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 [inlined]
launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006536812A)
e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:unknown function (ip: 000000006537CC8C)
e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006535BC85)
#_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62
#_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62
5
[i#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:83
mapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81
unknown function (ip: 000000006535BC85)
#_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62
#_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62
81
ned]
#mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
mapreduce at C:\Users\Lepre\.juliapro\Jumapreduce at C:\Users\Lepre\.juliapro\Ju_mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:34 _mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:34 [inlined]
#mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
mapreduce at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
_sum at .\reducedim.jl:657 [inlined]
_sum at .\reducedim.jl:656 [inlined]
#sum#583 at .\reducedim.jl:652 [inlined]
sum at .\reducedim.jl:652 [inlined]
_mean at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:160 [inlined]
#mean#4 at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined]
mean##kw at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined]
#adjoint#667 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\array.jl:278 [inlined]
adjoint at .\none:0 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
loss at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined]
adjoint at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:179 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
#15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
d]
lined]
#mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
_sum at .\reducedim.jl:679 [inlined]
_sum at .\reducedim.jl:678 [inlined]
#sum#583 at .\reducedim.jl:652 [inlined]
sum##kw at .\reducedim.jl:652 [inlined]
#accum_sum#1083 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:42 [inlined]
accum_sum##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:42 [inlined]
unbroadcast at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:53
unbroadcast at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:53
[gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:88 [inlined]
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined]
#train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81
train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inlined]
#373#threadsfor_fun at .\threadingconstructs.jl:61
#373#threadsfor_fun at .\threadingconstrmacro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlined]_sum at .\reducedim.jl:657 [inlined]
_sum at .\reducedim.jl:656 [inlined]
#sum#583 at .\reducedim.jl:652 [inlined]
sum at .\reducedim.jl:652 [inlined]
_mean at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:160 [inlined]
#mean#4 at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined]
mean##kw at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined]
#adjoint#667 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\array.jl:278 [inlined]
adjoint at .\none:0 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
loss at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined]
adjoint at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:179 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
#15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:172
map at .\tuple.jl:158 [inlined]
#1090 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:74 [inlined]
#2457#back at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined]
Dense at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\layers\basic.jl:122 [inlined]
Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
Dense at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\layers\basic.jl:133 [inlined]
Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.unknown function (ip: 000000003568D1C3)
_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined]
ned]
line#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevegradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53
gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53
l.jl:83
ed]
#mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jPullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
#175 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:182
erface2.jl:0
28 [inlimapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81
unknown function (ip: 000000006535BC85)
#_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62
81
ed]
mapreduce at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
_sum at .\reducedim.jl:657 [inlined]
_sum at .\reducedim.jl:656 [inlined]
#sum#583 at .\reducedim.jl:652 [inlined]
sum at .\reducedim.jl:652 [inlined]
_mean at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:160 [inlined]
#mean#4 at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined]
mean##kw at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined]
#adjoint#667 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\array.jl:278 [inlined]
adjoint at .\none:0 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
loss at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined]
adjoint at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:179 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined]
#15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined]
_pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:172
gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:88 [inlined]
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined]
#train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81
train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inlined]
#373#threadsfor_fun at .\threadingconstructs.jl:61
#373#threadsfor_fun at .\threadingconstructs.jl:28
_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inliunknown function (ip: 000000006537CC8C)
unknown function (ip: 000000006537CC8C)
cts.jl:28
_v1.4.2-1\dev\SIREN\test\simple_double_t#347#back at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined]
#15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined]
Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0
#50 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:177
unknown function (ip: 000000006536C31A)
v1.4.2-1\packages\Zygote\1GXzF\src\compiler\instart_task at /cygdrive/d/buildbot/worker/package_win64/build/src\task.c:687
Allocations: 216670028 (Pool: 216622719; Big: 47309); GC: 146
Allocations: 216670028 (Pool: 216622719; Big: 47309unknown function (ip: 000000003568D1C3)
unknown function (ip: 000000003568D1C3)
gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:54
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:88 [inlined]
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined]
#train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81
train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79
81
inlined]
ed]
d]
d]
launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
unknown function (ip: 000000006536812A)
unknown function (ip: 000000006536812A)
e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55
]
d]
#mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined]
_sum at .\reducedim.jl:679 [inlined]
_sum at .\reducedim.jl:678 [inlined]
#sum#583 at .\reducedim.jl:652 [inlined]
sum##kw at .\reducedim.jl:652 [inlined]
#accum_sum#1083 at C:\Users\Lepre\.juliamacro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\sstart_task at /cygdrive/d/buildbot/worker/package_win64/build/src\task.c:687
Allocations: 216670028 (Pool: 216622719; Big: 47309); GC: 146
rc\task.c:687
s\YFdj7\src\mapreduce.jl:199 [inlined]
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined]
#train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81
train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79
macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inlined]