Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CNTK C# Crash when layer is deeper #3861

Closed
peterkim333 opened this issue Dec 21, 2021 · 0 comments
Closed

CNTK C# Crash when layer is deeper #3861

peterkim333 opened this issue Dec 21, 2021 · 0 comments

Comments

@peterkim333
Copy link

peterkim333 commented Dec 21, 2021

Hi, I am a developer who use CNTK C# package with RTX 3090 GPU.

our team have rebuilt the CNTK C# for higher CUDA and cuDNN version.
usually, it works for training and so on.

However, our program dies Irregularly during neural network learning. Usually Deep Learning is finished well. But sometimes it dies.

Error is below.

:System.AccessViolationException

Location: CNTK.CNTKLibPINVOKE.Variable__Name(System.Runtime.InteropServices.HandleRef)
Location: CNTK.Variable._Name()
Location: DeepLearningCore.SegmentationNetwork.GetParameters()
Location: DeepLearningCore.Segmentation.TrainNetwork()
Location: DeepLearningCore.Segmentation.Run()
Location: System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
Location: System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
Location: System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
Location: System.Threading.ThreadHelper.ThreadStart()

I have no idea. we try to trace the error and figure out that the error occurs when try to get the specific parameter and its name, it is failed.
It is not certain, about 500 or more layers make this error.
Anybody who has same problem?

@peterkim333 peterkim333 changed the title CNTK C# Crush when layer is deeper CNTK C# Crash when layer is deeper Dec 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant