Version 0.99.6 training process is normal, 0.100.3 training process error

[lindadamama]

[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\third_party\nvfuser\csrc\graph_fuser.cpp:108] Warning: operator () profile_node %987 : int[] = prim::profile_ivalue(%dims.39) does not have profile information (function operator ())

[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\third_party\nvfuser\csrc\manager.cpp:340] Warning: FALLBACK path has been taken inside: torch::jit::fuser::cuda::runCudaFusionGroup. This is an indication that codegen Failed for some reason.
To debug try disable codegen fallback path via setting the env variable export PYTORCH_NVFUSER_DISABLE=fallback
(function runCudaFusionGroup)

[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\third_party\nvfuser\csrc\manager.cpp:340] Warning: FALLBACK path has been taken inside: torch::jit::fuser::cuda::runCudaFusionGroup. This is an indication that codegen Failed for some reason.

To debug try disable codegen fallback path via setting the env variable export PYTORCH_NVFUSER_DISABLE=fallback
(function runCudaFusionGroup)

2023/7/3 0:23:21
ehope: 0/80000 :
loss:0.048035335
acc:0.9519647
model directory：
【D:\Inno.LabelAssistantDatas\电路板\电路板\AIModel\model.pt】

D:\C#\Inno.ImgProcess.UI\Inno.AIFrameWork.TrainTest\bin\Debug\net6.0-windows\Inno.AIFrameWork.TrainTest.exe (process 22168) exited with code -1073741819.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

[GeorgeS2019]

@lindadamama
Because this has Chinese characters, it is important you share in TEXT not in image <=====

@ChengYen-Tang Please advice this person so this will not lead to abuse of our kindness

[GeorgeS2019]

@lindadamama
Please take some time to format

[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\third_party\nvfuser\csrc\graph_fuser.cpp:108] Warning: operator () profile_node %987 : int[] = prim::profile_ivalue(%dims.39) does not have profile information (function operator ()) [W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\third_party\nvfuser\csrc\manager.cpp:340] Warning: FALLBACK path has been taken inside: torch::jit::fuser::cuda::runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (function runCudaFusionGroup) [W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\third_party\nvfuser\csrc\manager.cpp:340] Warning: FALLBACK path has been taken inside: torch::jit::fuser::cuda::runCudaFusionGroup. This is an indication that codegen Failed for some reason. To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback` (function runCudaFusionGroup) 2023/7/3 0:23:21 0/80000次训练: loss(训练集错误率):0.048035335 acc(训练集准确率):0.9519647 模型的保存路径为： 【D:\Inno.LabelAssistantDatas\电路板\电路板\AIModel\model.pt】
D:\C#\Inno.ImgProcess.UI\Inno.AIFrameWork.TrainTest\bin\Debug\net6.0-windows\Inno.AIFrameWork.TrainTest.exe (process 22168) exited with code -1073741819.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

[GeorgeS2019]

@lindadamama

Your history

• 1 #1017

[GeorgeS2019]

To all users here who are sharing in Chinese characters <== Please read this <====

the people who normally reply DO NOT UNDERSTAND Chinese

Please be KIND <=====

Please do not let others have wrong image of Chinese

[GeorgeS2019]

@Oceania2018

I am including here as @lindadamama is a regular user of https://github.com/SciSharp

Please advice @lindadamama if the people here, often from US,, who would reply to requests involving Chinese Characters, this is really OUT of Kindness

[lindadamama]

I'm sorry,I have modified it

[GeorgeS2019]

@lindadamama

Let us all work together with Kindness to keep this community keep on active

[lindadamama]

There have been new errors again

[lindadamama]

Error message exited with code -1073741819. It should be that Torch2.0.1 occupies more memory than Torch1.13, causing a memory overflow,
Another error is that some Tensors in dataloader were released due to unknown reasons during the training process of the training dataset,but there was no similar situation at 0.99.6
The main problem currently is the release of ResultMasks
Var loss=criterion.call (result, item. ResultMasks);
@GeorgeS2019
@NiklasGustafsson
code of lost tensor ：

 foreach (var item in dataloader)
                {
                    using (var d = torch.NewDisposeScope())
                    {
                        var output = this.module.call(item.Images);
                        if (output is Tensor result)
                        {
                            var loss = criterion.call(result, item.ResultMasks);
                            optimizer.zero_grad();
                            loss.backward();
                            var opt = optimizer.step();
                            var loss_val = loss.to(CPU).item<float>();
                            lossArry.Add(loss_val);
                            result?.Dispose();
                        }
                    };
                }

Is there a problem with the code written above? Or does it need to be changed to the following notation?

   using (var d = torch.NewDisposeScope())
                {
                    using var dataset = new SegDataset(datadir, labelDir, numClass);
                    using var dataloader = new torch.utils.data.DataLoader<DataSetItem, BatchItem>(dataset, batchSize, CollateFn.Collate, doShuffle, device, num_worker: 4);
                    foreach (var item in dataloader)
                    {
                        var output = this.module.call(item.Images);
                        if (output is Tensor result)
                        {
                            var loss = criterion.call(result, item.ResultMasks);
                            optimizer.zero_grad();
                            loss.backward();
                            var opt = optimizer.step();
                            var loss_val = loss.to(CPU).item<float>();
                            lossArry.Add(loss_val);
                            result?.Dispose();
                        }

                    }
                };

[GeorgeS2019]

@NiklasGustafsson

Is the pilar of this project.

I hope when he has time soon to get back to you.

I am independent but someone who believe strongly we need this alternative to pytorch in DeepAI for .Net

[ChengYen-Tang]

@lindadamama ,
Can you prepare a simple example and dataset so that everyone can reproduce the problem. You currently provide too little information, it is difficult to analyze the problem.

[lindadamama]

Sorry, there was an issue with the code just uploaded. It should look like this
@GeorgeS2019

void checkDiretory(string path) 
{
    if (Directory.Exists(path))
    {
        Directory.Delete(path, true);
    }
    Directory.CreateDirectory(path);
}

[ChengYen-Tang]

An interesting question!! Without using breakpoints, it crashes in 3 ephos, but with a large number of breakpoints, it can run at least 15 ephos.

I suspect it has something to do with GC.

[NiklasGustafsson]

@lindadamama -- I was on vacation last week.

@ChengYen-Tang referred to code crashing, but I don't see a link anywhere -- there is no way for me to help diagnose this without a repro.

[NiklasGustafsson]

Please advice @lindadamama if the people here, often from US,, who would reply to requests involving Chinese Characters, this is really OUT of Kindness

I don't mind it if posts include pieces that I cannot understand, but anytime there is important information in Chinese (or any other number of languages that I don't know), I will be either unable to assist, or delayed in doing so. I'm not sure where @GeorgeS2019's concerns about kindness come from, nothing in this post seems unkind to me, but the language issue is definitely a matter of efficiency -- I can only realistically help if the relevant information is in English. Same thing goes for comments in code -- if the comments are not in English, and you upload a repro, it may slow down the process of getting help.

I'm sure there are other community members who can pitch in and help with other languages, but I cannot.

Thanks,

Niklas