Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Target 4294967295 is out of bounds - failure during Text Classification CPU Training #2368

Closed
uuxxaa opened this issue Nov 12, 2022 · 2 comments

Comments

@uuxxaa
Copy link

uuxxaa commented Nov 12, 2022

System Information (please complete the following information):

  • Model Builder Version (available in Manage Extensions dialog): 16.14.0.2255902
  • Visual Studio Version VS 2022 17.4.0

Describe the bug

  • On which step of the process did you run into an issue: Training
  • Clear description of the problem: Training threw an exception

Target 4294967295 is out of bounds.
Exception raised from nll_loss_out_frame at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\LossNLL.cpp:230 (most recent call first):
00007FF829E5A4C200007FF829E5A460 c10.dll!c10::Error::Error [ @ ]
00007FF829E33ED500007FF829E33E60 c10.dll!c10::IndexError::IndexError [ @ ]
00007FFFDD491FB400007FFFDD48CE60 torch_cpu.dll!at::native::multi_margin_loss_cpu_out [ @ ]
00007FFFDD4958F300007FFFDD48CE60 torch_cpu.dll!at::native::multi_margin_loss_cpu_out [ @ ]
00007FFFDD49773400007FFFDD4976D0 torch_cpu.dll!at::native::structured_nll_loss_forward_out_cpu::impl [ @ ]
00007FFFDDCDC3DE00007FFFDDCCB710 torch_cpu.dll!at::cpu::zero_ [ @ ]
00007FFFDDCA19AE00007FFFDDC76730 torch_cpu.dll!at::cpu::bucketize_outf [ @ ]
00007FFFDD8DB89000007FFFDD8874F0 torch_cpu.dll!at::_ops::zeros_out::redispatch [ @ ]
00007FFFDDA6308300007FFFDDA62FE0 torch_cpu.dll!at::_ops::nll_loss_forward::redispatch [ @ ]
00007FFFDE7FAFA300007FFFDE67A050 torch_cpu.dll!torch::autograd::GraphRoot::apply [ @ ]
00007FFFDE7C60F200007FFFDE67A050 torch_cpu.dll!torch::autograd::GraphRoot::apply [ @ ]
00007FFFDDA1698C00007FFFDDA16800 torch_cpu.dll!at::_ops::nll_loss_forward::call [ @ ]
00007FFFDD497F0F00007FFFDD497E90 torch_cpu.dll!at::native::nll_loss [ @ ]
00007FFFDDE5B6B200007FFFDDE57680 torch_cpu.dll!at::compositeimplicitautograd::where [ @ ]
00007FFFDDE3AA5D00007FFFDDE0FD00 torch_cpu.dll!at::compositeimplicitautograd::broadcast_to [ @ ]
00007FFFDDAE0C6F00007FFFDDAE0AE0 torch_cpu.dll!at::_ops::nll_loss::call [ @ ]
00007FFFDD49888F00007FFFDD497F80 torch_cpu.dll!at::native::nll_loss_nd [ @ ]
00007FFFDDE5B6E200007FFFDDE57680 torch_cpu.dll!at::compositeimplicitautograd::where [ @ ]
00007FFFDDE3AACD00007FFFDDE0FD00 torch_cpu.dll!at::compositeimplicitautograd::broadcast_to [ @ ]
00007FFFDD92142F00007FFFDD9212A0 torch_cpu.dll!at::_ops::nll_loss_nd::call [ @ ]
00007FFFDD49653F00007FFFDD496250 torch_cpu.dll!at::native::cross_entropy_loss [ @ ]
00007FFFDDE5968100007FFFDDE57680 torch_cpu.dll!at::compositeimplicitautograd::where [ @ ]
00007FFFDDE3AB5200007FFFDDE0FD00 torch_cpu.dll!at::compositeimplicitautograd::broadcast_to [ @ ]
00007FFFDDAC6B2300007FFFDDAC6980 torch_cpu.dll!at::_ops::cross_entropy_loss::call [ @ ]
00007FFFDD1BFC7100007FFFDD1BFC40 torch_cpu.dll!at::cross_entropy_loss [ @ ]
00007FF807BB5E0500007FF807BB5C60 LibTorchSharp.DLL!THSNN_cross_entropy [ @ ]
00007FF878AE5734 ! [ @ ]


at TorchSharp.torch.CheckForErrors()
at TorchSharp.Modules.CrossEntropyLoss.forward(Tensor input, Tensor target)
at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer2.TrainerBase.TrainStep(DataViewRowCursor cursor, ValueGetter1 sentence1Getter, ValueGetter1 sentence2Getter, ValueGetter1 labelGetter, List1& inputTensors, List1& targets)
at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer2.TrainerBase.Train(IDataView input) at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer2.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings) at System.Threading.Tasks.Task1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.ML.AutoML.AutoMLExperiment.d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLService.TextClassificationExperiment.d__13.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/TextClassificationExperiment.cs:line 131
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__21.MoveNext() in /
/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 141

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

@uuxxaa uuxxaa changed the title Target 4294967295 is out of bounds - failure during Text Sentiment Training Target 4294967295 is out of bounds - failure during Text Classification Training Nov 12, 2022
@uuxxaa uuxxaa changed the title Target 4294967295 is out of bounds - failure during Text Classification Training Target 4294967295 is out of bounds - failure during Text Classification CPU Training Nov 12, 2022
@beccamc
Copy link
Contributor

beccamc commented Nov 14, 2022

@uuxxaa Can you give more details about your dataset? What is the size?

@beccamc
Copy link
Contributor

beccamc commented Nov 30, 2022

Duplicate. See 2369 for tracking.

@beccamc beccamc closed this as completed Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants