Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Value goes invalid when using TestMinibatch #3859

Open
viraptor opened this issue Sep 8, 2021 · 0 comments
Open

Value goes invalid when using TestMinibatch #3859

viraptor opened this issue Sep 8, 2021 · 0 comments

Comments

@viraptor
Copy link

viraptor commented Sep 8, 2021

I'm running an issue when calling trainer.PreviousMinibatchLossAverage() after a TestMinibatch run:

var minibatchDataTest = minibatchTest.GetNextMinibatch(10, device);
var mbData = new UnorderedMapVariableMinibatchData();
mbData.Add(features, minibatchDataTest[featureTestStreamInfo]);
mbData.Add(labels, minibatchDataTest[labelTestStreamInfo]);
var testRes = trainer.TestMinibatch(mbData, device);
Console.WriteLine($"test loss {trainer.PreviousMinibatchLossAverage()}");  // this raises exception

The printed stack is:

[CALL STACK]
    > CNTK::Internal::  UseSparseGradientAggregationInDataParallelSGD
    - CNTK::Value::  Create
    - CNTK::Internal::  UseSparseGradientAggregationInDataParallelSGD
    - CNTK::TrainingParameterSchedule<unsigned __int64>::  Transform
    - CNTK::Trainer::  PreviousMinibatchLossAverage
    - CSharp_CNTK_Trainer_PreviousMinibatchLossAverage
    - 00007FFDE9FB695B (SymFromAddr() error: The specified module could not be found.)

I'm not even sure which value it is, since both minibatchDataTest[featureTestStreamInfo] and minibatchDataTest[labelTestStreamInfo] are valid both before and after the TestMinibatch call. (as reported by data.IsValid)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant