Run on all devices even if other device types fail #186
Conversation
ryanlai2
changed the title
… bind / eval can be returned
Tools/WinMLRunner/src/Run.cpp
Outdated
| { | ||
| for (const auto& inputBindingType : inputBindingTypes) | ||
| HRESULT evaluateModelGivenDeviceTypeResult = | ||
| EvaluateModelGivenDeviceType(model, deviceType, inputBindingTypes, inputDataTypes, |
There was a problem hiding this comment.
Can we rename this to EvaluateModelWithDeviceType?
There was a problem hiding this comment.
Sure sure! I can rename this.
Tools/WinMLRunner/src/Run.cpp
Outdated
| } | ||
| lastEvaluateModelResult = evaluateModelGivenDeviceTypeResult; | ||
| std::cout << "Run failed for DeviceType: " << TypeHelper::Stringify(deviceType) << std::endl; | ||
| ; |
|
It would be great if we can add test cases for this, but I understand that this might be hard to repro (since you should purposely fail in one of the devices, which is not what where should be at all) |
Right, it would be great to add test cases for this. However, it would be hard to find a model that purposely fails on one of the devices but passes on others. In the future, I believe using mocking framework would be best to test this. |
This PR pulls out the code in EvaluateModels that loops through all inputBindingTypes inputDataTypes and deviceCreationLocations into a separate function.
EvaluateModels loops through all deviceTypes and calls EvaluateModelGivenDeviceType