-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Complete a test with report on specific issues #5
Comments
Yes, i agree, this should be high priority on our roadmap. And good point about sometimes having known issues. The way I've been thinking about it is that there are a few different layers to Orca_test:
What you're suggesting would be a new mode in the middle layer, so that we run a diagnosis report for a spec instead of just asserting it. Hopefully this will be easy to implement through some simple changes to the existing middle-layer functions. This is a bit speculative, but potentially we could have an Orca_test "mode" setting, and a user could choose whether to run a simulation in "strict" mode or to just log the Orca_test warnings. Or, a user could have multiple sets of specs with different strictnesses. This would be a stretch to implement right now, but it's something to think about as we make architecture decisions. |
Yes, I think it can be a mid layer. It would call the same lower level calls, but each in a |
I got started experimenting on my fork. DIFF Thoughts? Comments? Catching each error, looses a lot of the traceback info. P.S. sorry for the white space diff noise, pycharm atom fixed it. |
Excellent! I like this as a general behavior, that the functions like Some thoughts:
|
If we are logging log.exception(e). Which also gets the traceback. I don't know how to give the details to the caller. |
Right now any assertion error breaks the checking routine. I think it is preferable to have a complete test with a diagnosis report on all the errors. So we can focus on fixing issues all together before performing another test.
Also, in data development stage, typically we have some known issues in our data. For minor issues, we may choose to keep it that way temporarily but still get the alerts for other problems. A full test will help in that situation.
Thanks.
The text was updated successfully, but these errors were encountered: