Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Testing API doesn't yield useful results #4302
The current testing API doesn’t provide useful results as a traceback if a given test fails. It becomes a strenuous and time consuming job to follow the traceback searching for errors.
Eg. For the check_results(...) method, http://paste.ubuntu.com/24269550/ is the output that I received when I had a minor difference of line=5 instead of line=6 in the given code snippet :
The given output below appear to be strange as both the Result objects contain the same number of characters (632) due to which testing transforms to a challenging task. Hit and trial methods or the usage of a debugger are the only possible ways to overcome these errors.
referenced this issue
Jun 2, 2017
The design at https://yash-nisar.github.io/blog/testing-api-coala/ doesnt indicate how you will improve on the pytest output (assuming self.maxDiff is disabled), which is mostly ugly and unusable because of the depth of the stack, and volume of information with self.maxDiff is disabled, and I dont see that being addressed in the proposal.
I think addressing the output stack is quite some problem that is beyond gsoc. The pytest guys didn't do ugly outputs on purpose, I assume they already tried to make it as readable as possible without loosing information. You are right, there's still much space for improvement, but that's another topic.
We are concentrating on results from bears. This is bad:
Ideally all objects that use
Yes, that is bad, but that is without
I would very much like to see the output of the same assertion with
I dont see anything in the design to show how we will get the errors to look like
The design indicates this will happen without any change in the bear tests, so it will happen here in the coala test helpers.
One way to do that is to iterate over each attribute and do an assert. Is that the plan?
Another consideration (which I havent looked into at all), is whether a pytest extension might also provide better test error information. There is a very large ecosystem of pytest extensions.
See https://yash-nisar.github.io/blog/testing-api-coala/ on how to achieve this.