-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
unit tests sometimes have sporadic failures #7
Comments
Hi, Great! Build issues are very welcome. I'll fix soonish when I have spare time. My *BSD buildbots are currently offline, so might be that I haven't noticed some issue on BSDish platforms. There have been many issues with unit tests on OSX due to minor differences in arguments etc. Might make sense to use the version of owl used for building also for sorting and echoing in tests. |
So the behavior is intended, because radamsa is supposed to pad the input with a low probability with random data, if the input is very short. This is done to improve coverage of test with tiny fixed inputs. The only issues is with OSX sort getting confused by non-textual data. This does not matter, because the tests are probabilistic and are expected to fail on occasion, in which case they are tried again many times before tests/run considers them to fail. Sort stderr is now piped to /dev/null, so it doesn't get in the way. Does the build work otherwise on OSX? |
Oh, I did not realize that the output was not representative of a failure. Sorry for the false alarm on those tests. I've modified ab.sh as follows to determine why it is sometimes failing:
Attached is one the output from one such test run which failed.
When building Radamsa the unit tests automatically run. As unit tests may sporadically fail, could the unit tests be sectioned into their own make target (e.g. "make check") so that they do not run automatically? Otherwise, one may build Radamsa, see a failure, and wonder if the sporadic failure indicates an issue with Radamsa on one's platform. Sure, I can understand that Radamsa, being probabilistic, may not always fuzz as expected. However, is it possible to modify the unit tests which may fail with some probability so that the tests will be more deterministic?
Seems to work so far. I've not tried the TCP stuff yet, which looks interesting. |
Makes sense. There is now a separate test build target, which runs the tests if necessary. I also added a thanks-section to readme.md. |
It has been observed at least on OSX that some of the unit tests sometimes have sporadic failures. Following are some example failures as output by tests/run:
Some of the unit tests when they fail do not emit output to help diagnose the failure. Here is an example invoking tests/ab.sh directly:
After 12 attempts a failure was observed, but the reason for the failure is not emitted.
Likely it is the expectation that the unit test results be consistent. If the current revision in git is under development and the sporadic failure is expected or some cleanup is still underway, kindly ignore. I was unable to determine if release v0.4's unit tests encountered sporadic failures as issue #5 affects the v0.4 release.
As a comparison, release v0.3 had consistently passing unit tests.
(As a side note, at least on OSX the built-in echo command in sh does not support the -n option. This is the reason that "-n" is printed before all of the tests in tests/run . Consider reworking when echo is used in that script so that the -n option is unnecessary, if relevant).
The text was updated successfully, but these errors were encountered: