-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add benchmarks for extran and routing tests #133
Conversation
@bemcdonnell appveyor is running now with 21/22 tests passing. |
@michaeltryby, it would be really helpful if you could have the testing framework print a few lines of the diff so we can get a quick preview of what is going on with it. How complicated would it be if the tool would print ~10 lines of a failed RPT diff? |
@michaeltryby, maybe make a small injection here: def report_compare(path_test, path_ref, (comp_args)):
'''
Compares results in two report files ignoring contents of header and footer.
'''
with open(path_test ,'r') as ftest, open(path_ref, 'r') as fref:
for (test_line, ref_line) in it.izip(hdf.parse(ftest, 4, 4)[1],
hdf.parse(fref, 4, 4)[1]):
if test_line != ref_line:
print('benchmark-> {}\ntest ->{}'.format(ref_line.replace('\n',''), test_line.replace('\n','')))
return False
return True |
@bemcdonnell couple of thought ...
|
On Appveyor it looks like you are able to associate and retrieve artifacts with a CI build. The support for savings test artifacts isn't as good on Travis. They only support S3. |
@bemcdonnell I propose we accept this PR as is and create an issue to configure appveyor to retrieve failed test artifacts. What do you think? |
@bemcdonnell I can also create an issue to update the readme for running |
@michaeltryby, i wonder why we have one failure... |
@bemcdonnell Let's find out! |
This PR addresses Issue #132. The scripts
gen-config.cmd
andrun-nrtest.cmd
have also been added to thetools
folder.