Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sim_if: continue after compilation failure if '--keep-compiling' is True #742

Merged
merged 3 commits into from
Apr 23, 2023

Conversation

umarcor
Copy link
Member

@umarcor umarcor commented Sep 21, 2021

Currently, option --keep-compiling will raise an exception/error after all files are compiled. That is, it allows to continue while compilation failures are produced, but it does not allow to run the tests (those whose sources were successfully compiled, at least).
This PR changes it, so that execution continues after compilation when -k is used.

@umarcor umarcor force-pushed the fix-continue-on-error branch 2 times, most recently from 38aaedd to c7cbac1 Compare September 27, 2021 22:59
@eine eine added this to the v4.6.0 milestone Sep 28, 2021
@eine eine modified the milestones: v4.6.0, v5.0.0 Oct 21, 2021
@eine eine modified the milestones: v5.0.0, v4.7.0 Oct 25, 2021
@LarsAsplund
Copy link
Collaborator

I haven't reviewed this yet but a gitter discussion reminded me that the compile error must never be forgotten. It should be seen in the test summary and the return code should not be zero

@umarcor
Copy link
Member Author

umarcor commented Dec 1, 2021

@LarsAsplund the main purpose of this PR is to remove https://github.com/VHDL/Compliance-Tests/blob/master/run.py#L30-L69 by using --keep-compiling in order to run the tests which could be compiled. If we force the return code to be non-zero, we need some additional feature to cover the capability required in Compliance-Tests.

@LarsAsplund
Copy link
Collaborator

I think that use case is perfectly valid but do you really want the default behavior for a failing compliance test to be return code = 0?

@umarcor
Copy link
Member Author

umarcor commented Dec 1, 2021

I think that use case is perfectly valid but do you really want the default behavior for a failing compliance test to be return code = 0?

Only in the cases when the user has explicitly set --keep-compiling. To my understanding, --keep-compiling invalidates the meaning of the exit code. The user is forced to read the log (in the terminal or the XML report) in order to know what happened. It is known that the exit code is a failure, because using --keep-compiling would not be required otherwise.

I would expect the exit code to be meaningful if we had some xfail behaviour, such as the one in pytest. So that the user would say "I want to keep compiling and I expect it to fail (exit code 0) but let me know if nothing fails (exit code 1)". That is, an inversion of the logic/meaning of the exit code when --keep-compiling is used.

@umarcor umarcor force-pushed the fix-continue-on-error branch 2 times, most recently from 33eb0f1 to 7b74a29 Compare January 9, 2022 11:38
@eine eine modified the milestones: v5.0.0, v5x Apr 19, 2023
@umarcor umarcor force-pushed the fix-continue-on-error branch 4 times, most recently from ddf45a7 to cc8c70d Compare April 23, 2023 19:25
@eine eine modified the milestones: v5.1.0, v5.0.0 Apr 23, 2023
@eine eine merged commit 8654f1d into VUnit:master Apr 23, 2023
13 checks passed
@umarcor umarcor deleted the fix-continue-on-error branch April 23, 2023 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants