New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validator library #124

Closed
troeger opened this Issue May 26, 2016 · 7 comments

Comments

1 participant
@troeger
Owner

troeger commented May 26, 2016

In reaction to #68, we are planning a validator library that becomes part of the executor installation. This issue is intended to collect all feature request for such a library.

@troeger troeger added the new feature label May 26, 2016

@troeger

This comment has been minimized.

Owner

troeger commented May 26, 2016

Feature idea: Scan for binaries in the student archive and remove them. This would close #68.

@troeger

This comment has been minimized.

Owner

troeger commented May 26, 2016

Feature idea: Scan for keywords in the student code files. Would close #6.

@troeger

This comment has been minimized.

Owner

troeger commented May 26, 2016

Feature idea: Compile code. This would mean that we can remove the compile test feature completely. Compilation becomes a step of validation, which would (as side effect) solve the problem described in #81. Practice has shown that compile-only cases are rare enough to reason that.

@troeger

This comment has been minimized.

Owner

troeger commented Jun 1, 2016

Feature idea: Offer meta-data about submission to the validation code. This would close #100 and #113.

@troeger troeger modified the milestones: Next major release, Executor Re-Design Oct 26, 2016

@troeger

This comment has been minimized.

Owner

troeger commented Nov 11, 2016

Feature idea: Report a multi-line text result from the student submission validation. Discussed to a larger extend as part of #148.

@troeger troeger modified the milestones: Executor Re-Design, Next release Oct 18, 2017

@troeger

This comment has been minimized.

Owner

troeger commented Oct 26, 2017

Some more specific ideas from our current course run:

  • Call 'make' and check if a binary with the correct name is produced.
  • Run the student binary with one / two / three command-line arguments and check the output.
  • Run the student binary with predefined input (through stdin) and check the output.
  • Check the output either for exact match or sub-string match.
  • Easy sending of a default error text if the output does not match the expectation. Make it configurable if the expected output for the given input is shown in the error report.
@troeger

This comment has been minimized.

Owner

troeger commented Nov 30, 2017

Done with v0.7b1. The only remaining issue is #6, which will be added later. Check the Job() class and the new examples folder.

@troeger troeger closed this Nov 30, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment