Add support for unittest evaluation using snekbox

[jb3]

This is a long term goal!

We should have the ability to define a set of unittests and run user code on snekbox to automatically calculate the results of this.

This could be used for code jam qualifiers and other tasks which require a completed code element.

If users fail the unittests we will return this feedback to them in the browser so they cannot actually submit until their code works.

[swfarnsworth]

Would this be as simple as concatenating the unit tests onto the end of the user's submission and evaluating it in snekbox as one script?

[MarkKoz]

From this end, possibly yes. I have concerns about namespace conflicts but it should be possible to write a test suite in a manner that minimally pollutes the namespace. From the snekbox end, the endpoint may need to be adjusted to allow disabling/changing time and memory limits.

[swfarnsworth]

@MarkKoz we could start every name specific to the test suite with something like __PyDisTest_ (I believe PyCharm does something like this for its debugger), while also checking if the set of names defined in the user's submission is a subset superset of the ones we expect to run tests with. I don't know off the top of my head what changing those configurations for snekbox would entail but I'll of course wait to see what Akarys says before doing any work towards this.

[jb3]

We have migrated Python Discord Form issue tracking to Notion.

This specific ticket can be found here.

This discussion will be locked for now, all future conversation should occur in the Notion documents.

If you are an external collaborator who is interested in contributing to forms please shoot me a DM on Discord, joe#6000.

This GitHub discussion has been transferred onto the issue.