Testing elements on specific inputs

[SethPoulsen]

Inspired by this and similar past issues: #7527 (comment)

Do we want to have a way to test specific inputs for our element testing?

Its not clear what the format would look like, since different elements take inputs in different formats. I supposed we could just have a system where you manually write the JSON that should be serialized to the server for the specific input you want to test.

[eliotwrobson]

Having a system to manually write JSON is probably the most robust. However, elements like the pl-big-o-input, pl-symbolic-input, and pl-units-input all use an architecture that allows testing the grading algorithm using pytest, which covers a similar use case to this the proposed discussion here. It allows for manually inputting a suite of specific inputs to the grading algorithm, and there should be an easy way to extend this to fuzz testing. I think an architecture like this could handle the use case detailed in the linked comment with substantially lower developer overhead than extending the existing PL test suite to handle JSON files.

[SethPoulsen]

It would be nice if it integrated somehow with the test function that the elements defines though.

[nwalters512]

If you want to use test(), just call it! I don't think we need anything to "integrate", though there might be some useful abstractions that fall out of this work. We can worry about that later.

[eliotwrobson]

Per the discussion above, it's doable (but a bit more effort) to test a full parse + grade flow purely in pytest. However, even without that additional effort, the current grade function for pl-order-blocks isn't fully covered by the existing pytest tests (just some of the grading algorithms). This is what I meant about changing the existing architecture. To have better test coverage, there should be better abstraction between the grading code itself and the grade function. In the other elements I mentioned, those grading algorithms are abstracted heavily from the structure of the element itself, making them very easy to test.

If you want to use test(), just call it! I don't think we need anything to "integrate", though there might be some useful abstractions that fall out of this work. We can worry about that later.

Unfortunately, anything within an element controller is not easily importable from another Python file without some additional effort because of the naming convention. However per the rest of my comment, I don't think this is really super necessary.

[nwalters512]

I'm also in favor of abstractions where appropriate, but there's nothing stopping us from just changing the Python element controller file names to use underscores instead of dashes if that's a roadblock.

[eliotwrobson]

I'm also in favor of abstractions where appropriate, but there's nothing stopping us from just changing the Python element controller file names to use underscores instead of dashes if that's a roadblock.

I think that's a fine course of action, but if this is done it should accompany some work on a larger testing framework that prevents code duplication for all tests of this type across elements (at the very least, come up with something that's extensible enough that it could be applied to other elements later, since this will effectively be adding a third testing architecture to PL). That's why I mentioned the existing level of abstraction used for pytest in the other elements, it requires pretty minimal effort to set up with the current architecture and catches most of the common issues.