Feature: hidden tests

[jmr-ua-pt]

Codecheck programming exercises currently expose all tests to the student.
It would be nice If Codecheck could have some hidden tests.
These would simply report a Pass/Fail result, without exposing parameters or results.

Hidden tests should prevent students from tailoring their solutions just to pass specific tests and not solve the general problem.

[cayhorstmann]

Here are some implementation thoughts.

A test could be marked as //HIDDEN (or maybe the existing //HIDE)?

Going through the four kinds of test, three of them seem simple:

• Don't print the unit test
• Don't print the table of method arguments/return values
• Don't print the table of substitution values and program outputs

When running a program, it could be more complex. Do we suppress the //ARGS? The `//OUT' files (text or images)? For sure the inputs and outputs. This takes some planning.

[cayhorstmann]

Start looking at Report.java. When a source file is submitted for reporting, see if it is meant to be hidden from the student. You can find that out from Annotation.java. But you need to teach Annotation.java about the HIDDEN pseudocomment.

[jmr-ua-pt]

Thanks for considering this request.

The multiple kinds of test in Codecheck are quite convenient, but do introduce some complications, I guess.¹

My gut feeling is that marking tests with an additional //HIDE could be confusing, given the use of that keyword for starting hidden code blocks. An additional //HIDDEN should be somewhat better, but how would that work for unit tests? One per file?

How do you feel about having alternate forms for each of the current test keywords, such as:
//INH, //ARGSH, //OUTH, //CALLH, //SUBH?
And maybe a convention of TesterH or TestH filenames for hidden unit test files?
I guess this is a bit too restrictive, but would make it a simple convention.

Each of these would just set a flag in the test objects, and this flag would censor some parts of the test report.

I'll have a look at the files you mentioned. Thanks.

Footnotes

1. In fact, I always have some doubts about the interactions between different kinds of test, and about //CALL tests with instance methods, but I'd rather leave that for some other thread. ↩

[cayhorstmann]

Hi João, my remarks are for the benefit of Ammol Gil, a student volunteer from CodeDay who will implement this feature. Right now, the plan is to use a HIDDEN pseudocomment. This should work with unit tests, CALL and 'SUBsince you can always have multiple files, some hidden and some not. The sticky widget is running a file. I am pretty sure one can't currently have two files, each with its ownIN`. But that's a bug that should be addressed. We'll keep you in the loop as this feature evolves.

[cayhorstmann]

This is now implemented. https://horstmann.com/codecheck/authoring.html#hidden-tests

[jmr-ua-pt]

I have just tried submitting this small Python problem (single file):

"""
This function should compute the area of a rectangle with given width and height.
Fix it.
"""
##CALL 2, 2
##CALL HIDDEN 3, 5
def areaRectangle(w, h):
   ##HIDE
   return w * h
   ##EDIT return w + h

The public URL seems to work fine: the report hides the Arguments and Expected columns for the hidden test. However, since it does not hide the Actual output, the student can easily print the arguments in the hidden test, partially defeating the purpose of the hidden test. So, if we're hiding the arguments, I guess we must hide the output and results as well. (I just tried and failed to expose the argument in one of Nick Parlante's CodingBat problems.)

Another problem is that when I access the Edit URL, the page reports Cannot edit problem with directories. This happens even though there were no directories in the submitted problem, just a single file. I started noticing this yesterday, when submitting a problem with no hidden tests, so it may very well be unrelated to this new feature. (Weeks ago, I filed issue #40 , regarding the Cannot edit problem with directories, but back then, this only happened on problems that actually included directories.)

[cayhorstmann]

Responding to the first issue, what should be hidden? I am actually unsure why the arguments should be hidden. How is that going to help the student fix their issue? It seems pedagogically unsound. They are told that something gives an output of 8 and that's what isn't expected. What are they going to do with that information?

Perhaps we should display the arguments and hide the actual and expected results? That way, students know that SOMETHING is wrong with the provided inputs. Of course, they could compute by hand what is expected and then code an if/else on the arguments and print the answer. But look at the bright side--they have to compute what is expected. That shows some engagement.

[jmr-ua-pt]

Yes, I see your point, but it's not a simple matter to settle. I took some time to review the pros and cons of different options before forming an opinion. So, I'll try to summarize my reasoning here.

The way I see it, there are four possibly useful combinations or modes of operation for the test report.

args/input	actual output	expected output	mode
show	show	show	normal
show	show	hide	A
show	hide	hide	B
hide	hide	hide	C

(The other four combinations don't seem to make much sense.)

Of course, show/show/show is the normal mode of operation. We want to keep this.

Mode A provides info to the student about the failing case and corresponding output. The student has an opportunity to find and fix the bug. That's good. On the other hand, he/she may just figure out what the expected output should be and hard-code the result for that particular case, without actually programming a general solution. Figuring out the solution for a particular case has some merit, of course, but it really is not the same as actually solving the issue in the general case. And figuring out the result is quite simpler than actually programming a solution in many problems (think sorting). There is hardly any benefit over the normal mode in such problems.

Mode B provides info about the failing case, but hides the actual output, making it harder to pinpoint the bug. This could be a stronger incentive for the honest student to trace the algorithm, which might be good, but it may just as well encourage the student to take the shortcut of hard-coding the output for the particular case. This looks more confusing than mode A, and has no clear advantage.

Mode C hides everything. A failing test only tells the student that the solution does not handle all cases, with no further clues. I understand that this feels confusing and frustrating, but we may argue that this closer to the "real world" experience, where we may not even know the expected output for most inputs. Pinpointing the bug becomes a much more demanding, more abstract task. There are merits in this, too. Most importantly, I think it is the best way to avoid tailoring the code just to pass the tests, which was the motivating reason for this feature request in the first place.

So, other than providing all the possible modes, which would be too complex to implement and confusing to use, I would argue that mode C would be the most useful addition, exactly because it is the most different from the current normal mode, and the one that most effectively discourages "tailoring".

P.S.:
For debugging, students should be able to provide their own test inputs and see the actual outputs, as they do when developing programs outside of Codecheck. But this seems to be a different feature request altogether. And we can already do this, sort of. If the problem is suitably designed, the students may temporarily modify the main program to do whatever tests they want, even if that makes it fail the provided tests for a while. On some assignments, I provide students with a "sandbox" dummy (SAMPLE) problem were they can paste and run whatever code want.

[cayhorstmann]

I implemented it in this way, to hide everything. It would certainly be possible to separately hide input/args, actual, and expected, but let's wait if there is demand.

[jmr-ua-pt]

So far I only tested using ##CALL HIDDEN comments in a Python problem, and it worked fine.

This is a really nice feature.

Thanks.