Home

pythonwhat?

pythonwhat is a Python package that can help you write submission correctness tests (or SCTs) for interactive Python exercises on DataCamp. It allows you to easily compare parts in the solution code with parts that the student submits. Most functions in pythonwhat will allow you to define feedback for specific errors which the student has made, or will automatically generate a meaningful feedback message. A complete SCT for an interactive Python exercise should consist of a group of pythonwhat functions that will lead the student toward a correct submission with tailored feedback messages.

pythonwhat is heavily inspired on testwhat, an R package with functions to create SCTs for interactive exercises on DataCamp.

Writing SCTs, for which pythonwhat is built, is only one part of creating DataCamp exercises. For general documentation on creating Python courses on DataCamp, visit the Teach Documentation.

How does it work?

When a student starts an exercise on DataCamp, a Python session is started and the pre_exercise_code (PEC) is run. This code, that the author specifies, initializes the Python workspace with data, loads relevant packages etc, such that students can start coding the essence of the topics treated. Next, a separate solution environment is created, in which the same PEC and actual solution code, also coded by the author, is executed.

When a student submits an answer, his or her submission is executed and the output is shown in the IPython Shell. Then, the correctness of the submission is checked by executing the Submission Correctness Test, or SCT. Basically, your SCT is a Python script with calls to pythonwhat test functions. pythonwhat features a variety of functions to test a user's submission in different ways; examples are test_object(), test_function() and test_output_contains(). To do this properly, pythonwhat uses several resources:

• The student submission as text, to e.g. figure out which functions have been called.
• The solution code as text, to e.g. figure out whether the student called a particular function in the same way as it is called in the solution.
• The student environment (a.k.a student scope), to e.g. figure out whether a certain object was created.
• The solution environment (a.k.a. solution scope), to e.g. figure out whether an object that the student created corresponds to the object that was created by the solution.
• The output that's generated when executing the student code, to e.g. figure out if the student printed out something.

If, during execution of the SCT, a test function notices a mistake, an appropriate feedback will be generated and presented to the student. It is always possible to override these feedback messages with your own messages. Defining custom feedback will make your SCTs longer and they may be error prone (typos, etc.), but they typically give the exercise a more natural and personalized feel.

If all test functions pass, a success message is presented to the student. pytonwhat has some messages in store from which it can choose randomly, but you can override this with the success_msg() function.

Why does it work like this?

The fact that pythonwhat can use the solution code and the environment it produces, makes writing SCTs easy, powerful and less error-prone.

Example 1: Instead of testing whether an object x equals 3, you can simply use

test_object("x")

in your SCT. pythonwhat will automatically compare the value of x in the student environment with x in the solution environment.

Example 2: Suppose you want to test whether the function pow() was called correctly. If you use

test_function("pow")

in your SCT, pythonwhat will look for the function calls of pow in both student and solution code, extract their arguments and parameter values and compare them, generating meaningful feedback messages along the way.

Overview

To get started, make sure to check out the Quickstart Guide. This wiki also goes over every test function that pythonwhat features, explaining all arguments and covering different use cases. They will give you an idea of how, why and when to use them.

Basic functions

• Multiple choice exercises: test_mc()
• What student typed: test_student_typed()
• Which output is generated: test_output_contains()
• Value of object in ending scope: test_object()
• Usage of function: test_function()
• Usage of operators: test_operator()
• Package imports: test_import()

Logic-inducing functions

• If some tests fail, do other tests to pinpoint the problem: test_correct()
• Specify several tests, only one of which should pass: test_or()

Advanced functions

• Value of data frame in ending scope: test_data_frame()
• If ... else ... constructs: test_if_else()
• For loops: test_for_loop()
• While loops: test_while_loop()
• Context managers, the with statement: test_with()
• Test user-defined functions: test_function_definition()

All these functions are also documented inside the pythonwhat source code itself. The documentation there goes into the tiny details of all functions' implementations. Follow the steps in the README of this repository to generate a PDF version of the documentation.

For more full examples of SCTs for Python exercises on DataCamp, check out the source files of the introduction to Python course. In the chapter files there, you can can see the SCTs that have been written for several exercises.

After reading through this documentation, we hope writing SCTs for DataCamp becomes a painless experience. If this is not the case and you think improvements to pythonwhat and this documentation are possible, please let us know!