diff --git a/docs/source/Home.md b/docs/source/Home.md index cac5a0cc..c4bb782a 100644 --- a/docs/source/Home.md +++ b/docs/source/Home.md @@ -1,6 +1,8 @@ Home ==== +At DataCamp we build tools to [learn data science](https://www.datacamp.com) interactively. See e.g. our online [R tutorial](https://www.datacamp.com/courses/free-introduction-to-r) to learn R Programming and our Python For Data Science tutorial to [learn Python](https://www.datacamp.com/courses/intro-to-python-for-data-science). + pythonwhat? ----------- @@ -25,52 +27,17 @@ When a student submits an answer, his or her submission is executed and the outp If, during execution of the SCT, a test function notices a mistake, an appropriate feedback will be generated and presented to the student. It is always possible to override these feedback messages with your own messages. Defining custom feedback will make your SCTs longer and they may be error prone (typos, etc.), but they typically give the exercise a more natural and personalized feel. -If all test functions pass, a success message is presented to the student. `pytonwhat` has some messages in store from which it can choose randomly, but you can override this with the `success_msg()` function. +If all test functions pass, a success message is presented to the student. `pythonwhat` has some messages in store from which it can choose randomly, but you can override this with the `success_msg()` function. Overview -------- -To get started, make sure to check out the [Quickstart Guide](https://www.github.com/datacamp/pythonwhat/wiki/Quickstart_Guide). +To get started, make sure to check out the [Quickstart Guide](quickstart_guide.md). -To robustly test the equality of objects, and results of evaluations, it has to fetch the information from the respective processes, i.e. the student and solution processes. By default, this is done through a process of 'dilling' and 'undilling', but it's also possible to define your own converters to customize the way objects and results are compared. For more background on this, check out the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). For some more background on the principle of 'sub-SCTs', i.e. sets of tests to be called on a particular part or a particular state of a student's submission, have a look at the [Sub-SCTs article](https://github.com/datacamp/pythonwhat/wiki/Sub-SCTs). +To robustly test the equality of objects, and results of evaluations, it has to fetch the information from the respective processes, i.e. the student and solution processes. By default, this is done through a process of 'dilling' and 'undilling', but it's also possible to define your own converters to customize the way objects and results are compared. For more background on this, check out the [Processes article](expression_tests.md). For some more background on the principle of 'sub-SCTs', i.e. sets of tests to be called on a particular part or a particular state of a student's submission, have a look at the [Part Checks article](part_checks.rst). The remainder of the wiki goes over every test function that `pythonwhat` features, explaining all arguments and covering different use cases. They will give you an idea of how, why and when to use them. -**Basic functions** - -- Multiple choice exercises: [test_mc()](https://github.com/datacamp/pythonwhat/wiki/test_mc) -- What student typed: [test_student_typed()](https://github.com/datacamp/pythonwhat/wiki/test_student_typed) -- Which output is generated: [test_output_contains()](https://github.com/datacamp/pythonwhat/wiki/test_output_contains) -- Value of object: [test_object()](https://github.com/datacamp/pythonwhat/wiki/test_object) -- Usage of objects: [test_object_accessed()](https://github.com/datacamp/pythonwhat/wiki/tets_object_accessed) -- Usage of function: [test_function()](https://github.com/datacamp/pythonwhat/wiki/test_function) -- Usage of function, v2 (improved): [test_function_v2()](https://github.com/datacamp/pythonwhat/wiki/test_function_v2) -- Usage of operators: [test_operator()](https://github.com/datacamp/pythonwhat/wiki/test_operator) -- Package imports: [test_import()](https://github.com/datacamp/pythonwhat/wiki/test_import) - -**Logic-inducing functions** - -- If some tests fail, do other tests to pinpoint the problem: [test_correct()](https://www.github.com/datacamp/pythonwhat/wiki/test_correct) -- Specify several tests, only one of which should pass: [test_or()](https://www.github.com/datacamp/pythonwhat/wiki/test_or) - -**Advanced functions** - -- Value of data frame: [test_data_frame()](https://github.com/datacamp/pythonwhat/wiki/test_data_frame) -- Value of dictionary: [test_dictionary()](https://github.com/datacamp/pythonwhat/wiki/test_dictionary) -- If ... else ... constructs: [test_if_else()](https://github.com/datacamp/pythonwhat/wiki/test_if_else) -- For loops: [test_for_loop()](https://github.com/datacamp/pythonwhat/wiki/test_for_loop) -- While loops: [test_while_loop()](https://github.com/datacamp/pythonwhat/wiki/test_while_loop) -- Test value of object after expression: [test_object_after_expression()](https://github.com/datacamp/pythonwhat/wiki/test_object_after_expression) -- Test output of an expression: [test_expression_output()](https://github.com/datacamp/pythonwhat/wiki/test_expression_output) -- Test result of an expression: [test_expression_result()](https://github.com/datacamp/pythonwhat/wiki/test_expression_result) -- Context managers, the with statement: [test_with()](https://github.com/datacamp/pythonwhat/wiki/test_with) -- Test user-defined functions: [test_function_definition()](https://github.com/datacamp/pythonwhat/wiki/test_function_definition) -- Test lambda function definitions: [test_lambda_function()](https://github.com/datacamp/pythonwhat/wiki/test_lambda_function) -- Test comprehensions: [test_list_comp(), test_dict_comp() and test_generator_exp()](https://github.com/datacamp/pythonwhat/wiki/test_comprehension) -- Test try except blcoks: [test_try_except()](https://github.com/datacamp/pythonwhat/test_try_except) - -All these functions are also documented inside the `pythonwhat` source code itself. The documentation there goes into the tiny details of all functions' implementations. Follow the steps in the README of this repository to generate a PDF version of the documentation. - For more full examples of SCTs for Python exercises on DataCamp, check out the [source files of the introduction to Python course](http://www.github.com/datacamp/courses-intro-to-python). In the chapter files there, you can can see the SCTs that have been written for several exercises. To test your understanding of writing SCTs for Python exercises on the DataCamp platform, you can take the course [Writing SCTs with pythonwhat](https://www.datacamp.com/courses/writing-scts-with-pythonwhat) course. diff --git a/docs/source/_static/theme_overrides.css b/docs/source/_static/theme_overrides.css new file mode 100644 index 00000000..24899663 --- /dev/null +++ b/docs/source/_static/theme_overrides.css @@ -0,0 +1,3 @@ +code.docutils.literal { + color: black; +} diff --git a/docs/source/conf.py b/docs/source/conf.py index 7bd6be8b..510bc33d 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -307,9 +307,12 @@ def setup(app): 'auto_toc_tree_section': 'Contents', }, True) app.add_transform(AutoStructify) + app.add_stylesheet('theme_overrides.css') on_rtd = os.environ.get('READTHEDOCS', None) == 'True' if not on_rtd: # only import and set the theme if we're building docs locally import sphinx_rtd_theme html_theme = 'sphinx_rtd_theme' html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] + def setup(app): + app.add_stylesheet('theme_overrides.css') diff --git a/docs/source/expression_tests.md b/docs/source/expression_tests.md index ecaa05b0..034532d7 100644 --- a/docs/source/expression_tests.md +++ b/docs/source/expression_tests.md @@ -1,18 +1,295 @@ Expressions =========== -[TODO: Managing Processes taken from wiki, will update with spec2 functions] +Expression tests run pieces of the student and solution code, and then check the resulting value, printed output, or errors they produce. `has_equal` Syntax ------------------ +Once student/submission code has been selected using a check test, we can run it using one of three functions. +They all take the same arguments, and run the student and submission code in the same way. +However, they differ in how they compare the outcome: + +* has_equal_value - compares the value returned by the code. +* has_equal_output - compares printed output. +* has_equal_error - compares any errors raised. + +```eval_rst +.. autofunction:: pythonwhat.check_funcs.has_expr +``` + +### Basic Usage + +#### Running the whole code submission + +In the example below, we re-run the entire student and submission code, and check that they print out the same output. + + + *** =solution + ```{python} + x = [1,2,3] + print(x) + ``` + + *** =sct + ```{python} + # run all code and compare output + Ex().has_equal_output() + # equivalent to + # Ex().test_output_contains('[1,2,3]') + ``` + +Note that while we could have used `test_output_contains` to verify that the student printed `"[1, 2, 3]"`, +using `has_equal_output` simply requires that the student output matches the solution output. + +#### Running part of the code + +Combining an expression test with part checks will run only a piece of the submitted code. +The example below first uses `has_equal_value` to run an entire if expression, and then to run only its body. + + *** =solution + ```{python} + x = [1,2,3] + sum(x) if x else None + ``` + + *** =sct + ```{python} + # test body of if expression + (Ex().check_if_exp(0) # focus on if expression + .has_equal_value() # run entire if expression, check value + .check_body() # focus on body "sum(x)" + .has_equal_value() # run body, check value + ) + ``` + +Note that commands chaining off of `has_equal_value` behave as they would have if `has_equal_value` weren't used. +In this sense, the `check_body` behaves the same in + +```python +Ex().check_if_exp(0).has_equal_value().check_body() +``` + +and + +```python +Ex().check_if_exp(0).check_body() +``` + +in that it gets "sum(x)" in the solution code (and its corresponding code in the submission). + +### Context Values + +Suppose we want the student to define a function, that loops over the elements in a dictionary, and prints out each key and value, as follows: + + *** =solution + ```{python} + def print_dict(my_dict): + for key, value in my_dict.items(): + print(key + " - " + str(value)) + ``` + +An appropriate SCT for this exercise could be the following (for clarity, we're not using any default messages): + + *** =sct + ```{python} + # get for loop code, set context for my_dict argument + for_loop = (Ex() + .check_function_def('print_dict') # ensure 'print_dict' is defined + .check_body() # get student/solution code in body + .set_context(my_dict = {'a': 2, 'b': 3}) # set print_dict's my_dict arg + .check_for_loop(0) # ensure for loop is defined + ) + + # test for loop iterator + for_loop.check_iter().has_equal_value() # run iterator (my_dict.items()) + # test for loop body + for_loop.check_body().set_context(key = 'c', value = 3).has_equal_value() + + ``` + +Assuming the student coded the function in the exact same way as the solution, the following things happen: + +- checks whether `print_dict` is defined, then gets the code for the function definition body. +- because `print_dict` takes an argument `my_dict`, which would be undefined if we ran the body code, `set_context` defines what `my_dict` should be when running the code. Note that its okay if the submitted code named the argument `my_dict` something else, since set_context matches submission / solution arguments up by position. + +When running the bottom two SCTs for the for_loop + +- `for_loop.check_iter().has_equal_value()` - runs the code for the iterator, `my_dict.items()` in the solution and its corresponding code in the submission, and compares the values they return. +- `for_loop.check_body().set_context(key = 'c', value = 3).has_equal_value()` - runs the code in the for loop body, `print(key + " - " + str(value))` in the solution, and compares outputs. + Since this code may use variables the for loop defined, `key` and `value`, we need to define them using `set_context`. + +### How are Context Values Matched? + +Context values are matched by position. For example, the submission and solution codes... + + *** =solution + ```{python} + for ii, x in enumerate(range(3)): print(ii) + ``` + + *** =submission + ```{python} + for jj, y in enumerate(range(3)): print(jj) + ``` + +Using `Ex().check_for_loop(0).check_body().set_context(...)` will do the following... + +```eval_rst +====================== ======================= ========================== + statement solution (ii, x) submission (jj, y) +====================== ======================= ========================== +set_context(ii=1, x=2) ii = 1, x = 2 jj = 1, y = 2 +set_context(ii=1) ii = 1, x is undefined jj = 1, y is undefined +set_context(x=2) ii is undefined, x = 2 jj is undefined, y = 2 +====================== ======================= ========================== + +.. note:: + + If ::set_context:: does not define a variable, nothing is done with it. + This means that in the code examples above, running the body of the for loop would call print with ::ii:: or ::jj:: left at 2 (the values they have in the solution/submission environments). +``` + +### pre_code: fixing mutations + +Python code commonly mutates, or changes values within an object. +For example, the variable `x` points to an object that is mutated every time a function is called. + +```python +x = {'a': 1} + +def f(d): d['a'] += 1 + +f(x) # x['a'] == 2 now +f(x) # x['a'] == 3 now +``` + +In this case, when `f` is run, it changes the contents of `x` as a side-effect and returns None. +When using SCTs that run expressions, mutations in either the solution or submission environment can cause very confusing results. +For example, calling `np.random.random()` will advance numpy's random number generator. +In the code below the random seed is set to 42, but the solution code advances the random generator further than the submission code. As a result the SCT will fail. + + *** =pre_exercise_code + ```{python} + import numpy as np + np.random.seed(42) # set random generator seed to 42 + ``` + + *** =solution + ```{python} + if True: np.random.random() # 1st random call: .37 + + np.random.random() # 2nd random call: .95 + ``` + + *** =submission + ```{python} + if True: np.random.random() # 1st random call: .37 + + # forgot 2nd call to np.random.random() + ``` + + *** =sct + ```{python} + # Should pass but fails, because random generator has advanced + # twice in solution, but only once in submission + Ex().check_if_else(0).check_body().has_equal_value() + ``` + +In order to test random code, the random generator needs to be at the same state between submission and solution environments. +Since their generators can be thrown out of sync, the most reliable way to do this is to set the seed using the `pre_code` argument to `has_equal_value`. +In the case above, the sct may be fixed as follows + + *** =sct + ```{python} + Ex().check_if_else(0).check_body().has_equal_value(pre_code = "np.random.seed(42)") + ``` + +More generally, it can be helpful to define a pre_code variable to use before expression tests... + + *** =sct + ```{python} + pre_code = """ + np.random.seed(42) + """ + + Ex().has_equal_output(pre_code=pre_code) + Ex().check_if_else(0).check_body().has_equal_value(pre_code = pre_code) + ``` + +### extra_env: fixing slow SCTs + +The `extra_env` argument is similar to `pre_code`, in that you can (re)define objects in the student and submission environment before running an expression. +The difference is that, rather than passing a string that is executed in each environment, extra_env lets you pass objects directly. +For example, the two SCTs below are equivalent... + + *** =sct + ```{python} + Ex().has_equal_value(pre_code="x = 10") + Ex().has_equal_value(extra_env = {'x': 10}) + ``` + +In practice they can often be used interchangably. +However, one area where `extra_env` may shine is in mocking up data objects before running tests. +For example, if the SCT below didn't use extra_env, then it would take a long time to run. + + *** =pre_exercise_code + ```{python} + a_list = list(range(10000000)) + ``` + + *** =solution + ```{python} + print(a_list[1]) + ``` + + *** =sct + ```{python} + extra_env = {'a_list': list(range(10))} + Ex().has_equal_output(extra_env = extra_env) + ``` + +The reason extra_env is important here, is that pythonwhat tries to make a deepcopy of lists, so that course developers don't get bit by unexpected mutations. +However, the larger the list, the longer it takes to make a deepcopy. +If an SCT is running slowly, there's a good chance it uses a very large object that is being copied for every expression test. + +### name: run tests after expression + +### expr_code: change expression + +The `expr_code` argument takes a string, and uses it to replace the code that would be run by an expression test. +For example, the following SCT simply runs `len(x)` in the solution and student environments. + + *** =solution + ```{python} + # keep x the same length + x = [1,2,3] + ``` + + *** =SCT + ```{python} + Ex().has_equal_value(expr_code="len(x)") + ``` + +```eval_rst +.. note:: + + Using `expr_code` does not change how expression tests perform highlighting. + This means that `Ex().for_loop(0).has_equal_value(expr_code="x[0]")` would highlight the body of the checked for loop. +``` + `call` Syntax ------------- +* almost identical to has_equal_\*. +* special call syntax + - string form + - dict form + Managing Processes ----------------- -As mentioned on the [wiki Homepage](https://github.com/datacamp/pythonwhat/wiki), DataCamp uses two separate processes. One process to run the solution code, and one process to run the student's submission. This way, `pythonwhat` has access to the 'ideal ending scenario' of an exercises; this makes it easier to write SCTs. Instead of having to specify which value an object should be, we can have `test_object()` look into the solution process and compare the object in that process with the object in the student process. +As mentioned on the [Homepage](Home.md), DataCamp uses two separate processes. One process to run the solution code, and one process to run the student's submission. This way, `pythonwhat` has access to the 'ideal ending scenario' of an exercises; this makes it easier to write SCTs. Instead of having to specify which value an object should be, we can have `test_object()` look into the solution process and compare the object in that process with the object in the student process. ### Problem @@ -39,7 +316,7 @@ To be able to handle these errors, `pythonwhat` allows you to write your own con *** =sct ```{python} - test_object('xl') + Ex().test_object('xl') ``` Suppose now that objects such as `xl`, which are of the type `pandas.io.excel.ExcelFile`, can't be properly dilled and undilled. (Note: because of hardcoded converters inside `pythonwhat`, they can, see below). To make sure that you can still use `test_object('xl')` to test the equality of the `xl` object between student and solution process, you can manually define a converter with the `set_converter()` function. You can extend the SCT as follows: @@ -49,7 +326,7 @@ Suppose now that objects such as `xl`, which are of the type `pandas.io.excel.Ex def my_converter(x): return(x.sheet_names) set_converter(key = "pandas.io.excel.ExcelFile", fundef = my_converter) - test_object('xl') + Ex().test_object('xl') ``` With a lambda function, it's even easier: @@ -57,7 +334,7 @@ With a lambda function, it's even easier: *** =sct ``` set_converter(key = "pandas.io.excel.ExcelFile", fundef = lambda x: x.sheet_names) - test_object('xl') + Ex().test_object('xl') ``` The first arguemnt of `set_converter()`, the `key` takes the type of the object you want to add a manual converter for as a string. The second argument, `fundef`, takes a function definition, taking one argument and returning a single object. This function definition converts the exotic object into something more standard. In this case, the function converts the object of type `pandas.io.excel.ExcelFile` into a simple list of strings. A list of strings is something that can easily be converted into a bytestream and back into a Python object again, hence solving the problem. @@ -81,7 +358,7 @@ The `set_converter()` function opens up possibilities for objects that can actua *** =sct ``` set_converter(key = "numpy.ndarray", fundef = lambda x: x.shape) - test_object('my_array') + Ex().test_object('my_array') ``` Both of the following submissions will be accepted by this SCT: diff --git a/docs/source/logic_tests/index.rst b/docs/source/logic_tests/index.rst index 6e757ad9..8227204b 100644 --- a/docs/source/logic_tests/index.rst +++ b/docs/source/logic_tests/index.rst @@ -6,5 +6,4 @@ Logic Tests test_correct test_or - test_not - misc + test_no diff --git a/docs/source/logic_tests/misc.md b/docs/source/logic_tests/misc.md deleted file mode 100644 index 78c7f839..00000000 --- a/docs/source/logic_tests/misc.md +++ /dev/null @@ -1,61 +0,0 @@ -Misc ----- - -[TODO: this is a previous version, will update with spec2 functions] - -Several functions, such as `test_correct()`, `test_or()`, `test_not()`, among others, also have arguments that expect another set of tests. This article will explain the different ways of specifying these 'sub-tests'. - -Let's take the example of `test_correct()`; this function takes two sets of tests. The first set is executed, and if it's fine, the second set is left alone. If the first set of tests results in an error, the second set is executed and the feedback is logged. - -### Example 1 - -As an example, suppose you want the student to calculate the mean of a Numpy array `arr` and store it in `res`. A possible solution could be: - - *** =solution - ```{python} - # Import numpy and create array - import numpy as np - arr = np.array([1, 2, 3, 4, 5, 6]) - - # Calculate result - result = np.mean(arr) - ``` - -The first part of the tests here would be to check `result`. If `result` is not correct, you want to check whether `np.mean()` has been called. - -The most concise way to do this, is with lambda functions; you specify two sets of tests, that in this case consist of one test each: - - *** =sct - ```{python} - test_correct(check_object('result').has_equal_value(), - test_function('numpy.mean')) - success_msg("You own numpy!") - ``` - -### Example 2 - -When writing SCTs for more complicated exercises, you'll probably want to pass along several tests to an argument. - -Suppose that you expect the student to create an object `result` once more, but this time it's the sum of calling the `np.mean()` function twice; once on `arr1`, once on `arr2`: - - *** =solution - ```{python} - # Import numpy and create array - import numpy as np - arr1 = np.array([1, 2, 3, 4, 5, 6]) - arr2 = np.array([6, 5, 4, 3, 2, 1]) - - # Calculate result - result = np.mean(arr) + np.mean(arr) - ``` - -Now, in the 'digging deeper' part of `test_correct()`, you want to check the `np.mean()` function twice. -To do this, you'll want to use a function definition; lambda functions are not practical anymore: - - *** =sct - ```{python} - diagnose = [test_function('numpy.mean', index = i) for i in [1,2]] - - test_correct(test_object('result'), diagnose) - success_msg("You own numpy!") - ``` diff --git a/docs/source/logic_tests/test_correct.md b/docs/source/logic_tests/test_correct.md index ae1a2ac4..0c1350ae 100644 --- a/docs/source/logic_tests/test_correct.md +++ b/docs/source/logic_tests/test_correct.md @@ -8,7 +8,7 @@ test_correct A wrapper function around `test_or()`, `test_correct()` allows you to add logic to your SCT. Normally, your SCT is simply a script with subsequent `pythonwhat` function calls, all of which have to pass. `test_correct()` allows you to bypass this: you can specify a "sub-SCT" in the `check` part, that should pass. If these tests pass, the "sub-SCT" in `diagnose` is not executed. If the tests don't pass, the "sub-SCT" in `diagnose` is run, typically to dive deeper into what the error might be and give more specific feedback. -To accomplish this, the lambda function in `check` is executed silently, so that failure will not cause the SCT to stop and generate a feedback message. If the execution passes, all is good and `test_correct()` is abandoned. If it fails, `diagnose` is executed, not silently. If the `diagnose` part fails, the feedback message that it generates is presented to the student. If it passes, the `check` part is executed again, this time not silently, to make sure that a `test_correct()` that contains a failing `check` part leads to a failing SCT. +To accomplish this, the SCT in `check` is executed silently, so that failure will not cause the SCT to stop and generate a feedback message. If the execution passes, all is good and `test_correct()` is abandoned. If it fails, `diagnose` is executed, not silently. If the `diagnose` part fails, the feedback message that it generates is presented to the student. If it passes, the `check` part is executed again, this time not silently, to make sure that a `test_correct()` that contains a failing `check` part leads to a failing SCT. ### Example 1 @@ -28,28 +28,51 @@ You want the SCT to pass when the student manages to store the correct value in *** =sct ```{python} - test_correct(lambda: test_object('result'), - lambda: test_function('numpy.mean')) + test_correct(test_object('result'), + test_function('numpy.mean')) success_msg("You own numpy!") ``` -Notice that you have to use lambda functions to use Python as a functional programming language. Let's go over what happens when the student submits different pieces of code: -- The student submits `result = np.mean(arr)`, exactly the same as the solution. `test_correct()` executes the first lambda function, `test_object('result')`. This test passes, so `test_correct()` is exited and `test_function()` is not executed. The SCT passes. -- The student submits `result = np.sum(arr) / arr.size`, which also leads to the correct value in `result`. `test_correct()` executes the first lambda function, `test_object('result')`. This test passes, so `test_correct()` is exited and `test_function()` is not executed. So the entire SCT passes even though `np.mean()` was not used. -- The student submits `result = np.mean(arr + 1)`. `test_correct()` executes the first lambda function, `test_object('result')`. This test fails, so `test_correct()` heads over to second, 'diagnose' lambda function and executes `test_function('numpy.mean')`. This function will fail, because the argument passed to `numpy.mean()` in the student submission does not correspond to the argument passed in the solution. A meaningful, specific feedback message is presented to the student: you did not correctly specify the arguments inside `np.mean()`. -- The student submits `result = np.mean(arr) + 1`. `test_correct()` executes the first lambda function, `test_object('result')`. This test fails, so `test_correct()` heads over to the second, 'diagnose' lambda function and executes `test_function('numpy.mean'). This function passes, because `np.mean()` is called in exactly the same way in the student code as in the solution. Because there is something wrong - `result` is not correct - the 'check' lambda function, `test_object('result')` is executed again, and this time its feedback on failure is presented to the student. The student gets the message that `result` does not contain the correct value. +- The student submits `result = np.mean(arr)`, exactly the same as the solution. + `test_correct()` runs `test_object('result')`. + This test passes, so `test_correct()` stops. + The SCT passes. +- The student submits `result = np.sum(arr) / arr.size`, which also leads to the correct value in `result`. + `test_correct()` runs `test_object('result')`. + This test passes, so `test_correct()` stops before running `test_function()`. + The entire SCT passes even though `np.mean()` was not used. +- The student submits `result = np.mean(arr + 1)`. + `test_correct()` runs `test_object('result')`. + This test fails, so `test_correct()` continues with 'diagnose', running `test_function('numpy.mean')`. + This function fails, since the argument passed to `numpy.mean()` in the student submission does not correspond to the argument passed in the solution. + A meaningful, specific feedback message is presented to the student: you did not correctly specify the arguments inside `np.mean()`. +- The student submits `result = np.mean(arr) + 1`. + `test_correct()` runs `test_object('result')`. + This test fails, so `test_correct()` continues with'diagnose', running `test_function('numpy.mean'). + This function passes, because `np.mean()` is called in exactly the same way in the student code as in the solution. + Because there is something wrong - `result` is not correct - the 'check' SCT, `test_object('result')` is executed again, and this time its feedback on failure is presented to the student. + The student gets the message that `result` does not contain the correct value. + ### Multiple functions in `diagnose` and `check` -You can also use `test_correct()` with entire 'sub-SCTs' that are composed of several SCT calls. In this case, you have to define an additional function that executes the tests you want to perform in this sub-SCT, and pass this function to `test_correct()`. +You can also use `test_correct()` with entire 'sub-SCTs' that are composed of several SCT calls. In this case, you may put multiple tests inside `multi()`, as below.. + + *** =sct + ```{python} + Ex().test_correct( + multi(test_object('a'), test_object('b')), # multiple check SCTs + test_function('numpy.mean') + ) + ``` ### Why to use `test_correct()` You will find that `test_correct()` is an extremely powerful function to allow for different ways of solving the same problem. You can use `test_correct()` to check the end result of a calculation. If the end result is correct, you can go ahead and accept the entire exercise. If the end result is incorrect, you can use the `diagnose` part of `test_correct()` to dig a little deeper. -It is also perfectly possible to use `test_correct()` inside another `test_correct()`, although things can get funky with the lambda functions in this case. +It is also perfectly possible to use `test_correct()` inside another `test_correct()`. ### Wrapper around `test_or()` diff --git a/docs/source/logic_tests/test_or.md b/docs/source/logic_tests/test_or.md index 7763a28c..6b3cc5b6 100644 --- a/docs/source/logic_tests/test_or.md +++ b/docs/source/logic_tests/test_or.md @@ -6,7 +6,7 @@ test_or :members: ``` -This function simply tests whether one of the SCTs you specify inside it passes. You should pass your SCTs within lambda expressions or custom defined functions (like you do for functions like `test_correct()`, `test_for_loop()`, or `test_function_definition`). +This function simply tests whether one of the SCTs you specify inside it passes. Suppose you want to check whether people correctly printed out any integer between 3 and 7. A solution could be: @@ -19,9 +19,9 @@ To test this in a robust way, you could use `test_output_contains()` with a suit *** =sct ```{python} - test_or(lambda: test_output_contains('4'), - lambda: test_output_contains('5'), - lambda: test_output_contains('6')) + test_or(test_output_contains('4'), + test_output_contains('5'), + test_output_contains('6')) success_msg("Nice job!") ``` diff --git a/docs/source/part_checks.rst b/docs/source/part_checks.rst index ce1568ba..c4de9e14 100644 --- a/docs/source/part_checks.rst +++ b/docs/source/part_checks.rst @@ -12,7 +12,7 @@ In Brief While functions beginning with ``test_``, such as ``test_student_typed`` look over some code or output, ``check_`` functions allow us to zoom in on parts of that student and solution code. -For example, [check_list_comp](#check_list_comp) examines list comprehensions by breaking them into 3 parts: ``body``, ``comp_iter``, and ``ifs``. This is shown below. +For example, ``check_list_comp`` examines list comprehensions by breaking them into 3 parts: ``body``, ``comp_iter``, and ``ifs``. This is shown below. :code:`[i*2 for i in range(10) if i>2]` => :code:`[BODY for i in COMP_ITER if IFS]` diff --git a/docs/source/pythonwhat.wiki/test_data_frame.md b/docs/source/pythonwhat.wiki/test_data_frame.md index fb0b403c..efb8944b 100644 --- a/docs/source/pythonwhat.wiki/test_data_frame.md +++ b/docs/source/pythonwhat.wiki/test_data_frame.md @@ -34,4 +34,4 @@ To test this we simply use: This SCT will first test if `pandas` is correctly imported, and will then check if the student created a Pandas DataFrame called `my_df`. If it was not defined, a message is generated that you can override with `undefined_msg`. If the object was defined but it isn't a Pandas DataFrame, as message is generated that you can override wiht `not_data_frame_msg`. If `my_df` is a Pandas DataFrame, `test_data_frame()` goes on to check if all columns that are specified in the `columns` argument are defined in the data frame, and next whether these columns are correct. The messages that are generated in case of an incorrect submission can be overrided with `undefined_cols_msg` and `incorrect_msg`, respectively. -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.rst). diff --git a/docs/source/pythonwhat.wiki/test_dictionary.md b/docs/source/pythonwhat.wiki/test_dictionary.md index 51785d76..bf59cc92 100644 --- a/docs/source/pythonwhat.wiki/test_dictionary.md +++ b/docs/source/pythonwhat.wiki/test_dictionary.md @@ -40,4 +40,4 @@ Suppose you want the student to create a dictionary `x`, that contains three key - Step 3: if the student submits `x = {'a':123, 'b':456, 'd':78}`, the feedback _Have you specified a key `c` inside `x`?_ will be presented. You can override this by specifying `key_missing_msg` yourself. - Step 4: if the student submits `x = {'a':123, 'b':456, 'c':78}`, the feedback _Have you specified the correct value for the key `c` inside `x`?_ will be presented. You can override this by specifying `incorrect_value_msg` yourself. -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_expression_result.md b/docs/source/pythonwhat.wiki/test_expression_result.md index 5ce97de7..4f82c059 100644 --- a/docs/source/pythonwhat.wiki/test_expression_result.md +++ b/docs/source/pythonwhat.wiki/test_expression_result.md @@ -16,4 +16,4 @@ test_expression_result `test_expression_result()` works pretty much the same as `test_expression_output()` and takes the same arguments. However, in this case, the expression should be a single expression and can't be a 'tree of expressions', such as the entire body of a function definition for example. Currently, the only places where `test_expression_result()` is used, is inside inherently 'single expression parts' of your code, such as the sequence specification of a `for` loop, the expression of a lambda function, etc. -The example in the [`test_expression_output()` article](https://github.com/datacamp/pythonwhat/wiki/test_expression_output) also explains the use of `test_expression_result()`. +The example in the [`test_expression_output()` article](test_expression_output.md) also explains the use of `test_expression_result()`. diff --git a/docs/source/pythonwhat.wiki/test_function_definition.md b/docs/source/pythonwhat.wiki/test_function_definition.md index a0439dcc..293b4792 100644 --- a/docs/source/pythonwhat.wiki/test_function_definition.md +++ b/docs/source/pythonwhat.wiki/test_function_definition.md @@ -191,4 +191,4 @@ There are different things to note: ### Sidenote -Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_function_v2.md b/docs/source/pythonwhat.wiki/test_function_v2.md index 6b9c0da4..4e3c6395 100644 --- a/docs/source/pythonwhat.wiki/test_function_v2.md +++ b/docs/source/pythonwhat.wiki/test_function_v2.md @@ -16,7 +16,7 @@ test_function_v2 params_not_specified_msg=None, incorrect_msg=None) -`test_function_v2()` enables you to test whether the student called a function correctly. The function first tests if the specified function is actually called by the student, and then compares the call with calls of the function in the solution code. Next, it can compare the arguments passed to these functions. Because `test_function_v2()` also uses the student and solution process, this can be done in a very concise way. `test_function_v2()` is an improved version of [`test_function()`](https://github.com/datacamp/pythonwhat/wiki/test_function), where: +`test_function_v2()` enables you to test whether the student called a function correctly. The function first tests if the specified function is actually called by the student, and then compares the call with calls of the function in the solution code. Next, it can compare the arguments passed to these functions. Because `test_function_v2()` also uses the student and solution process, this can be done in a very concise way. `test_function_v2()` is an improved version of [`test_function()`](test_function.md), where: - there is resilience against different ways of calling a function (arguments vs keywords), - you have to be specific about which parameters you want to check, @@ -262,7 +262,7 @@ The following full example shows how it's done: ### Extra: Argument equality -Just like with `test_object()`, evaluated arguments are compared using the `==` operator (check out [the section about Object equality](https://github.com/datacamp/pythonwhat/wiki/test_object#object-equality)). For a lot of complex objects, the implementation of `==` causes the object instances to be compared... not their underlying meaning. For example when the solution is: +Just like with `test_object()`, evaluated arguments are compared using the `==` operator (check out [the section about Object equality](test_object.md#object-equality)). For a lot of complex objects, the implementation of `==` causes the object instances to be compared... not their underlying meaning. For example when the solution is: *** =solution ``` @@ -292,6 +292,6 @@ The SCT will fail even if the student uses this exact solution code. The reason lambda: test_function_v2("pandas.read_sql_query", do_eval=False) ) -This SCT will not do exactly the same, but it will test enough in practice 99% of the time. Check out [the section about Object equality](https://github.com/datacamp/pythonwhat/wiki/test_object#object-equality) for complex objects that DO have a good equality implementation. +This SCT will not do exactly the same, but it will test enough in practice 99% of the time. Check out [the section about Object equality](test_object.md#object-equality) for complex objects that DO have a good equality implementation. -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_if_exp.md b/docs/source/pythonwhat.wiki/test_if_exp.md index 606845cf..9ff8f097 100644 --- a/docs/source/pythonwhat.wiki/test_if_exp.md +++ b/docs/source/pythonwhat.wiki/test_if_exp.md @@ -5,7 +5,7 @@ test_if_exp .. autofunction:: pythonwhat.test_funcs.test_if_else.test_if_exp ``` -`test_if_exp` is a wrapper around `test_if_else`, which tells it to look for inline `if` expressions. As such, it uses the same arguments. [See `test_if_else` for more info](test_if_else). +`test_if_exp` is a wrapper around `test_if_else`, which tells it to look for inline `if` expressions. As such, it uses the same arguments. [See `test_if_else` for more info](test_if_else.md). ### What is an inline `if` expression? diff --git a/docs/source/pythonwhat.wiki/test_lambda_function.md b/docs/source/pythonwhat.wiki/test_lambda_function.md index 79ffe488..b5a9bca5 100644 --- a/docs/source/pythonwhat.wiki/test_lambda_function.md +++ b/docs/source/pythonwhat.wiki/test_lambda_function.md @@ -83,4 +83,4 @@ As usual, the `test_lambda_function()` will generate a bunch of meaningful autom This is practically impossible to do in a robust way; we suggest you do this in an indirect way (checking the output that should be generated, checking the object that should be created, etc). -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_operator.md b/docs/source/pythonwhat.wiki/test_operator.md index 75f35668..dff7e113 100644 --- a/docs/source/pythonwhat.wiki/test_operator.md +++ b/docs/source/pythonwhat.wiki/test_operator.md @@ -59,4 +59,4 @@ You can learn about `test_object()` and `test_function()` in the other articles, This SCT will be more forgiving, but the result is still checked with `test_object()` so the student will still have to calculate the correct value. This time, however, it is not checked by `test_operator()` because `do_eval = False`. `used = ["**"]` is used to tell the system to only check on the `**` operator for the first operation group. -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_try_except.md b/docs/source/pythonwhat.wiki/test_try_except.md index 0b210b56..1dab8e8d 100644 --- a/docs/source/pythonwhat.wiki/test_try_except.md +++ b/docs/source/pythonwhat.wiki/test_try_except.md @@ -22,7 +22,7 @@ With `test_try_except`, you can check whether the student correctly coded a `try As usual, `index` controls which try-except block to check. With `not_called_msg` you can choose a custom message to override the automatically defined message in case not enough try-except blocks weren't found in the student code. `body` is a sub-sct to test the code of the `try` block. `orelse` and `finalbody` work the same way, but here there are also `_msg` arguments to provide custom messages in case these parts ar missing. Finally, there's also `handlers` and `except_missing_msg`. `handlers` should be a dictionary, where the keys are the error classes you expect the student to capture (for the general `except:`, use `'all'`), and the values are sub-SCTs for each of these `except` blocks. An `except` block is only checked for existence and correctness if you mention it inside `handlers`. If it is not available, an automatic message will be generated, but this can ge overriden with `expect_missing_msg`. -Note: For more information on sub-SCTs, visit [the dedicated article](https://github.com/datacamp/pythonwhat/wiki/Sub-SCTs). +Note: For more information on sub-SCTs, visit [part checks](../part_checks.rst). ### Example 1 diff --git a/docs/source/simple_tests/index.rst b/docs/source/simple_tests/index.rst index a6c0fbbd..f8ec5528 100644 --- a/docs/source/simple_tests/index.rst +++ b/docs/source/simple_tests/index.rst @@ -2,7 +2,7 @@ Simple Tests ============ Simple tests are the most basic tests available in pythonwhat. -They don't focus on specific pieces of a submission (like part checks [LINK]), or re-run any code (like expression tests [LINK]). +They usually don't focus on specific pieces of a submission (:doc:`like part checks `), or re-run any code (:doc:`like expression tests ). Instead, they simply look at things like imports, printed output, or raw code text. A final, common use is to test the value of a variable in the final environment (that is, after the submission of solution code have been run). diff --git a/docs/source/simple_tests/test_object.md b/docs/source/simple_tests/test_object.md index 21520501..4e0120fd 100644 --- a/docs/source/simple_tests/test_object.md +++ b/docs/source/simple_tests/test_object.md @@ -13,7 +13,7 @@ test_object `test_object()` enables you to test whether a student correctly defined an object. -As explained on the [wiki home](https://github.com/datacamp/pythonwhat/wiki), both the student's submission as well as the solution code are executed, in separate processes. `test_object()` looks at these processes and checks if the object specified in `name` is available in the student process. Next, it checks whether the object in the student and solution process correspond. In case of a failure along the way, `test_object()` will generate a meaningful feedback message that you can override. +As explained on the [docs home](/Home.md), both the student's submission as well as the solution code are executed, in separate processes. `test_object()` looks at these processes and checks if the object specified in `name` is available in the student process. Next, it checks whether the object in the student and solution process correspond. In case of a failure along the way, `test_object()` will generate a meaningful feedback message that you can override. ### Example 1 @@ -94,6 +94,6 @@ Of course primitive classes like `str`, `int`, `list`, `dict`, ... can be tested #### Manually define a converter -As explained in the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes), objects are extracted from their respected processes by 'dilling' and 'undilling' them. However, you can manually set a 'converter' with the `set_converter()` function. This will override the default dilling and undilling behavior, and enables you to make simplified representations of custom objects, testing only exactly what you want to test. Learn more about it [here](https://github.com/datacamp/pythonwhat/wiki/Processes). +As explained in the [Processes article](../expression_tests.md), objects are extracted from their respected processes by 'dilling' and 'undilling' them. However, you can manually set a 'converter' with the `set_converter()` function. This will override the default dilling and undilling behavior, and enables you to make simplified representations of custom objects, testing only exactly what you want to test. -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). +**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md).