From b82ea1884a8dc39e82d9066fa061b5aab52681f5 Mon Sep 17 00:00:00 2001 From: Michael Chow Date: Thu, 5 Jan 2017 10:40:22 -0500 Subject: [PATCH] Doc squash - check_function, has_equal_ast, call tests commit d1efd6a25a51d29292a211580236ff0c6da5be59 Merge: cb2836b ef3cf4a Author: Michael Chow Date: Thu Jan 5 10:39:46 2017 -0500 Merge branch 'master' into doc-update commit cb2836ba85931b443030553e2a9963390f40c738 Author: Michael Chow Date: Thu Jan 5 10:20:54 2017 -0500 Update has_equal_ast.md commit 869d1e99f9578c50f98a0a5375107c6a1995e8ed Author: Michael Chow Date: Thu Jan 5 10:20:33 2017 -0500 Update has_equal_ast.md commit 70eaab98d4effe61f54cb1bc0fffd596f643082b Author: Michael Chow Date: Thu Jan 5 10:10:11 2017 -0500 Create has_equal_ast.md commit 9bf24d223788f250d23c8fafb6e3637743a9bfd2 Author: Michael Chow Date: Thu Jan 5 10:09:07 2017 -0500 Update index.rst commit 1ecc95aa1fd323482a3cdcc7386b6ec16c3c297c Author: Michael Chow Date: Thu Jan 5 10:07:04 2017 -0500 Update part_checks.rst commit 74be751eeba3263b57edf7a7e50e4897fd20ad36 Author: Michael Chow Date: Thu Jan 5 10:06:36 2017 -0500 Update part_checks.rst commit 570dab040d2dee32658facb0f36c9b7ffef15141 Author: Michael Chow Date: Thu Jan 5 09:51:39 2017 -0500 Update expression_tests.md commit 919cafe05f3ff5ad8dbad9df8de214455a2c8e3d Author: Michael Chow Date: Thu Jan 5 09:50:01 2017 -0500 Update expression_tests.md commit 21da3992929603efa08cb6e96d52b9370aeef845 Author: Michael Chow Date: Thu Jan 5 09:44:00 2017 -0500 Update expression_tests.md commit f3b4fe21f1b4ea20ea4fd87a8e16b08d6b856f6c Author: Michael Chow Date: Thu Jan 5 09:28:19 2017 -0500 Update part_checks.rst commit e886696312b8b960f36197e20c9b59d4e7528938 Author: Michael Chow Date: Wed Jan 4 18:00:30 2017 -0500 Update spec2_summary.rst commit b49560d95c7d4772254b5905edec1b5469db42ab Author: Michael Chow Date: Wed Jan 4 17:57:23 2017 -0500 Update part_checks.rst commit 16fd7e1832860a2f92716baa2542c360fa5c6288 Author: Michael Chow Date: Wed Jan 4 17:50:03 2017 -0500 Update part_checks.rst commit 581f8ab5d21a5296c627df0b2a001bcd04652017 Author: Michael Chow Date: Wed Jan 4 17:47:46 2017 -0500 Update part_checks.rst commit 6f7e0a7ac02a9e71159edb29be763479090a357b Author: Michael Chow Date: Wed Jan 4 17:47:10 2017 -0500 Update part_checks.rst commit 83ba826c2f242286edc4ce6bfcd201446b0107c0 Author: Michael Chow Date: Wed Jan 4 17:40:54 2017 -0500 Update part_checks.rst commit a917161c79a7b30a09808f5b8b7c5036d30e9d60 Author: Michael Chow Date: Wed Jan 4 17:40:23 2017 -0500 Update part_checks.rst commit 018cd3888a3a4848419394d38c47c37aa9f864f8 Author: Michael Chow Date: Wed Jan 4 17:39:58 2017 -0500 Update part_checks.rst commit 6b0799895233c6d26119c7b26061128ea731baea Author: Michael Chow Date: Wed Jan 4 17:34:59 2017 -0500 Update part_checks.rst commit 243ad0c00ff754b65aecf5e20b61b3accb8aae55 Author: Michael Chow Date: Wed Jan 4 17:14:15 2017 -0500 Update part_checks.rst commit 15daa43cb2935cc9d9d6d57dbd7544da313a67d9 Author: Michael Chow Date: Wed Jan 4 16:55:40 2017 -0500 Update quickstart_guide.md commit 3adf5223f74fab01f3263744b14cd361ffdff66c Author: Michael Chow Date: Wed Jan 4 16:54:48 2017 -0500 Update quickstart_guide.md commit f649b3678060bcd20ff57fdbacbb7cc40ad2d14d Author: Michael Chow Date: Wed Jan 4 16:52:25 2017 -0500 Update quickstart_guide.md commit 72ebbe7669aa2d5921ad79fa0f1e5806aa70eee4 Author: Michael Chow Date: Wed Jan 4 16:47:01 2017 -0500 doc - fix indexing to be 0-based --- docs/source/expression_tests.md | 43 +++++- docs/source/part_checks.rst | 170 +++++++++++++++++++--- docs/source/quickstart_guide.md | 10 +- docs/source/simple_tests/has_equal_ast.md | 39 +++++ docs/source/simple_tests/index.rst | 1 + docs/source/spec2_summary.rst | 2 +- 6 files changed, 235 insertions(+), 30 deletions(-) create mode 100644 docs/source/simple_tests/has_equal_ast.md diff --git a/docs/source/expression_tests.md b/docs/source/expression_tests.md index 034532d7..0ded5b4f 100644 --- a/docs/source/expression_tests.md +++ b/docs/source/expression_tests.md @@ -281,10 +281,45 @@ For example, the following SCT simply runs `len(x)` in the solution and student `call` Syntax ------------- -* almost identical to has_equal_\*. -* special call syntax - - string form - - dict form +Testing a function definition or lambda may require calling it with some arguments. +In order to do this, use the `call()` SCT. +There are two ways to tell it what arguments to pass to the function/lambda, + +* `call("f (1, 2, x = 3)")` - as a string, where `"f"` gets substituted with the function's name. +* `call([1,2,3])` - as a list of positional arguments. + +Below, two alternative ways of specifying the arguments to pass are shown. + + *** =solution + ```{python} + def my_fun(x, y = 4, z = ('a', 'b'), *args, **kwargs): + return [x, y, *z, *args] + ``` + + *** =sct + ```{python} + Ex().check_function_def('my_fun').call("f(1, 2, (3,4), 5, kw_arg='ok')") # as string + Ex().check_function_def('my_fun').call([1, 2, (3,4), 5]) # as list + ``` + +```eval_rst +.. note:: + + Technically, you can get crazy and replace the list approach with a dictionary of the form ``{'args': [POSARG1, POSARG2], 'kwargs': {KWARGS}}``. +``` + +### Additional Parameters + +In addition to its first argument, `call()` accepts all the parameters that the expression tests above can (i.e. `has_equal_value`, `has_equal_error`, `has_equal_output`). +The function call is run at the point where these functions would evaluate an expression. +Moreover, setting the argument `test` to either "value", "output", or "error" controls which expression test it behaves like. + +For example, the SCT below shows how to run some `pre_code`, and then evaluate the output of a call. + +``` +Ex().check_function_def('my_fun').call("f(1, 2)", test="output", pre_code="x = 1") +``` + Managing Processes ----------------- diff --git a/docs/source/part_checks.rst b/docs/source/part_checks.rst index c4de9e14..1d8d0896 100644 --- a/docs/source/part_checks.rst +++ b/docs/source/part_checks.rst @@ -29,7 +29,7 @@ For example, in order to test the body of the comprehension above, we could crea *** =sct ```{python} - (Ex().check_list_comp(1) # focus on first list comp + (Ex().check_list_comp(0) # focus on first list comp .check_body().test_student_typed('i\*2') # focus on its body for test ) ``` @@ -52,10 +52,10 @@ This section expands the above example to run tests on each part: body, iter, an *** =sct ```{python} - list_comp = Ex().check_list_comp(1, missing_msg="Did you include a list comprehension?") + list_comp = Ex().check_list_comp(0, missing_msg="Did you include a list comprehension?") list_comp.check_body().test_student_typed('i\*2') list_comp.check_iter().has_equal_value() - list_comp.check_ifs(1).multi([has_equal_value(context_vals=[i]) for i in range(0,10)]) + list_comp.check_ifs(0).multi([has_equal_value(context_vals=[i]) for i in range(0,10)]) ``` In this SCT, the first line focuses on the first list comprehension, and assigns it to ``list_comp``, so we can test each part in turn. As a reminder, the code corresponding to each part in the solution code is.. @@ -87,14 +87,14 @@ The line .. code-block:: python - list_comp.check_ifs(1).multi([has_equal_value(context_vals=[i] for i in range(0,10))]) + list_comp.check_ifs(0).multi([has_equal_value(context_vals=[i] for i in range(0,10))]) is a doozy, but can be broken down into .. code-block:: python equal_tests = [has_equal_value(context_vals=[i] for i in range(0,10))] # collection of has_equal_tests - list_comp.check_ifs(1).multi(equal_tests) # focus on IFS run equal_tests` + list_comp.check_ifs(0).multi(equal_tests) # focus on IFS run equal_tests` In this case ``equal_tests`` is a list of ``has_equal_value`` tests that we'll want to perform. ``check_ifs(1)`` grabs the first IFS part, and ``multi(equal_tests)`` runs each ``has_equal_value`` test on that part. @@ -133,9 +133,9 @@ in order to test running the inline if expression we could go from list_comp => *** =sct ```{python} - (Ex().check_list_comp(1) # first comprehension + (Ex().check_list_comp(0) # first comprehension .check_body().set_context(i=6) # comp's body - .check_if_exp(1).has_equal_value() # body's inline IFS + .check_if_exp(0).has_equal_value() # body's inline IFS ) ``` @@ -148,7 +148,7 @@ If we left out the ``check_if_exp`` above, the resulting SCT, .. code:: python - (Ex().check_list_comp(1).check_body().set_context(i=6) + (Ex().check_list_comp(0).check_body().set_context(i=6) #.check_if_exp(1) .has_equal_value() ) @@ -326,7 +326,7 @@ but its BODY does. +------------------------+------------------------------------------------------+-------------------+ | check | parts | target variables | +========================+======================================================+===================+ -|check_if_else | .. code:: | | +|check_if_else(0) | .. code:: | | | | | | | | if TEST: | | | | BODY | | @@ -335,7 +335,7 @@ but its BODY does. | | | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_while | .. code:: python | | +|check_while(0) | .. code:: python | | | | | | | | while TEST: | | | | BODY | | @@ -343,22 +343,22 @@ but its BODY does. | | ORELSE | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_list_comp | .. code:: | ``i`` | +|check_list_comp(0) | .. code:: | ``i`` | | | | | | | [BODY for i in ITER if IFS[0] if IFS[1]] | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_generator_exp | .. code:: | ``i`` | +|check_generator_exp(0) | .. code:: | ``i`` | | | | | | | (BODY for i in ITER if IFS[0] if IFS[1]) | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_dict_comp | .. code:: | ``k``, ``v`` | +|check_dict_comp(0) | .. code:: | ``k``, ``v`` | | | | | | | {KEY : VALUE for k, v in ITER if IFS[0]} | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_for_loop | .. code:: | ``i`` | +|check_for_loop(0) | .. code:: | ``i`` | | | | | | | for i in ITER: | | | | BODY | | @@ -366,7 +366,7 @@ but its BODY does. | | ORELSE | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_try_except | .. code:: python | ``e`` | +|check_try_except(0) | .. code:: python | ``e`` | | | | | | | try: | | | | BODY | | @@ -380,24 +380,30 @@ but its BODY does. | | FINALBODY | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_with | .. code:: python | `f`` | +|check_with(0) | .. code:: python | `f`` | | | | | | | with CONTEXT_TEST as f: | | | | BODY | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_function_def | .. code:: python | argument names | +|check_function_def('f') | .. code:: python | argument names | | | | | | | def f(ARGS[0], ARGS[1]): | | | | BODY | | | | | | +------------------------+------------------------------------------------------+-------------------+ -|check_lambda | .. code:: | argument names | +|check_lambda(0) | .. code:: | argument names | | | | | | | lambda ARGS[0], ARGS[1]: BODY | | | | | | | | | | +------------------------+------------------------------------------------------+-------------------+ +|check_function('f', 0) | .. code:: | argument names | +| | | | +| | f(ARGS[0], ARGS[1]) | | +| | | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ More ------ @@ -425,8 +431,8 @@ can be checked with the following SCT .. code:: python - (Ex().check_if_else(1) # lines 1-3 - .check_orelse().check_if_else(1) # lines 2-3 + (Ex().check_if_else(0) # lines 1-3 + .check_orelse().check_if_else(0) # lines 2-3 .check_orelse().has_equal_output() # line 3 ) @@ -434,13 +440,129 @@ can be checked with the following SCT function definition / lambda args ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +the ARGS part in function definitions and lambdas may be selected by position or keyword. +For example, the arguments `a` and `b` below, + .. code:: python - def f(a, b): + def f(a, b=2, *some_name): BODY -*args and **kwargs +Could be tested using, + +.. code:: python + + Ex().check_function_def('f').multi( + check_args('a').is_default(), + check_args('b').is_default().has_equal_value(), + check_args('*args', 'missing a starred argument!') + ) + +Note that ``check_args('*args')`` and ``check_args('**kwargs')`` may be used to test *args, and **kwargs style parameters, regardless of their name in the function definition. + +function call args +~~~~~~~~~~~~~~~~~~~ + +Behind the scenes, ``check_function`` uses the same logic for matching arguments to function signatures as `test_function_v2 `__. +It also has a ``signature`` argument that accepts a custom signature. + +Matching Signatures +^^^^^^^^^^^^^^^^^^^^ + +By default, ``check_function`` tries to match each argument in the function call with the appropriate parameters in that function's call signature. +For example, all the calls to ``f`` below use ``a = 1`` and ``b = 2``. + +.. code:: + + def f(a, b): pass + + f(1, 2) # by position + f(a = 1, b = 2) # by keyword + f(1, b = 2) # mixed + +However, when testing a submission, we may not care how the argument was specified. + +.. code:: + + *** =pre_exercise_code + ```{python} + def f(a, b): pass + ``` + + *** =solution + ```{python} + f(1, b=2) + ``` + + *** =sct + ```{python} + Ex().check_function('f', 0).check_args('a').has_equal_value() + ``` + +will pass for all the ways of calling ``f`` listed above. + +signature = False ^^^^^^^^^^^^^^^^^^ -testing default values -^^^^^^^^^^^^^^^^^^^^^^ +Setting signature to false, as below, only allows you to check an argument by name, if the name was explicitly specified in the function call. +For example, + +.. code:: + + *** =solution + ```{python} + dict( [('a', 1)], c = 2) + ``` + + *** =sct + ```{python} + Ex().check_function('dict', 0, signature=False)\ + .multi( + check_args(0), # can only select by position + check_args('c') # could use check_args(1) + ) + ``` + +Note that here, an argument's position is referring to its position in the function call (not its signature). + +Example: testing a list passed as an argument +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Suppose you want to test the first argument passed to `sum`. +Below, we show how this can be down, using `has_equal_ast()` to check that the abstract syntax trees for the 1st argument match. + +.. code:: python + + *** =solution + ```{python} + sum([1, 2, 3]) + ``` + + *** =sct + ```{python} + (Ex().check_function('sum', 0) + .check_args(0) + .has_equal_ast("ast fail") # compares abstract representations + .test_student_typed("\[1, 2, 3\]", "typed fail") # alternative, more rigid test + ) + ``` + +Notice that testing the argument is similar to testing, say, the body of an if statement. +In this sense, we could even do deeper checks into an argument. +Below, the SCT verifies that the first argument passed to sum is a list comprehension. + +.. code:: python + + *** =solution + ```{python} + sum([i for i in range(10)]) + ``` + + *** =sct + ```{python} + (Ex().check_function('sum', 0) + .check_args(0) + .check_list_comp(0) + .has_equal_ast() + ) + ``` diff --git a/docs/source/quickstart_guide.md b/docs/source/quickstart_guide.md index d50a73df..8fdcb08c 100644 --- a/docs/source/quickstart_guide.md +++ b/docs/source/quickstart_guide.md @@ -55,4 +55,12 @@ the automatic messages for when `x` is undefined or incorrect are replaced with The same holds for `test_output_contains()`: you can use the `no_output_msg` argument to specify a custom message. For more information on all the different arguments you can set in the different `pythonwhat` functions, have a look at the articles in this wiki, describing them in detail. -[TODO: quick outline of next steps] +Next Steps +---------- + +Test functions in pythonwhat are broken into 4 groups: + +* [Simple tests](simple_tests/index.rst): look at, e.g., the output produced by an entire code submission. +* [Part checks](part_checks.rst): focus on specific pieces of code, like a particular for loop. +* [Expression tests](expression_tests.md): combined with part checks, these run pieces of code and evaluate the outcome. +* [Logic tests](logic_tests/index.rst): these allow logic like an or statement to be used with SCTs. diff --git a/docs/source/simple_tests/has_equal_ast.md b/docs/source/simple_tests/has_equal_ast.md new file mode 100644 index 00000000..25cfda6b --- /dev/null +++ b/docs/source/simple_tests/has_equal_ast.md @@ -0,0 +1,39 @@ +has_equal_ast +-------------- + +```eval_rst +.. automodule:: pythonwhat.check_funcs.has_equal_ast + :members: +``` + +An abstract syntax tree (AST) is a way of representing the high-level structure of python code. + +### Example: quotes + +Whether you use the concrete syntax `x = "1"` or `x = '1'`, the abstract syntax is the same: x is being assigned to the string "1". + +### Example: parenthesis + +Grouping by parentheses produces the same AST, when the same statement would work the same without them. +For example, `(True or False) and True`, and `True or False and True`, are the same due to operator precedence. + +### Example: spacing + +The same holds for different types of spacing that essentially specify the same statement: `x = 1` or `x = 1`. + +### Caveat: evaluating + +What the AST doesn't represent is values that are found through evaluation. For example, the first item in the list in + +```python +x = 1 +[x, 2, 3] +``` + +and + +```python +[1, 2, 3] +``` + +Is not the same. In the first case, the AST represents that a variable `x` needs to be evaluated in order to find out what its value is. In the second case, it just represents the value `1`. diff --git a/docs/source/simple_tests/index.rst b/docs/source/simple_tests/index.rst index f8ec5528..cf037998 100644 --- a/docs/source/simple_tests/index.rst +++ b/docs/source/simple_tests/index.rst @@ -13,4 +13,5 @@ A final, common use is to test the value of a variable in the final environment test_object test_output_contains test_student_typed + has_equal_ast test_mc diff --git a/docs/source/spec2_summary.rst b/docs/source/spec2_summary.rst index 934a38cf..63b8cc3f 100644 --- a/docs/source/spec2_summary.rst +++ b/docs/source/spec2_summary.rst @@ -1,4 +1,4 @@ -Spec2 Improvements +Spec2 Changes ================== .. role:: python(code)