diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index fc11e8c..f8fc38b 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -34,6 +34,9 @@ jobs: - name: Run unit-tests run: | tox -e py37 + - name: Run examples + run: | + tox -e examples - name: Upload test coverage uses: codecov/codecov-action@v1 - name: Build and publish mechanical-markdown diff --git a/examples/README.md b/examples/README.md index 2433b45..cd24af1 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,47 +1,174 @@ -# Quick Start +# Mechanical markdown by example -Once you have the package installed, if you want to try out some simple examples, navigate to the examples/ directory and have a look at start.md: +## Prerequisites - # Quick Start Example +Be sure you have mechanical markdown [installed](../README.md#installing) and that the mm.py utility is in your $PATH. These examples were written with basic bash commands in mind, but any bash like shell should work. See [Shells](#shells) below for alternatives. + +## Using this guide + +All markdown files in examples/ (including this one!) are annotetated and can be executed and validated with: + +```bash +mm.py filename.md +``` + +This guide automatically executed as part of this project's continuous integration pipeline. It serves as both user guide and integration test suite for this package. If you use it run through it, hopefully you will see what a powerful concept self executing documentation can be. + +## How mechanical-markdown works + +One of the beautiful features of markdown is that it is both human and machine readable. A human can read a user guide and copy paste steps into their terminal. A machine can do the same and do some extra validation on the steps to make sure they executed correctly. Also, most markdown engines support embedded HTML comments ``````. We can use these HTML comments to embed extra information to tell our validation program what output we expect from a command. The ```mm.py``` utility will automatically extract this information, allong with the commands to execute from our markdown files. + +## Annotation format + +To tell mechanical-markdown what parts of your document need to be executed as code, you must add an HTML comment that begins with the token ```STEP```. After this token, mechanical-markdown will interpret the rest of the comment as a yaml document with instructions for how the code blocks should be executed and verified. All fields in this yaml document are optional. Finish the comment to denote then end of the yaml document and the beginning of your exectuable markdown code. Finally, finish with an html comment like this: `````` to denote the end of a step. Let's look at a basic example: - This is an example markdown file with an annotated test command - - + + You can use regular markdown anywhere during a step. It will be ignored. Only denoted as bash or sh will be executed. + ```bash - echo "test" + echo "Hello World!" ``` - + + This python code block will not be executed: + + ```python + print("This python script will not be run") + ``` + - + This unannotated command will not be run: + ```bash echo "A command that will not be run" ``` +Here's how the above will render in your markdown interpreter. + +---- + + + +You can use regular markdown anywhere during a step. It will be ignored. Only code blocks denoted as bash or sh will be executed. + +```bash +echo "Hello World!" +``` + +This python code block will not be executed: + +```python +print("This python script will not be run") +``` + + + +This unannotated command will not be run: + +```bash +echo "A command that will not be run" +``` + +---- + +Let's breakdown what this embedded yaml annotation is doing. There are two fields in our yaml document ```name``` and ```expected_stdout_lines```. The name field simply provides a name for the step that will be printed to the report that mm.py generates. The expected_stdout_lines field is actually telling mm.py what it should be looking for from stdout when it executes our code block(s). For more on this, checkout [io.md](io.md). + + +Code blocks must be tagged as either "bash" or "sh". Code from other languages will be ignored. + + + + + +## CLI + +### Help +For a list of options: + + + +```bash +mm.py --help +``` + + + +``` +usage: mm.py [-h] [--dry-run] [--manual] [--shell SHELL_CMD] markdown_file + +Auto validate markdown documentation + +positional arguments: +optional arguments: + -h, --help show this help message and exit + --dry-run, -d Print out the commands we would run based on + markdown_file + --manual, -m If your markdown_file contains manual validation + steps, pause for user input + --shell SHELL_CMD, -s SHELL_CMD + Specify a different shell to use +``` + +### Dry Run You can do a dry run to print out exactly what commands will be run using the '-d' flag. + + ```bash -mm.py -d start.md +mm.py -d README.md ``` -You'll see output like the following: + + + +This will print out all the steps that would be run, without actually running them. Output looks something like this ``` Would run the following validation steps: -Step: First Step +Step: Hello World commands to run with 'bash -c': - `echo "test"` + `echo "Hello World!"` Expected stdout: - test + Hello World! Expected stderr: +... ``` +### Run and Validate + Now you can run the steps and verify the output: ```bash @@ -51,18 +178,54 @@ mm.py start.md The script will parse the markdown, execute the annotated commands, and then print a report like this: ``` -Running shell 'bash -c' with command: `echo "test"` -Step: First Step - command: `echo "test"` +Running shell 'bash -c' with command: `echo "Hello World!"` +Running shell 'bash -c' with command: `mm.py --help` +Running shell 'bash -c' with command: `mm.py -d README.md` + +Step: Hello World + command: `echo "Hello World!"` return_code: 0 Expected stdout: - test + Hello World! Actual stdout: - test + Hello World! Expected stderr: Actual stderr: +... ``` If anything unexpected happens, you will get report of what went wrong, and mm.py will return non-zero. + +### Shells + +The default shell used to execute scripts is ```bash -c```. You can use a different shell interpreter by specifying one via the cli: + +```bash +mm.py -s 'zsh -c' README.md +``` + +### Manual validation + +You can add manual validation steps to your document. A manual validation step is just a pause message to allow the user to take some manual step like opening a browser. These steps normally get ignored, as ```mm.py``` is designed to do automated validation by default. If you run the following, it will enable mm.py to pause for user input. (View raw markdown for an example of what a manual_pause_message looks like): + + + + + + + + +```bash +mm.py -m README.md +``` + +# More examples: + +- For more details on checking stdout/stderr: [I/O Validation](io.md) +- For more details on setting up the execution environment: [Environment Variables](env.md) and [Working Directory](working_dir.md) +- For controlling timeouts, backgrounding, and adding delay between steps: [Sleeping, Timeouts, and Backgrounding](background.md) diff --git a/examples/background.md b/examples/background.md new file mode 100644 index 0000000..e22778d --- /dev/null +++ b/examples/background.md @@ -0,0 +1,110 @@ +# Sleeping, Timeouts, and Backgrounding + +## Using this example +To see a summary of what commands will be run: + +```bash +mm.py -d env.md +``` + +To run this file and validate the expected output: + +```bash +mm.py +``` + +Be sure to checkout the raw version of this file to see the annotations. + +## Sleeping + +Sometimes when running a series of steps automatically, they will run much faster than a human executing steps manually. If this leads to trouble for your procedures, you can insert a delay after running a command by using the ```sleep``` directive. + +This first date command has a 5 second sleep: + + + +```bash +date +echo "First step" +``` + + + +You'll see at least a 5 second delay before this sleep gets executed: + + + +```bash +date +echo "Second step" +``` + + + +## Backgrounding + +Conversely, if you want to run a step in the background without waiting to move on to the next step. This is great for starting services or daemons that you will clean up later on in the procedure. All backgrounded steps will be waited for at the end of execution so that stdout and stderr and the return code can all be checked. If a processes hasn't finished it will be timed out after 60s by default. See Timeout section below for more info. + +In this first step, run a command that will take at least 5 seconds, but mm.py doesn't wait for it before executing the next step. + + + +```bash +sleep 5 && echo "Background step" +date +``` + + + +The next step will be exeuted right away, and the background step will be joined after all non-backgrounded steps have completed. + + + +```bash +echo "Foreground step" +date +``` + + + +## Timeouts + +By default, all commands timeout after 60s. They will receive a SIGTERM, followed by a SIGKILL. Script that reach their timeout and are killed will cause validation to fail and mm.py will return non-zero. You can change the duration of the timeout for an individual step by setting ```timeout_seconds``` . + +> **Note:** sleep time does not count towards timeout_seconds. + + + +```bash +echo "Timeout step" +date +``` + + + +# Navigation + +Back to [Working Directory](working_dir.md) diff --git a/examples/env.md b/examples/env.md new file mode 100644 index 0000000..10df3b2 --- /dev/null +++ b/examples/env.md @@ -0,0 +1,69 @@ +# Environment Variables + +## Using this example +To see a summary of what commands will be run: + +```bash +mm.py -d env.md +``` + +To run this file and validate the expected output: + +```bash +mm.py +``` + +Be sure to checkout the raw version of this file to see the annotations. + +## Setting environment variables for your commands + +You can add environent variables by adding env to your step description yaml. +You can specify multiple environment variables as a dictionary of key value pairs. + + + +```bash +echo $ENV_VARIABLE +echo $SECOND_ENV_VARIABLE +``` + + + + + +If your code block overwrites an environment variable, it will remain overridden to the end of the code block. + +```bash +echo $OVERRIDE_VARIABLE +export OVERRIDE_VARIABLE=overridden +echo $OVERRIDE_VARIABLE +``` + +Context is not shared between code blocks (even in the same step). Each code block gets their own exectution environment. + +```bash +echo $OVERRIDE_VARIABLE +``` + + + +# Navigation + +* Back to [I/O Validation](io.md) +* On to [Working Directory](working_dir.md) diff --git a/examples/io.md b/examples/io.md new file mode 100644 index 0000000..6b85990 --- /dev/null +++ b/examples/io.md @@ -0,0 +1,60 @@ +# I/O Validation + +This is an example markdown file with an annotated test command. + +To see a summary of what commands will be run: + +```bash +mm.py -d io.md +``` + +To run this file and validate the expected output: + +```bash +mm.py io.md +``` + +Be sure to checkout the raw version of this file to see the annotations. + +This is an annotated command. When the ```mm.py``` utility is run, the following code block will be executed: + + + +```bash +echo "test" +``` + + + +You can run multiple commands within the same block, and validate stderr as well. + +> **Note:** The ```expected_stdout_lines``` and ```expected_stderr_lines``` directives ignore output that doesn't match an expected line. Validation will fail only if expected lines do not appear in stdout/stderr. You can have extra output and still pass validation. + + + +```bash +echo "test" +echo "another_output_line" +echo "test2" + +echo "error" 1>&2 +``` + + + +# Navigation + +* Back to [README](README.md) +* On to [Environment Variables](env.md) + diff --git a/examples/start.md b/examples/start.md deleted file mode 100644 index 80edcbe..0000000 --- a/examples/start.md +++ /dev/null @@ -1,20 +0,0 @@ -# Quick Start Example - -This is an example markdown file with an annotated test command - - - -```bash -echo "test" -``` - - - -This unannotated command will not be run: -```bash -echo "A command that will not be run" -``` diff --git a/examples/working_dir.md b/examples/working_dir.md new file mode 100644 index 0000000..5f7f5ee --- /dev/null +++ b/examples/working_dir.md @@ -0,0 +1,81 @@ +# Working directory example + +## Using this example +To see a summary of what commands will be run: + +```bash +mm.py -d env.md +``` + +To run this file and validate the expected output: + +```bash +mm.py +``` + +Be sure to checkout the raw version of this file to see the annotations. + +> **Note:** This example requires that the current working directory be writable + +## Changing the working directory + +The current working directory of the scripts will default to whatever the working directory the mm.py utility was called with. You can always change the working directory using a shell command, but as with setting environment variables, it will only remain to the end of the code block: + + + +```bash +pwd +mkdir working_dir_test +cd working_dir_test +pwd +echo "file contents" > test_file +cat test_file +``` + +The next code block will revert back to the default working directory: + +```bash +pwd +cat test_file || echo "File Not Found" +``` + + + +Adding the ```working_dir``` directive to your step annotations will change the default working directory for the duration of the step. + + + +```bash +pwd +cat test_file || echo "File Not Found" +cd .. +pwd +``` + +Other code blocks will revert back to the custom default within this step: + +```bash +pwd +cat test_file || echo "File Not Found" +rm -v test_file +cd .. +rmdir working_dir_test +``` + + + +# Navigation + +* Back to [Environment Variables](env.md) +* On to [Sleeping, Timeouts, and Backgrounding](background.md) diff --git a/tox.ini b/tox.ini index 515ce18..77d5398 100644 --- a/tox.ini +++ b/tox.ini @@ -14,10 +14,19 @@ commands = coverage run -m unittest discover -v ./tests coverage xml python3 setup.py install - mm.py examples/start.md commands_pre = pip3 install -e {toxinidir}/ +[testenv:examples] +changedir=./examples/ +commands = + mm.py README.md + mm.py io.md + mm.py background.md + mm.py env.md + mm.py working_dir.md + + [testenv:flake8] basepython = python3.7 usedevelop = False