Added userguide and examples

.github/workflows/build.yaml

@@ -34,6 +34,9 @@ jobs:
     - name: Run unit-tests
       run: |
         tox -e py37
+    - name: Run examples
+      run: |
+        tox -e examples
     - name: Upload test coverage
       uses: codecov/codecov-action@v1
     - name: Build and publish mechanical-markdown

examples/README.md

@@ -1,47 +1,174 @@
 
-# Quick Start
+# Mechanical markdown by example
 
-Once you have the package installed, if you want to try out some simple examples, navigate to the examples/ directory and have a look at start.md:
+## Prerequisites
 
-    # Quick Start Example
+Be sure you have mechanical markdown [installed](../README.md#installing) and that the mm.py utility is in your $PATH. These examples were written with basic bash commands in mind, but any bash like shell should work. See [Shells](#shells) below for alternatives.
+
+## Using this guide
+
+All markdown files in examples/ (including this one!) are annotetated and can be executed and validated with:
+
+```bash
+mm.py filename.md
+```
+
+This guide automatically executed as part of this project's continuous integration pipeline. It serves as both user guide and integration test suite for this package. If you use it run through it, hopefully you will see what a powerful concept self executing documentation can be.
+
+## How mechanical-markdown works
+
+One of the beautiful features of markdown is that it is both human and machine readable. A human can read a user guide and copy paste steps into their terminal. A machine can do the same and do some extra validation on the steps to make sure they executed correctly. Also, most markdown engines support embedded HTML comments ```<!-- -->```. We can use these HTML comments to embed extra information to tell our validation program what output we expect from a command. The ```mm.py``` utility will automatically extract this information, allong with the commands to execute from our markdown files.
+
+## Annotation format
+
+To tell mechanical-markdown what parts of your document need to be executed as code, you must add an HTML comment that begins with the token ```STEP```. After this token, mechanical-markdown will interpret the rest of the comment as a yaml document with instructions for how the code blocks should be executed and verified. All fields in this yaml document are optional. Finish the comment to denote then end of the yaml document and the beginning of your exectuable markdown code. Finally, finish with an html comment like this: ```<!-- END_STEP -->``` to denote the end of a step. Let's look at a basic example:
 
-    This is an example markdown file with an annotated test command
-
     <!-- STEP 
-    name: First Step
+    name: Hello World
     expected_stdout_lines:
-      - "test"
+      - "Hello World!"
     -->
-
+
+    You can use regular markdown anywhere during a step. It will be ignored. Only denoted as bash or sh will be executed.
+
     ```bash
-    echo "test"
+    echo "Hello World!"
     ```
-
+
+    This python code block will not be executed:
+
+    ```python
+    print("This python script will not be run")
+    ```
+        
     <!-- END_STEP -->
-
+    
     This unannotated command will not be run:
+
     ```bash
     echo "A command that will not be run"
     ```
 
+Here's how the above will render in your markdown interpreter.
+
+----
+
+<!-- STEP 
+name: Hello World
+expected_stdout_lines:
+  - "Hello World!"
+-->
+
+You can use regular markdown anywhere during a step. It will be ignored. Only code blocks denoted as bash or sh will be executed.
+
+```bash
+echo "Hello World!"
+```
+
+This python code block will not be executed:
+
+```python
+print("This python script will not be run")
+```
+
+<!-- END_STEP -->
+
+This unannotated command will not be run:
+
+```bash
+echo "A command that will not be run"
+```
+
+----
+
+Let's breakdown what this embedded yaml annotation is doing. There are two fields in our yaml document ```name``` and ```expected_stdout_lines```. The name field simply provides a name for the step that will be printed to the report that mm.py generates. The expected_stdout_lines field is actually telling mm.py what it should be looking for from stdout when it executes our code block(s). For more on this, checkout [io.md](io.md). 
+
+
+Code blocks must be tagged as either "bash" or "sh". Code from other languages will be ignored.
+
+
+<!-- END_STEP -->
+
+
+## CLI
+
+### Help
+For a list of options:
+
+<!-- STEP 
+name: CLI help
+expected_stdout_lines:
+  - "usage: mm.py [-h] [--dry-run] [--manual] [--shell SHELL_CMD] markdown_file"
+  - "Auto validate markdown documentation"
+  - "positional arguments:"
+  - "  markdown_file"
+  - "optional arguments:"
+  - "  -h, --help            show this help message and exit"
+  - "  --dry-run, -d         Print out the commands we would run based on"
+  - "                        markdown_file"
+  - "  --manual, -m          If your markdown_file contains manual validation"
+  - "                        steps, pause for user input"
+  - "  --shell SHELL_CMD, -s SHELL_CMD"
+  - "                        Specify a different shell to use"
+-->
+
+```bash
+mm.py --help
+```
+
+<!-- END_STEP -->
+
+```
+usage: mm.py [-h] [--dry-run] [--manual] [--shell SHELL_CMD] markdown_file
+
+Auto validate markdown documentation
+
+positional arguments:
+optional arguments:
+  -h, --help            show this help message and exit
+  --dry-run, -d         Print out the commands we would run based on
+			markdown_file
+  --manual, -m          If your markdown_file contains manual validation
+			steps, pause for user input
+  --shell SHELL_CMD, -s SHELL_CMD
+			Specify a different shell to use
+```
+
+### Dry Run
 You can do a dry run to print out exactly what commands will be run using the '-d' flag.
 
+<!-- STEP 
+name: CLI dry run
+expected_stdout_lines:
+  - "Would run the following validation steps:"
+  - "Step: Hello World"
+  - "Step: CLI help"
+  - "Step: CLI dry run"
+  - "Step: Pause for manual validation"
+-->
+
 ```bash
-mm.py -d start.md
+mm.py -d README.md
 ```
-You'll see output like the following:
+
+<!-- END_STEP -->
+
+This will print out all the steps that would be run, without actually running them. Output looks something like this
 
 ```
 Would run the following validation steps:
-Step: First Step
+Step: Hello World
 	commands to run with 'bash -c':
-		`echo "test"`
+		`echo "Hello World!"`
 	Expected stdout:
-		test
+		Hello World!
 	Expected stderr:
 
+...
 ```
 
+### Run and Validate
+
 Now you can run the steps and verify the output:
 
 ```bash
@@ -51,18 +178,54 @@ mm.py start.md
 The script will parse the markdown, execute the annotated commands, and then print a report like this:
 
 ```
-Running shell 'bash -c' with command: `echo "test"`
-Step: First Step
-	command: `echo "test"`
+Running shell 'bash -c' with command: `echo "Hello World!"`
+Running shell 'bash -c' with command: `mm.py --help`
+Running shell 'bash -c' with command: `mm.py -d README.md`
+
+Step: Hello World
+	command: `echo "Hello World!"`
 	return_code: 0
 	Expected stdout:
-		test
+		Hello World!
 	Actual stdout:
-		test
+		Hello World!
 		
 	Expected stderr:
 	Actual stderr:
+...
 ```
 
 If anything unexpected happens, you will get report
 of what went wrong, and mm.py will return non-zero.
+
+### Shells
+
+The default shell used to execute scripts is ```bash -c```. You can use a different shell interpreter by specifying one via the cli:
+
+```bash
+mm.py -s 'zsh -c' README.md
+```
+
+### Manual validation
+
+You can add manual validation steps to your document. A manual validation step is just a pause message to allow the user to take some manual step like opening a browser. These steps normally get ignored, as ```mm.py``` is designed to do automated validation by default. If you run the following, it will enable mm.py to pause for user input. (View raw markdown for an example of what a manual_pause_message looks like):
+
+
+<!-- STEP
+name: Pause for manual validation
+manual_pause_message: "Waiting for user input"
+-->
+
+<!-- We will pause here and print the above message when mm.py is run with '-m'. Otherwise, this step does nothing -->
+
+<!-- END_STEP -->
+
+```bash
+mm.py -m README.md 
+```
+
+# More examples:
+
+- For more details on checking stdout/stderr: [I/O Validation](io.md)
+- For more details on setting up the execution environment: [Environment Variables](env.md) and [Working Directory](working_dir.md)
+- For controlling timeouts, backgrounding, and adding delay between steps: [Sleeping, Timeouts, and Backgrounding](background.md)

examples/background.md

@@ -0,0 +1,110 @@
+# Sleeping, Timeouts, and Backgrounding
+
+## Using this example
+To see a summary of what commands will be run:
+
+```bash
+mm.py -d env.md
+```
+
+To run this file and validate the expected output:
+
+```bash
+mm.py
+```
+
+Be sure to checkout the raw version of this file to see the annotations.
+
+## Sleeping
+
+Sometimes when running a series of steps automatically, they will run much faster than a human executing steps manually. If this leads to trouble for your procedures, you can insert a delay after running a command by using the ```sleep``` directive.
+
+This first date command has a 5 second sleep:
+
+<!-- STEP 
+name: Step with a sleep
+sleep: 5
+expected_stdout_lines:
+  - First step
+-->
+
+```bash
+date
+echo "First step"
+```
+
+<!-- END_STEP -->
+
+You'll see at least a 5 second delay before this sleep gets executed:
+
+<!-- STEP 
+name: Delayed step
+expected_stdout_lines:
+  - Second step
+-->
+
+```bash
+date
+echo "Second step"
+```
+
+<!-- END_STEP -->
+
+## Backgrounding
+
+Conversely, if you want to run a step in the background without waiting to move on to the next step. This is great for starting services or daemons that you will clean up later on in the procedure. All backgrounded steps will be waited for at the end of execution so that stdout and stderr and the return code can all be checked. If a processes hasn't finished it will be timed out after 60s by default. See Timeout section below for more info.
+
+In this first step, run a command that will take 10 seconds, but don't wait for it
+
+<!-- STEP 
+name: Backgrounded step
+background: true
+expected_stdout_lines:
+  - Background step
+-->
+
+```bash
+sleep 5 && echo "Background step"
+date
+```
+
+<!-- END_STEP -->
+
+The next step will be exeuted right away, and the background step will be joined after all non-backgrounded steps have completed.
+
+<!-- STEP 
+name: Foreground step
+expected_stdout_lines:
+  - Foreground step
+-->
+
+```bash
+echo "Foreground step"
+date
+```
+
+<!-- END_STEP -->
+
+## Timeouts
+
+By default, all commands timeout after 60s. They will receive a SIGTERM, followed by a SIGKILL. Script that reach their timeout and are killed will cause validation to fail and mm.py will return non-zero. You can change the duration of the timeout for an individual step by setting ```timeout_seconds``` .
+
+> **Note:** sleep time does not count towards timeout_seconds.
+
+<!-- STEP 
+name: Timeout step
+timeout_seconds: 5
+expected_stdout_lines:
+  - Timeout step
+-->
+
+```bash
+echo "Timeout step"
+date
+```
+
+<!-- END_STEP -->
+
+# Navigation
+
+Back to [Working Directory](working_dir.md)