Initial working python module with tests

.codecov.yml

@@ -0,0 +1,10 @@
+coverage:
+  status:
+    project:
+      default:
+        # basic
+        target: auto
+        threshold: 0%
+
+ignore:
+  - tests      # - tests

.github/workflows/build.yaml

@@ -0,0 +1,43 @@
+name: mechanical-markdown
+
+on:
+  push:
+    branches:
+      - main
+    tags:
+      - v*
+  pull_request:
+    branches:
+      - main
+      - release-*
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    env:
+      PYTHON_VER: 3.7
+      TWINE_USERNAME: ${{ secrets.PYPI_UPLOAD_USER }}
+      TWINE_PASSWORD: ${{ secrets.PYPI_UPLOAD_PASS }}
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ env.PYTHON_VER }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ env.PYTHON_VER }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine tox
+    - name: Run Linter
+      run: |
+        tox -e flake8
+    - name: Run unit-tests
+      run: |
+        tox -e py37
+    - name: Upload test coverage
+      uses: codecov/codecov-action@v1
+    - name: Build and publish mechanical-markdown
+      if: github.event_name != 'pull_request'
+      run: |
+        python setup.py sdist bdist_wheel
+        twine upload dist/*

README.md

@@ -1 +1,76 @@
-# mechanical-markdown
+# Mechanical Markdown
+
+[![codecov](https://codecov.io/gh/wcs1only/mechanical-markdown/branch/main/graph/badge.svg)](https://codecov.io/gh/wcs1only/mechanical-markdown)
+
+If you are using markdown to create tutorials for your users, these markdown files will often be a series of shell commands that a user will copy and paste into their shell of choice, along with detailed text description of what each command is doing.
+
+If you are regularly releasing software and having to manually verify your tutorials by copy pasting commands into a terminal every time you create a release, this is the package for you.
+
+The mechanical-markdown package is a python library and corresponding shell script that allow you to run annotated markdown tutorials in an automated fashion. It will execute your markdown tutorials and verify the ouput according to expected stdout/stderr that you can embed directly into your markdown tutorials. 
+
+# Installing 
+
+This package requires a working python3 environment. You can install it using pip:
+
+```bash
+pip install mechanical-markdown
+```
+
+This will install the python module, and create the ```mm.py``` CLI script.
+
+# Quick Start
+
+Check out the [examples](./examples) for some quick and easy examples.
+
+# Usage
+
+## CLI
+
+A command line utility called ```mm.py``` is included with this package.
+
+```bash
+% mm.py --help
+usage: mm.py [-h] [--dry-run] [--manual] [--shell SHELL_CMD] markdown_file
+
+Auto validate markdown documentation
+
+positional arguments:
+  markdown_file
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --dry-run, -d         Print out the commands we would run based on markdown_file
+  --manual, -m          If your markdown_file contains manual validation steps, pause for user input
+  --shell SHELL_CMD, -s SHELL_CMD
+                        Specify a different shell to use
+```
+
+## API
+
+Creating a MechanicalMarkdown instance from a string which contains a markdown document:
+```python
+from mechanical_markdown import MechanicalMarkdow
+
+mm = MechanicalMarkdown(markdown_string)
+```
+
+MechanicalMarkdown methods 
+
+```python
+# Returns a string describing the commands that would be run
+output = mm.dryrun(default_shell='bash -c')
+print(ouput)
+
+# Run the commands in the order they were specified and return a boolean for succes or failure
+# Also returns a report summarizing what was run and stdout/sterr for each command
+success, report = exectute_steps(manual, default_shell='bash -c')
+print(report)
+
+
+```
+
+# Contributing
+
+Issues and contributions are always welcome! Please make sure your submissions have appropriate unit tests (see tests/).
+
+This project was created to support [dapr/quickstarts](https://github.com/dapr/quickstarts). We're sharing it with the hope that it might be as usefull for somebody else as it was for us.

dev-requirements.txt

@@ -0,0 +1,7 @@
+setuptools
+mypy>=0.761
+mypy-extensions>=0.4.3
+flake8>=3.7.9 
+tox == 3.15.0
+coverage >= 5.3
+codecov >= 1.4.0

examples/README.md

@@ -0,0 +1,68 @@
+
+# Quick Start
+
+Once you have the package installed, if you want to try out some simple examples, navigate to the examples/ directory and have a look at start.md:
+
+    # Quick Start Example
+
+    This is an example markdown file with an annotated test command
+
+    <!-- STEP 
+    name: First Step
+    expected_stdout_lines:
+      - "test"
+    -->
+
+    ```bash
+    echo "test"
+    ```
+
+    <!-- END_STEP -->
+
+    This unannotated command will not be run:
+    ```bash
+    echo "A command that will not be run"
+    ```
+
+You can do a dry run to print out exactly what commands will be run using the '-d' flag.
+
+```bash
+mm.py -d start.md
+```
+You'll see output like the following:
+
+```
+Would run the following validation steps:
+Step: First Step
+	commands to run with 'bash -c':
+		`echo "test"`
+	Expected stdout:
+		test
+	Expected stderr:
+
+```
+
+Now you can run the steps and verify the output:
+
+```bash
+mm.py start.md
+```
+
+The script will parse the markdown, execute the annotated commands, and then print a report like this:
+
+```
+Running shell 'bash -c' with command: `echo "test"`
+Step: First Step
+	command: `echo "test"`
+	return_code: 0
+	Expected stdout:
+		test
+	Actual stdout:
+		test
+		
+	Expected stderr:
+	Actual stderr:
+```
+
+If anything unexpected happens, you will get report
+of what went wrong, and mm.py will return non-zero.

examples/start.md

@@ -0,0 +1,20 @@
+# Quick Start Example
+
+This is an example markdown file with an annotated test command
+
+<!-- STEP 
+name: First Step
+expected_stdout_lines:
+  - "test"
+-->
+
+```bash
+echo "test"
+```
+
+<!-- END_STEP -->
+
+This unannotated command will not be run:
+```bash
+echo "A command that will not be run"
+```

mechanical_markdown/__init__.py

@@ -0,0 +1,10 @@
+"""
+Copyright (c) Microsoft Corporation.
+Licensed under the MIT License.
+"""
+
+__version__ = '0.1.6'
+
+from mechanical_markdown.recipe import Recipe as MechanicalMarkdown
+
+__all__ = [MechanicalMarkdown]

mechanical_markdown/__main__.py

@@ -0,0 +1,40 @@
+from mechanical_markdown import MechanicalMarkdown
+
+import sys
+import argparse
+
+
+def main():
+    parse_args = argparse.ArgumentParser(description='Auto validate markdown documentation')
+    parse_args.add_argument('markdown_file', metavar='markdown_file', nargs=1, type=argparse.FileType('r'))
+    parse_args.add_argument('--dry-run', '-d',
+                            dest='dry_run',
+                            action='store_true',
+                            help='Print out the commands we would run based on markdown_file')
+    parse_args.add_argument('--manual', '-m',
+                            dest='manual',
+                            action='store_true',
+                            help='If your markdown_file contains manual validation steps, pause for user input')
+    parse_args.add_argument('--shell', '-s',
+                            dest='shell_cmd',
+                            default='bash -c',
+                            help='Specify a different shell to use')
+    args = parse_args.parse_args()
+
+    body = args.markdown_file[0].read()
+    r = MechanicalMarkdown(body)
+    success = True
+
+    if args.dry_run:
+        print("Would run the following validation steps:")
+        print(r.dryrun(args.shell_cmd))
+        sys.exit(0)
+
+    success, report = r.exectute_steps(args.manual, args.shell_cmd)
+    print(report)
+    if not success:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

mechanical_markdown/command.py

@@ -0,0 +1,43 @@
+
+"""
+Copyright (c) Microsoft Corporation.
+Licensed under the MIT License.
+"""
+
+import os
+import time
+
+from subprocess import Popen, PIPE, TimeoutExpired
+
+
+class Command:
+    def __init__(self, command_string):
+        self.command = command_string
+        self.process = None
+        self.return_code = -1
+        self.output = {'stdout': '', 'stderr': ''}
+
+    def wait_or_timeout(self, timeout):
+        if self.process is None:
+            return
+        try:
+            self.output['stdout'], self.output['stderr'] = self.process.communicate(timeout=timeout)
+        except TimeoutExpired:
+            self.process.terminate()
+            time.sleep(10)
+            self.process.kill()
+            try:
+                self.output['stdout'], self.output['stderr'] = self.process.communicate(timeout=timeout)
+            except TimeoutExpired:
+                pass
+
+        self.return_code = self.process.returncode
+
+    def run(self, cwd, env, shell):
+        args_list = shell.split()
+        args_list.append(self.command)
+        pwd = os.getcwd()
+        os.chdir(cwd)
+        print("Running shell '{}' with command: `{}`".format(shell, self.command))
+        self.process = Popen(args_list, stdout=PIPE, stderr=PIPE, universal_newlines=True, env=env)
+        os.chdir(pwd)

mechanical_markdown/parsers.py

@@ -0,0 +1,57 @@
+
+"""
+Copyright (c) Microsoft Corporation.
+Licensed under the MIT License.
+"""
+
+import yaml
+
+from html.parser import HTMLParser
+from mistune import Renderer
+from mechanical_markdown.step import Step
+
+start_token = 'STEP'
+end_token = 'END_STEP'
+
+
+class HTMLCommentParser(HTMLParser):
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.comment_text = ""
+
+    def handle_comment(self, comment):
+        self.comment_text += comment
+
+
+class RecipeParser(Renderer):
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.current_step = None
+        self.all_steps = []
+
+    def block_code(self, text, lang):
+        if lang is not None and lang.strip() in ('bash', 'sh') and self.current_step is not None:
+            self.current_step.add_command_block(text)
+        return ""
+
+    def block_html(self, text):
+        comment_parser = HTMLCommentParser()
+        comment_parser.feed(text)
+
+        comment_body = comment_parser.comment_text
+        if comment_body.find(end_token) >= 0:
+            if self.current_step is None:
+                return ""
+            self.all_steps.append(self.current_step)
+            self.current_step = None
+            return ""
+
+        start_pos = comment_body.find(start_token)
+
+        if start_pos < 0:
+            return ""
+
+        start_pos += len(start_token)
+        self.current_step = Step(yaml.safe_load(comment_body[start_pos:]))
+
+        return ""