updated documentation

docs/source/basic.rst

@@ -4,18 +4,22 @@
 basic
 =====
 
-Please run this expample using 
+This is by far the simplest example on how to get started with HoverPy. Please run this example using: 
 
-``env PYTHONPATH=.:${PYTHONPATH} python examples/basic/basic.py``
+``$ env PYTHONPATH=.:${PYTHONPATH} python examples/basic/basic.py``
 
-This is by far the simplest example on how to get started with HoverPy. Let's import our most important class HoverPy, along with whatever else we may need 
+You should see your IP address show up twice. Let's walk through the code to see what's happening. 
 
 ::
 
 >>> from hoverpy import HoverPy
 >>> import requests
 
 
+Above, we start by importing our most important class `HoverPy`. We also bring in ``requests`` for our http traffic.
+
+.. hoverpy: hoverpy.html#module-hoverpy
+
 Now let's create our HoverPy object in capture mode. We do so with a `with` statement as this is the pythonic way, although this is not a necessity. 
 
 ::

docs/source/index.rst

@@ -10,6 +10,7 @@ Contents
    :maxdepth: 3
 
    installation
+   introduction
    usage
 
 ==================

docs/source/installation.rst

@@ -71,3 +71,13 @@ You can also install HoverPy from PIP, however once again you're better off play
 .. code:: bash
 
     $ pip install --user -i https://testpypi.python.org/pypi hoverpy
+
+HoverFly binary
+---------------
+
+Please note that when you install HoverPy, the HoverFly binaries get downloaded and installed in your home directory, in
+
+.. code:: bash
+
+    ${home}/hoverfly/bin/dist_vX.X.X/${OS}_${ARCH}/hoverfly
+

docs/source/introduction.rst

@@ -0,0 +1,203 @@
+Introduction
+============
+
+Building and testing interdependent applications is difficult. Maybe
+you’re building a mobile application that needs to talk to a legacy API.
+Or a microservice that relies on two other services that are still in
+development The problem is the same: how do you develop and test against
+external dependencies which you cannot control?
+
+You could use mocking libraries as substitutes for external
+dependencies. But mocks are intrusive, and do not test all the way to
+the architectural boundary of your application. This means the client
+code for your external dependency is substituted and not tested.
+
+Stub services are better, but they often involve too much configuration
+or may not be transparent to your application. Then there is the problem
+of managing test data. Often, to write proper tests, you need
+fine-grained control over the data in your mocks or stubs. Managing test
+data across large projects with multiple teams introduces bottlenecks
+that impact delivery times.
+
+Integration testing “over the wire” is problematic too. When stubs or
+mocks are substituted for real services (in a continuous integration
+environment for example) new variables are introduced. Network latency
+and random outages can cause integration tests to fail unexpectedly.
+
+Service Virtualisation is software that records the interactions between
+*you*, and the *big unpredictible world*.
+
+.. figure:: mermaid/intro/graph1.png
+   :alt: hoverpy intro image1
+
+A very sturdy software solution for Service Virtualisation is
+`Mirage <https://github.com/SpectoLabs/mirage>`__, which is used
+extensively in the airline industry. Its successor,
+`HoverFly <http://hoverfly.io>`__, has taken all the lessons learned in
+the years of use of Mirage. Both Mirage and Hoverfly are open source
+software, developed at `specto.io <http://spectio.io>`__.
+
+HoverPy is the thin layer between Python and HoverFly. HoverFly is a
+light-weight and extremely fast proxy written in Go, and does the heavy
+lifting for HoverPy. So a more accurate picture might be:
+
+.. figure:: mermaid/intro/graph2.png
+   :alt: hoverpy intro image2
+
+--------------
+
+Feature overview
+----------------
+
+-  “Capture” traffic between a client and a server application
+-  Use captured traffic to simulate the server application
+-  Export captured service data as a JSON file
+-  Import service data JSON files
+-  Simulate latency by specifying delays which can be applied to
+   individual URLs based on regex patterns, or based on HTTP method
+-  Flexible request matching using templates
+-  Supports “middleware” (which can be written in any language) to
+   manipulate data in requests or responses, or to simulate unexpected
+   behaviour such as malformed responses or random errors
+-  Supports local or remote middleware execution (for example on AWS
+   Lambda)
+-  Uses BoltDB to persist data in a binary file on disk - so no
+   additional database is required
+-  REST API
+-  Run as a transparent proxy or as a webserver
+-  High performance with minimal overhead
+-  JUnit rule “wrapper” is available as a Maven dependency
+-  Supports HTTPS and can generate certificates if required
+-  Authentication (combination of Basic Auth and JWT)
+-  Admin UI to change state and view basic metrics
+
+Use cases
+---------
+
+Hoverfly is designed to cater for two high-level use cases. Capturing
+real HTTP(S) traffic between an application and an external service for
+re-use in testing or development.
+
+If the external service you want to simulate already exists, you can put
+Hoverfly in between your client application and the external service.
+Hoverfly can then capture every request from the client application and
+every matching response from the external service (capture mode).
+
+These request/response pairs are persisted in Hoverfly, and can be
+exported to a service data JSON file. The service data file can be
+stored elsewhere (a Git repository, for example), modified as required,
+then imported back into Hoverfly (or into another Hoverfly instance).
+
+Hoverfly can then act as a “surrogate” for the external service,
+returning a matched response for every request it received (simulate
+mode). This is useful if you want to create a portable, self-contained
+version of an external service to develop and test against.
+
+This could allow you to get around the problem of rate-limiting (which
+can be frustrating when working with a public API) You can write
+Hoverfly extensions to manipulate the data in pre-recorded responses, or
+to simulate network latency.
+
+You could work while offline, or you could speed up your workflow by
+replacing a slow dependency with a fast Hoverfly “surrogate”.
+
+Creating simulated services for use in a testing or development.
+----------------------------------------------------------------
+
+In some cases, the external service you want to simulate might not exist
+yet. You can create service simulations by writing service data JSON
+files. This is in line with the principle of design by contract
+development. Service data files can be created by each developer, then
+stored in a Git repository. Other developers can then import the service
+data directly from the repository URL, providing them with a Hoverfly
+“surrogate” to work with. Instead of writing a service data file, you
+could write a “middleware” script for Hoverfly that generates a response
+“on the fly”, based on the request it receives (synthesize mode). More
+information on this use-case is available here: Synthetic service
+example Easy API simulation with the Hoverfly JUnit rule Proceed to the
+“Modes” and middleware section to understand how Hoverfly is used in
+these contexts.
+
+--------------
+
+Modes and middleware
+--------------------
+
+Hoverfly modes
+~~~~~~~~~~~~~~
+
+Hoverfly has four modes. Detailed guides on how to use these modes are
+available in the Usage section.
+
+Capture mode
+^^^^^^^^^^^^
+
+.. figure:: mermaid/intro/graph3.png
+   :alt: hoverpy intro image3
+
+
+In this mode, Hoverfly acts as a proxy between the client application
+and the external service. It transparently intercepts and stores
+outgoing requests from the client and matching incoming responses from
+the external service. This is how you capture real traffic for use in
+development or testing.
+
+Simulate mode
+^^^^^^^^^^^^^
+
+.. figure:: mermaid/intro/graph4.png
+   :alt: hoverpy intro image4
+
+In this mode, Hoverfly uses either previously captured traffic, or
+imported service data files to mimic the external service. This is
+useful if you are developing or testing an application that needs to
+talk to an external service that you don’t have reliable access to. You
+can use the Hoverfly “surrogate” instead of the real service.
+
+Synthesize mode
+^^^^^^^^^^^^^^^
+
+.. figure:: mermaid/intro/graph5.png
+   :alt: hoverpy intro image5
+
+In this mode, Hoverfly doesn’t use any stored request/response pairs.
+Instead, it generates responses to incoming requests on the fly and
+returns them to the client. This mode is dependent on middleware (see
+below) to generate the responses.
+
+This is useful if you can’t (or don’t want to) capture real traffic, or
+if you don’t want to write service data files.
+
+Modify mode
+^^^^^^^^^^^
+
+.. figure:: mermaid/intro/graph6.png
+   :alt: hoverpy intro image6
+
+In this mode, Hoverfly passes requests through from to the server, and
+passes the responses back. However, it also executes middleware on the
+requests and responses. This is useful for all kinds of things such as
+manipulating the data in requests and/or responses on the fly.
+
+Middleware
+----------
+
+Middleware can be written in any language, as long as that language is
+supported by the Hoverfly host. For example, you could write middleware
+in Go, Python or JavaScript (if you have Go, Python or NodeJS installed
+on the Hoverfly host, respectively).
+
+Middleware is applied to the requests and/or the responses depending on
+the mode:
+
+-  Capture Mode: middleware affects only outgoing requests
+-  Simulate Mode: middleware affects only responses (cache contents
+   remain untouched)
+-  Synthesize Mode: middleware creates responses
+-  Modify Mode: middleware affects requests and responses
+-  Middleware can be used to do many useful things, such as simulating
+   network latency or failure, rate limits or controlling data in
+   requests and responses.
+
+A detailed guide on how to use middleware is available in the Usage
+section.

docs/source/mermaid/intro/graph1

@@ -0,0 +1,10 @@
+sequenceDiagram
+    participant You
+    participant Service Virtualisation
+    participant The World
+    Service Virtualisation-->> The World: request
+    loop every day
+        You->>Service Virtualisation: request
+        Service Virtualisation-->>You: response
+    end
+    The World -->> Service Virtualisation: "sometimes.."

docs/source/mermaid/intro/graph2

@@ -0,0 +1,12 @@
+sequenceDiagram
+    participant You
+    participant HoverPy
+    participant HoverFly
+    participant The World
+    HoverPy-->> HoverFly: Setup
+    HoverFly-->> The World: request
+    loop every day
+        You->>HoverFly: request
+        HoverFly-->>You: response
+    end
+    The World -->> HoverFly: "if I feel like it.."

docs/source/mermaid/intro/graph3

@@ -0,0 +1,8 @@
+sequenceDiagram
+    participant You
+    participant HoverPy
+    participant The World
+    loop proxied
+        You->>The World: interaction
+        The World-->>You: response
+    end

docs/source/mermaid/intro/graph4

@@ -0,0 +1,8 @@
+sequenceDiagram
+    participant You
+    participant HoverPy
+    participant The World
+    loop
+        You->>HoverPy: interaction
+        HoverPy-->>You: response
+    end

docs/source/mermaid/intro/graph5

@@ -0,0 +1,7 @@
+sequenceDiagram
+    participant You
+    participant HoverPy
+    loop synth.py
+        You->>HoverPy: interaction
+        HoverPy-->>You: response
+    end

docs/source/mermaid/intro/graph6

@@ -0,0 +1,8 @@
+sequenceDiagram
+    participant you
+    participant hoverpy
+    participant the world
+    loop modify.py
+        you->>the world: interaction
+        the world-->>you: response
+    end

docs/source/modify.rst

@@ -15,15 +15,15 @@ Let's look into mutating responses using middleware. This is particularly useful
 >>>         middleware="python examples/modify/modify_payload.py") as hoverpy:
 
 
-Above we created our HoverPy object with modify and middleware enabled. Please note this brings in ```python examples/modify/modify_payload.py``` which will get run on every request. 
+Above we created our HoverPy object with modify and middleware enabled. Please note this brings in ``python examples/modify/modify_payload.py`` which will get run on every request. 
 
 ::
 
 >>>     for i in range(30):
 >>>         r = requests.get("http://time.jsontest.com")
 
 
-Let's make 30 requests to time.jsontest.com which simply gets us the current local time 
+Let's make 30 requests to http://time.jsontest.com which simply gets us the current local time 
 
 ::
 
@@ -33,7 +33,7 @@ Let's make 30 requests to time.jsontest.com which simply gets us the current loc
 >>>                 r.json()["time"])
 
 
-Time ``time`` key is inside the response, which is what we expected. 
+The ``time`` key is inside the response, which is what we expected. 
 
 ::
 

examples/basic/basic.py

@@ -1,10 +1,14 @@
-# Please run this expample using <br><br>``env PYTHONPATH=.:${PYTHONPATH} python examples/basic/basic.py``<br><br>
 # This is by far the simplest example on how to get started with HoverPy.
-# Let's import our most important class HoverPy, along with whatever else
-# we may need
+# Please run this example using: <br><br>``$ env PYTHONPATH=.:${PYTHONPATH} python examples/basic/basic.py``<br><br>
+# You should see your IP address show up twice. Let's walk through the
+# code to see what's happening.
 from hoverpy import HoverPy
 import requests
 
+# Above, we start by importing our most important class `HoverPy`. We also
+# bring in ``requests`` for our http traffic.<br><br>
+# .. hoverpy: hoverpy.html#module-hoverpy<br><br>
+
 # Now let's create our HoverPy object in capture mode. We do so with a
 # `with` statement as this is the pythonic way, although this is not a necessity.
 with HoverPy(capture=True) as hoverpy:

examples/modify/modify.py

@@ -8,21 +8,21 @@
         middleware="python examples/modify/modify_payload.py") as hoverpy:
 
     # Above we created our HoverPy object with modify and middleware enabled.
-    # Please note this brings in ```python examples/modify/modify_payload.py```
+    # Please note this brings in ``python examples/modify/modify_payload.py``
     # which will get run on every request.
 
     for i in range(30):
         r = requests.get("http://time.jsontest.com")
 
-        # let's make 30 requests to time.jsontest.com which simply gets us the
+        # let's make 30 requests to http://time.jsontest.com which simply gets us the
         # current local time
 
         if "time" in r.json().keys():
             print(
                 "response successfully modified, current date is " +
                 r.json()["time"])
 
-        # time ``time`` key is inside the response, which is what we expected.
+        # The ``time`` key is inside the response, which is what we expected.
 
         else:
             print("something went wrong - deal with it gracefully")