Demonstrate an MVP for the reliability framework, created for PyCon Canada 2015 talk
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

Python Reliability Testing Pipeline Demo

The packaged iOS app brings up on safari when the launch button is clicked. The web page has a button app-switch with an on click handler. When clicked, it updates Browser with url appiumdemo:// which is an Url scheme that the ios app registered so on iOS simulator it will bring back the app.

System requirements

  • XCode
  • Node
  • Appium
  • Python and Pip
  • ElasticSearch
  • Kibana
  • Homebrew (OSX)
  • Java Runtime (for Elasticsearch)


(We strongly recommend that you set up a virtualenv for this project, and you may also want to check out virtualenvwrapper for convenience)


  1. $ sudo pip install virtualenv virtualenvwrapper
  2. add source /usr/local/bin/ to you your .profile
  3. create a testing environment: $ mkvitualenv reliability-demo
  4. Switch to using that environment: $ workon reliability-demo

Automate app using Appium


Start Appium server

    appium --help # Check that Appium is installed and available in your path

Check out v0.1.0 tag

    git clone
    cd reliability-demo/
    git checkout v0.1.0

Install requirements using pip:

    pip install -r requirements.txt

Then run the script


Describe Tests using Behave

Check out v0.2.0 of the repo

    git checkout v0.2.0

Install requirements

    pip install -r requirements.txt

Check that Appium server is still running. Restart may be necessary.

Run behave from the root of the repo

    behave --help # Check that Appium is installed and available in your path

Publish JSON formatted results of behave tests

    export PYTHONPATH=$PYTHONPATH:.; behave --no-capture  -f record --outfile=data/results.json -f pretty

Storage and Visualization using ElasticSearch and Kibana

Install ElasticSearch

    # install via Homebrew and check version
    brew install elasticsearch && brew info elasticsearch
    # Start ElasticSearch as a service
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plist

Then check ElasticSearch is running

    curl -XGET 'http://localhost:9200'

Install Kibana

    # install via homebrew and check version
    brew install kibana && brew info kibana
    # Start Kibana as a service
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.kibana.plist

Then navigate to http://localhost:5601/ in your browser to make sure that Kibana is running.

Check out v0.3.0 of the repo

    pip install -r requirements.txt

Set up Elasticsearch index

python db/

Creates an ElasticSearch Index named reliability

Set up ElasticSearch mapping

python db/

Creates an ElasticSearch Mapping named behave under the index reliability

Publish JSON formatted results of behave tests to ElasticSearch

$ python db/ 
4/4 test results submitted successfully.

Navigate back to http://localhost:5601/ and click on the discover tag to view the submitted data!