Human Framework: Test Automation Framework for Humans™
NOTICE: Human Framework is still in the Proof-of-Concept stage
CONTRIBUTE! I am releasing Human Framework under GPLv3 which makes it open source and free.
- Localization of training data (translation)
- Adding more intents and actions
- Reporting bugs
- Suggesting new features
NOTICE: I am working on a Rasa NLU support
Human Framework is a test automation framework designed for testers without software programming background by allowing them to write test steps or instructions in their own language like English.
Human Framework uses Natural Language Understanding (NLU) for intent classification and entity extraction to be able to perform specific actions. It currently depend on Microsoft LUIS (Language Understanding).
Below is an example of test instructions that Human Framework can understand.
To get an idea of what specific sentences are supported, check the the content of
Open chrome browser https://devpost.com Page title should be "Devpost - The home for hackathons" Page should contain link "About" Click link "About" Page title should be "About us · Devpost" Close browser
LUIS (and other NLU tools) makes it possible to write test cases on any human language. Current version of Human Framework only supports English.
Human Framework was inspired by Robot Framework but leans towards Natural Language Processing (NLP) for writing test cases.
"Human Framework is the Siri or Alexa of automation testing." (minus the voice, but it is possible in the future)
How it works
The architecture of Human Framework can be simplified into 2 steps- Intent Classification and Action Execution:
- Human Framework reads the test case files and then uses an Intent Classifier (e.g. LUIS.ai) to identify intents and entities. For example, the sentence "open chrome browser" will be identified with the intent "web.open_browser" and entity "chrome" as the browser type.
- Human Framework then executes the action specific for the classified intent. For example, the intent "web.open_browser" will use Selenium to automate opening a chrome browser.
Usage / Installation
Create an Azure account
Go to Azure portal and create an account.
On the Azure portal dashboard, click
Create a resource, look for
Cognitive Servicesand click
Fill up form and click
Create a LUIS.ai account
Download the Human Framework training data from https://raw.githubusercontent.com/roniemartinez/HumanFramework/master/train/en.json and save it with
Go to LUIS.ai and create an account
Import new app
Choose app file (JSON format) ...and select the downloaded Human Framework training data
Optionally, type your desired app name
Click on the application you created.
On the application dashboard, click
Trainto train your LUIS.ai app
Connecting Azure account to LUIS.ai and publishing
Add prediction resource
Productionand then click the
Go back to
Azure Resources. If you published your application to
Change query parametersto switch between environments.
Copy the endpoint assigned resource in #2
NOTE: After copying the endpoint, enter it to a browser's address bar to make sure that it is accessible., Otherwise, repeat the steps or use the
Starter_Key which will only give you 1000 API calls
Installing Human Framework
Human Framework is written in Python 3. Start by downloading/installing Python from python.org.
To install Human Framework, enter the following command into the terminal/cmd:
Directly from Github
pip install -e firstname.lastname@example.org:roniemartinez/HumanFramework.git#egg=humanframework
pip install humanframework
Setting up Human Framework
Use the copied endpoint from LUIS.ai and setup Human Framework. Paste the endpoint enclosed in double quotes.
human config --luis-endpoint "<endpoint>"
Write a text file (test_*.txt) containing your tests and save it to local working directory where you placed the Selenium Drivers.
For example, if we have a file named
open chrome https://python.org page title should be "Welcome to Python.org" close browser
To run the tests in
test_web.txt, enter the following on the terminal/cmd:
human --test test_web.txt
Files inside the
trials folder that starts with the text
test_ are automatically executed.
trials |- test_simple.txt |- test_simple_2.txt |- ...
With this structure, you can simply type
human to run the test cases.
The future of Human Framework
- Integrated Editor - While testing was greatly simplified because Human Framework will just execute sentences, using the terminal to install Python and several libraries is still more developer-centric. To remove this technical requirement, another major goal of the Human Framework project is to release a one-click installer which should include new features like an integrated IDE/Editor. This will greatly increase tester's productivity.
- Language Support - Not all testers use the English language. Supporting most languages will benefit more testers around the world.
- NLU Options - While LUIS.ai is very simple to use and accurate compared to other Intent Classifiers, it is not cheap. To support more testers, a free option like Rasa NLU can be used as an alternative.
I only implemented a few intents and actions for the proof-of-concept stage.
To know what are the possible tests that you can do with Human Framework, check the contents of