Web Automation Markup Language.

Give WAML a try with the online WAML editor.

Here you will find the full WAML Specification, examples of what it looks like, and general information regarding the project.

The Web Automation Markup Language (WAML) defines a standard, programming language-agnostic specification for a sequence of steps which can be performed on web pages within the context of a web browser.

The WAML schema is based on JSON Schema draft-07 defined here.

waml: 0.1.0

steps:
  - visit: https://google.com
  - fill:
      selector: input#search
      value: Ugandan knuckles
  - click: a#search
  - wait: 10
  - snap:
      filename: test
      format: png

WAML documents are represented in either YAML or JSON formats. These documents may be written manually or generated dynamically from an application (such as a browser extension.) WAML is both human and machine-readable.

The WAML specification can be used to define flows for:

• Web scraping / data extraction
  • Content monitoring (Online review monitoring, competitor monitoring, price drops, new releases)
  • Content aggregation (Business profile data, leads, jobs, news, events)
  • Turning websites into APIs
• Web automation
  • Integration end-to-end testing
  • Uptime monitoring
  • Evaluating web best practices (SEO, HTML best practices)

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

• Terminology
• Full Example
• Fields
  • waml
  • Info
  • Variables
  • Context
  • Steps
• Variable Store
• Step Library
  • Visit
  • Click
  • Hover
  • Focus
  • Select
  • Fill
  • Type
  • Scroll
  • Press
  • Wait
  • Refresh
  • Back
  • Forward
  • Scrape
  • SetContext
  • Snap
• Flow Control

A WAML document represents a single Scenario, a sequence of Steps in a specified browser Context. Each Step corresponds to a single user action such as clicking a button, typing into a form, and so on.

A WAML document consists of many Fields.

Here is an example WAML document which exercises all possible fields and options.

waml: 0.1.0

info:
  title: An example WAML document.
  description: High-level summary of the scenario.

variables:
  someValue: Hello world
  isMobile: true

context:
  viewport:
    width: 800
    height: 600
  headers:
    X-API-Key: 123
    Authorization: Bearer abc
  cookies:
    - cookieName: cookieValue
  mediaType: screen
  userAgent: myUserAgent
  device: iPhone4
  timeout: 5

steps:
  - visit: https://google.com
  - click: a.link
  - hover: a#about
  - focus: input#search
  - select:
      selector: select#colors
      value: 'blue'
  - fill:
      selector: input#search
      value: Ugandan knuckles
  - type: ${someValue}
  - press: Shift
  - scroll: 100
  - scroll: a#load-more
  - scroll:
  - wait: div#results
  - wait: 3000
  - wait:
  - refresh:
  - back:
  - forward:
  - scrape:
      articles:
        - select article {0,}
        - body: select .body | get html
          imageUrl: select img | get attr src
          summary: select .body p:first-child | get text
          title: select .title | get text
      checked: select input[type="checkbox"] | get prop checked
  - setContext:
      viewport:
        height: 400
        width: 300
  - snap:
      filename: test
      format: png
  - pdf: test

The table below lists the 5 root-level fields of a WAML document.

All field names in the specification are case sensitive.

The WAML Specification is versioned using Semantic Versioning 2.0.0 (semver) and follows the semver specification.

A WAML document must begin with a version declaration within the top-level waml field. The waml field must contain the document's WAML Specification's semantic version number.

The waml field is required.

waml: 0.1.0

The Info field provides general information about the Scenario.

The info field is optional.

info: # Metadata about the WAML document
  title: An example WAML document.
  description: High-level summary of the user scenario.

The Variables field stores variables usable elsewhere in the Scenario, such as in the steps field. You can store any key-value pairs. Values must be a literal value (string, number, or boolean).

The use of variables and the variables field is optional.

variables: # A mapping of variables to be defined in the scenario context
  searchTerm: Hello world
  isMobile: true

You can reference any pre-defined variables with the ${variable} syntax.

steps:
  - fill:
      selector: input#search
      value: ${searchTerm}

Referencing a variable in steps that is not defined beforehand in the variables field SHOULD cause a WAML implementation to raise a validation error.

The Context field defines the browser's context and emulation configuration.

The context field is optional.

context: # Browser emulation settings (optional)
  viewport: # Browser window dimensions
    width: 800
    height: 600
  headers: # Additional HTTP headers to be sent with every request
    X-API-Key: 123
    Authorization: Bearer abc
  cookies: # Browser cookies
    - cookieName: cookieValue
  mediaType: screen # screen / print / none
  userAgent: myUserAgent # Browser user agent
  timeout: 5 # Maximum navigation time
  device: 'iPhone4'

The Steps field contains a sequence of one or more steps. Each step is a StepObject that represents a particular user action, such as clicking an HTML element.

The steps field is required.

steps: # A sequence of user interactions
  - visit: https://google.com # Navigating to a URL
  - fill: # Focus to an element and type things in
      selector: input#search
      value: Ugandan knuckles
  - click: a#search # Click an HTML element
  - wait # Wait for a page load to finish
  - snap: # Take screenshot of the page
      filename: foo
      format: png

Steps can have a compact version which omits default and optional parameters, and an expanded version with additional parameters & features. For example, the click step normally just needs a CSS selector:

Compact version

- click: a.link # Click an HTML element

But you can also use the expanded version of click to specify a target from a list of elements:

Expanded version

- click: # Expanded version of click
    selector: a.link
    index: 2 # click 3rd element from a list

The variable store is an ephemeral, mutable key-value store for reading and storing values to be used elsewhere in the Scenario. At the start of a Scenario, the variable store can be initialized through the variables field:

variables:
  someValue: Hello world
  isMobile: true

Variables can be read using the ${} syntax:

steps:
  - fill:
      selector: input#search
      value: ${someValue}

Steps such as scrape saves its results into the variable store.

- scrape:
    articles:
      - select article {0,}
      - body: select .body | get html
        imageUrl: select img | get attr src
        summary: select .body p:first-child | get text
        title: select .title | get text
    checked: select input[type="checkbox"] | get prop checked

At the end of a WAML Scenario, the contents of the variable store should be returned and the variable store cleared.

The table below lists the step types available in WAML:

Bolded properties are required properties.

The visit step navigates to a specified URL.

- visit: https://google.com

The click step clicks a specified HTML element by CSS selector. If there are multiple elements satisfying the selector, the first will be clicked.

- click: a.link # Clicks first matching element

- click:
    selector: ytd-thumbnail.ytd-video-renderer
    index: 2 # Clicks 3rd video from a list of elements

The hover step hovers the mouse pointer over a specified HTML element. If there are multiple elements satisfying the selector, the first will be hovered.

- hover: a#sign-in

The focus step focuses over a specified HTML element. If there are multiple elements satisfying the selector, the first will be focused.

- focus: input#search

The select step focuses over a specified dropdown element and selects one or more values.

If the <select> HTML element has the multiple attribute, all values are considered, otherwise only the first one is taken into account.

- select:
    selector: select#colors
    value: 'blue'

- select: # Focusing and selecting one or more values from a dropdown
    selector: select#colors
    value:
      - 'blue'
      - 'red'
      - 'green'

The fill step types a string of text into a focused HTML element, usually form inputs.

- fill:
    selector: input#search
    value: Ugandan knuckles

The types step types a string of text.

- type: Hello world 123 嗨

The press step types special keys such as Shift and Enter.

- press: ArrowLeft

The scroll step scrolls the browser's viewport by a specified number of pixels, a specified HTML element, or to the bottom of the page (for infinite scrolls.)

- scroll: 100 # Scroll 100 pixels
- scroll: a#load-more # Scroll to HTML element
- scroll: # Infinite scroll

The wait step lets you wait by a specified interval, for an HMTL element to load, or for a page navigation to finish.

- wait: div#results # Selector of an element to wait for
- wait: 3000 # Time to wait
- wait: # Wait for a page load to finish

The refresh step refreshes the current page.

- refresh:

The back step moves the current page backward in browser navigation history.

- back:

The forward step moves the current page forward in browser navigation history.

- forward:

The scrape step lets extract and evaluate attributes and HTML elements of the current page.

The scrape step accepts as input a declarative DOM extraction expression based on surgeon.

You describe the content you wish to scrape using select and get keywords.

select '.post-title a' {0,}

Property | Description | Type | Default | Example ---------|:-----------:|:----:|-------------- :|-------------- CSS selector | A CSS selector. | string | undefined | div.my-class a quantifier | Slicing & indexing quantifier to pick amongst matches. | string | [0] | {0,4}[2]

The select keyword accepts as input a CSS selector (e.g. .body p:first-child).

By default, the first matching element is selected. To select multiple or the N-th element, supply a third argument. This 'quantifier' argument is used to select single or multiple CSS selector matches, and to pick specific indexes from the list. For example, select div {0:2}[1] selects the first two matches ({0:2}), and picks the second ([1]) - returning a single element. Read this for more details on the quantifier expression.

get attr src
get prop innerHTML
get html
get text

Property | Description | Type | Default | Example ---------|:-----------:|:----:|-------------- :|-------------- content type | Content to extract. | string | undefined | text, html, prop, attr content name | Name of property or attribute to extract. | string | undefined | src

The get keyword accepts as input one of text, html, prop, and attr for extracting text, raw HTML content, DOM properties, and HTML attributes respectively.

The get keyword accepts a name argument in the case of prop and attr, which specifies the name of the property (e.g. checked) or attribute (e.g. src) to get.

Below is an example of both select and get keywords used together.

- scrape:
    articles:
      - select article {0,}
      - body: select .body | get html
        imageUrl: select img | get attr src
        summary: select .body p:first-child | get text
        title: select .title | get text
    checked: select input[type="checkbox"] | get prop checked

The above WAML will produce the following structured data:

{
  articles: [
    {
      body: ...
      imageUrl: ...
      summary: ...
      title: ...
    },
    {
      body: ...
      imageUrl: ...
      summary: ...
      title: ...
    }
  ],
  checked: ...
}

Subroutines can be nested: HTML elements from a previous level is passed as arguments to the next level. Take a look at the following example:

- select article
- body: select .body | get html
  imageUrl: select img | get attr src
  summary: select .body p:first-child | get text
  title: select .title | get text

Let's walk through these steps:

• The select line returns a list of one or more HTML elements with the selector article. This is equivalent to document.querySelectAll('article').
• In the next line, we name and extract body, imageUrl, summary, and title with the selection in the first line as input.

In this example, body is equivalent to:

document.querySelector('article').querySelector('.body').innerHTML

imageUrl is equivalent to:

document.querySelector('article').querySelector('img').getAttribute('src')

You can use the | pipe operator to simplify a list of expressions:

# Before
- body:
  - select .body
  - get html

The two examples here are equivalent.

Into the following one-liner:

# After
- body: select .body | get html

The Assert Step allows you to perform runtime assertions with Javascript expressions. The results of the assertions are saved into the variable store.

- assert:
    expression: "'${pageTitle}' === 'My Page Title'"
    message: Valid page title

The SetContext step updates the browser context / emulation settings.

- setContext:
    viewport:
      width: 800
      height: 600
    headers:
      X-API-Key: 123
      Authorization: Bearer abc
    cookies:
      - cookieName: cookieValue
    mediaType: screen
    userAgent: myUserAgent
    device: 'iPhone4'
    timeout: 5

The snap step takes a screenshot of the current page and saves it into the local filesystem or a cloud storage.

- snap: # Take screenshot of the page
    filename: my-image
    format: png
    fullPage: true

The pdf step saves the current page as a PDF into the local filesystem, relative to current working directory.

- pdf: my-pdf

WAML provides the following flow control abstractions:

The if and unless optional attributes can be specified in a Step to perform conditional execution.

Pass in a Javascript expression that returns a boolean.

- click:
    selector: a#search
  if: ${someValue} === "submit"

The click Step above will be executed only if the expression ${someValue} === "submit" evaluates to true.

You can reference variables from the Variable Store within your Expression.

WAML is an evolving spec. Documentation, bug reports, pull requests, and all other contributions are welcome!

waml © 2018+, Yos Riady. Released under the MIT License.
Authored and maintained by Yos Riady with help from contributors (list).

yos.io  ·  GitHub @yosriady