AutoPy Introduction and Tutorial
AutoPy is a simple, cross-platform GUI automation library for Python. It includes functions for controlling the keyboard and mouse, finding colors and bitmaps on-screen, and displaying alerts.
Currently supported on macOS, Windows, and X11 with the XTest extension.
- Python 2.7, or Python 3.5 and up.
- Rust 1.23.0-nightly 2019-02-06 or later (unless using a binary wheel distribution).
- macOS 10.6 and up.
- Windows 7 and up.
- X11 with the XTest extension.
First, see if a binary wheel is available for your machine by running:
$ pip install -U autopy
If that fails, install rustup and then run:
$ rustup default nightly-2019-10-05 $ pip install -U setuptools-rust $ pip install -U autopy
Another option is to build from the latest source on the GitHub repository:
$ git clone git://github.com/autopilot-rs/autopy-rs.git $ cd autopy $ make $ make install
Note: AutoPy currently requires the
2019-10-05 Rust nightly in order to
build from source. This is to maintain compatibility with an older version of
PyO3, as the latest version has dropped Python 2 support. Python 2 support will
likely be dropped from AutoPy as well sometime later this year, depending on
how necessary it is to upgrade to a more recent version of PyO3 or Rust. In the
meantime, it may be necessary to install the required nightly via the following
when building locally:
rustup install nightly 2019-10-05 --force
This is due to rustup complaining that it doesn't include certain components
Additional instructions for installing from source on Windows are available here.
The following is the source for a "hello world" script in autopy. Running this code will cause an alert dialog to appear on every major platform:
import autopy def hello_world(): autopy.alert.alert("Hello, world") hello_world()
Controlling the Mouse
AutoPy includes a number of functions for controlling the mouse. For a full list, consult the API Reference. E.g., to immediately "teleport" the mouse to the top left corner of the screen:
>>> import autopy >>> autopy.mouse.move(0, 0)
To move the mouse a bit more realistically, we could use:
>>> import autopy >>> autopy.mouse.smooth_move(0, 0)
Even better, we could write our own function to move the mouse across the screen as a sine wave:
import autopy import math import time import random import sys TWO_PI = math.pi * 2.0 def sine_mouse_wave(): """ Moves the mouse in a sine wave from the left edge of the screen to the right. """ width, height = autopy.screen.size() height /= 2 height -= 10 # Stay in the screen bounds. for x in range(int(width)): y = int(height * math.sin((TWO_PI * x) / width) + height) autopy.mouse.move(x, y) time.sleep(random.uniform(0.001, 0.003)) sine_mouse_wave()
Controlling the Keyboard
The following will enter the keys from the string "Hello, world!" in the currently focused input at 100 WPM:
import autopy autopy.key.type_string("Hello, world!", wpm=100)
Alternatively, individual keys can be entered using the following:
import autopy autopy.key.tap(autopy.key.Code.TAB, [autopy.key.Modifier.META]) autopy.key.tap("w", [autopy.key.Modifier.META])
Working with Bitmaps
All of autopy's bitmap routines can be found in the module
useful way to explore autopy is to use Python's built-in
help() function, for
help(autopy.bitmap.Bitmap). AutoPy's functions are documented with
descriptive docstrings, so this should show a nice overview.
>>> import autopy >>> autopy.bitmap.capture_screen() <Bitmap object at 0x12278>
This takes a screenshot of the main screen, copies it to a bitmap, displays its memory address, and then immediately destroys it. Let's do something more useful, like look at its pixel data:
>>> import autopy >>> autopy.bitmap.capture_screen().get_color(0, 0) 15921906
AutoPy uses a coordinate system with its origin starting at the top-left, so
this should return the color of pixel at the top-left corner of the screen. The
number shown looks a bit unrecognizable, but we can format it with Python's
>>> import autopy >>> hex(autopy.bitmap.capture_screen().get_color(0, 0)) '0xF2F2F2'
Alternatively, we can use:
>>> import autopy >>> autopy.color.hex_to_rgb(autopy.screen.get_color(0, 0)) (242, 242, 242)
which converts that hex value to a tuple of
(r, g, b) values. (Note that
autopy.screen.get_color(), used here, is merely a more convenient and
efficient version of
To save the screen capture to a file, we can use:
>>> import autopy >>> autopy.bitmap.capture_screen().save('screengrab.png')
The filetype is either parsed automatically from the filename, or given as an optional parameter. Currently only jpeg and png files are supported.
>>> import autopy >>> autopy.bitmap.Bitmap.open('needle.png') <Bitmap object at 0x1001d5378>
Aside from analyzing a bitmap's pixel data, the main use for loading a bitmap is finding it on the screen or inside another bitmap. For example, the following prints the coordinates of the first image found in a bitmap (scanned from left to right, top to bottom):
import autopy def find_image_example(): needle = autopy.bitmap.Bitmap.open('needle.png') haystack = autopy.bitmap.Bitmap.open('haystack.png') pos = haystack.find_bitmap(needle) if pos: print("Found needle at: %s" % str(pos)) find_image_example()
It's also possible to do a bounded search by passing a tuple
((x, y), (width, height)):
haystack.find_bitmap(needle, rect=((10, 10), (100, 100)))
Projects using AutoPy
- AutoPyDriverServer, AutoPy through WebDriver or a webdriver-compatible server.
- guibot, A tool for GUI automation using a variety of computer vision and desktop control backends.
- spynner, Programmatic web browsing module with AJAX support for Python.
- SUMO, An open source, highly portable, microscopic and continuous road traffic simulation package designed to handle large road networks.
Hope you enjoy using autopy! For a more in depth overview, see the API Reference.
If you are interested in this project, please consider contributing. Here are a few ways you can help:
- Report issues.
- Fix bugs and submit pull requests.
- Write, clarify, or fix documentation.
- Suggest or add new features.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.