Skip to content

fMBT GUI Test Interface FAQ

Jump to bottom
Antti Kervinen edited this page Jan 17, 2019 · 17 revisions

Questions on support for different platforms:

Questions on matching bitmaps:

Questions on synthesizing input events:

Questions on fmbtandroid:

Questions on fmbtwindows:

Can I use fMBT for iPhone, iPad or iOS GUI testing?

fMBT does not support directly connecting to iOS at the moment. However, it is possible to install a VNC server on iOS, after which fMBT's VNC can be used for GUI testing. See

https://github.com/01org/fMBT/wiki/GUI-testing#fmbtvnc

What it takes to implement GUI testing support for a new platform?

Implementing GUI testing support for new platform requires implementing the GUITestConnection class for the platform. This class offers primitives for synthesizing user input (for instance, sendTap, sendTouchDown, sendTouchUp, sendTouchMove), and fetching screenshots (recvScreenshot) from the platform.

Example of implementing basic support for new platform ABC fmbtabc.py:

import fmbtgti

class Device(fmbtgti.GUITestInterface):
    def __init__(self, connectionParameter, **kwargs):
        fmbtgti.GUITestInterface.__init__(self, **kwargs)
        self.setConnection(AbcConnection(connectionParameter))

class AbcConnection(fmbtgti.GUITestConnection):
    def __init__(self, connectionParameter):
        fmbtgti.GUITestConnection.__init__(self)
        # TODO: connect to the device here
    def sendTap(self, x, y):
        # TODO: synthesize tap event to screenshot coordinates (x, y)
    def sendTouchDown(self, x, y):
        # TODO: synthesize touch down at screenshot coordinates (x, y)
    def sendTouchMove(self, x, y):
        # TODO: synthesize touch move to screenshot coordinates (x, y)
    def sendTouchUp(self, x, y):
        # TODO: synthesize touch up at screenshot coordinates (x, y)
    def recvScreenshot(self, filename):
        # TODO: fetch the screenshot and save it to filename

After this you can use visual log, OIR and OCR in your test scripts. Example myabctest.py:

import fmbtabc
d = fmbtabc.Device("my-abc-connection-param")
d.enableVisualLog("abc-device-log.html")
d.refreshScreenshot()
d.swipeBitmap("lock.png", "north")
d.waitBitmap("browser-icon.png")
d.tapBitmap("browser-icon.png")
d.waitOcrText("New window")
d.tapOcrText("New window")
...

How can I save reference bitmaps from screenshots?

fmbt-scripter helps with that. Move the text cursor is on a bitmap filename, such as "window-close.png" on the editor, click the Select button, and select an area from the screenshot on the left.

Why locating a bitmap fails?

Compared screenshot and reference bitmap do not match. For instance, transparency, scaling, antialiasing and slight palette changes are difficult or impossible to see with bare eye, but may cause recognition of the bitmap fail.

Bitmap methods (tapBitmap, ...) require 100 % exact match by default. This can be relaxed by adjusting OIR (optical image recognition) parameters. For instance,

>>> d.tapBitmap("ref-image.png", colorMatch=0.8)

allows 20 % difference on each color channel value on pixel-to-pixel comparisons. Even a bigger threshold may be needed to cope with partial transparency.

How can I find suitable OIR parameters for matching a bitmap?

fMBT's default OIR engine has a method that searches for OIR parameters that enable finding a reference bitmap on a screenshot. Example:

>>> d.oirEngine().adjustParameters(d.screenshot(), "ref-bitmap.png")

You can give adjustParameters a range of colorMatch, bitmapPixelSize, screenshotPixelSize, and scale values from which it should find a working combination. For more information, see

>>> help(d.oirEngine())
>>> help(d.oirEngine().adjustParameters)

How can I avoid repeating OIR parameters on every Bitmap method call?

OIR parameters can be provided in .fmbtoirrc files. Parameters given in the file apply to all reference bitmaps in the same directory. An example of a pretty liberal .fmbtoirrc:

colorMatch = 0.8
bitmapPixelSize = 2
screenshotPixelSize = 2

OIR parameters given in Bitmap method calls in test scripts will override parameters read from the .fmbtoirrc file.

How to cope with different appearances of the same icons, controls or graphics?

The same icons, controls and images may have different appearance due to different theming, localization, software version, orientation and mode (like disabled/enabled/selected), for instance. When the differences are not relevant for test logic, they can be handled without any changes on test scripts by using alternative bitmaps (available since fMBT v0.16.1).

Alternative bitmaps for IMAGE.png are named IMAGE.png.alt*.png and located in the same directory as the original IMAGE.png. Example: running

>>> d.tapBitmap("browser-icon.png", colorMatch=0.8)

will try to find browser-icon.png from the most recent screenshot. If not found and given that two alternative bitmap files (browser-icon.png.alt-selected.png and browser-icon.png.alt-running.png, for instance) are located in the same directory, fMBT will check if either of those could be found on the screenshot. If any of the alternatives is found, tapBitmap("browser-icon.png") will tap that location and return True.

You can find out the alternative bitmap that was actually detected by inspecting the visual log or the item returned by findItemsByBitmap("browser-icon.png").

Alternative bitmaps are supported by all *Bitmap methods.

How to deal with impossible-to-detect regions?

Some graphics are hard or impossible to detect because of transparency, for instance. You can still avoid hard coding coordinates to test script, if you use OIR "forced matching results". In that case coordinates are stored to bitmap.png.fmbtoir.loc.

You can create these files with fmbt-scripter by clicking the Select button twice and then selecting the region.

How can I get coordinates of a bitmap?

Coordinates of a bitmap are available through found Item instances. Example:

>>> allItems = d.screenshot().findItemsByBitmap("ref-image.png")
>>> firstItem = allItems[0]
>>> print "(x1, y1, x2, y2) ==", firstItem.bbox()
>>> print "center (x, y) ==", firstItem.coords()

What is the most efficient way to detect which screen the device under test is showing?

waitAnyBitmap(list-of-reference-bitmaps[, waitTime=n][, pollDelay=n]) waits until any of the reference bitmaps appears on the display, or waitTime expires. It returns a list of bitmaps that contains all reference bitmaps that are visible in the latest screenshot.

Example:

>>> d.tapBitmap("window-close.png")
>>> whatISee = d.waitAnyBitmap(["desktop.png", "browser.png", "camera.png"], waitTime=1.0)
>>> if "desktop.png" in whatISee:
...

What is fmbtwindows error on InjectTouchInput?

These error messages indicate that fmbtwindows cannot synthesize touch events:

  • NotImplementedError: this windows version does not support touch injection
  • AttributeError: function 'InjectTouchInput' not found

For workaround, see How can I synthesize mouse input instead of touch input?

How can I synthesize mouse input instead of touch input?

All tap methods accept optional button=X parameter. fmbtwindows synthesizes a touch event without that parameter, but a mouse click with given mouse button (>= 1) with the parameter. You can set the default value for the optional parameter with setTapDefaults(param=value). For instance, tap with the left mouse button:

d.setTapDefaults(button=1)

How to handle orientation (portrait and landscape) in test code?

This is specific to fmbtandroid. If you want that d.refreshScreenshot() automatically rotates screenshots when the display is rotated, use:

d.setAutoRotateScreenshot(True)

If you want to control display orientation from test code, use:

d.setAccelerometerRotation(False)
d.setUserRotation(fmbtandroid.ROTATION_90)

fmbtandroid defines ROTATION_0, ROTATION_90, ROTATION_180 and ROTATION_270. When you want to switch back to accelerometer-based rotation, use:

d.setAccelerometerRotation(True)

If fmbtandroid.Device is connected to an emulator instead of physical device, you can fake accelerometer data with:

d.setAccelerometer((a, b, c))

How to get UI elements from Android?

fmbtandroid implements two ways to get UI elements with Device.refreshView(): without and with UI Automator. The default is without, but new Android versions work better with it. Use UI automator by default like this:

import fmbtandroid
d = fmbtandroid.Device(uiautomatorDump=True)
ui_elements = d.refreshView()

refreshView() fails, saying Add-Type : Cannot add type. There were compilation errors.

Windows Powershell is too old. Update newer than 2.0.

Clone this wiki locally