diff --git a/.github/workflows/appium_Android.yml b/.github/workflows/appium_Android.yml index 58cd61fce..8a1a30000 100644 --- a/.github/workflows/appium_Android.yml +++ b/.github/workflows/appium_Android.yml @@ -4,6 +4,7 @@ on: push: branches: - 3.x + - 5228-fix-appium-tests env: CI: true diff --git a/docs/saucelabs-troubleshooting.md b/docs/saucelabs-troubleshooting.md new file mode 100644 index 000000000..be1b1e5bc --- /dev/null +++ b/docs/saucelabs-troubleshooting.md @@ -0,0 +1,182 @@ +# Sauce Labs Troubleshooting Guide + +This guide helps resolve common Sauce Labs infrastructure errors when running CodeceptJS tests with Appium. + +## Common Error: "Infrastructure Error -- The Sauce VMs failed to start the browser or device" + +This error typically occurs due to: + +### 1. Missing or Invalid Credentials +```bash +# Set these environment variables before running tests +export SAUCE_USERNAME="your_sauce_username" +export SAUCE_ACCESS_KEY="your_sauce_access_key" +``` + +### 2. Outdated Platform/Device Combinations +The error often occurs when requesting deprecated or unavailable devices/OS versions. + +**Critical Configuration Issues:** +- Using `app: localPath` instead of `app: 'storage:filename=app.apk'` +- Requesting too recent OS versions (Android 12+, iOS 16+) +- Using specific device models that may not be available +- Missing proper capability namespacing for Appium 2.x + +**Quick Fix Checklist:** +1. ✅ Use `storage:filename=` for app references +2. ✅ Use Android 10.0/11.0 and iOS 14.x/15.x versions +3. ✅ Use standard device names: "Android GoogleAPI Emulator", "iPhone 13 Simulator" +4. ✅ Set `noReset: false` for clean test state +5. ✅ Use regional endpoints: `ondemand.us-west-1.saucelabs.com` + +**Updated Android Configuration:** +```javascript +{ + helpers: { + Appium: { + host: 'ondemand.us-west-1.saucelabs.com', + port: 443, + protocol: 'https', + user: process.env.SAUCE_USERNAME, + key: process.env.SAUCE_ACCESS_KEY, + app: 'storage:filename=your-app.apk', // Use Sauce Storage reference + desiredCapabilities: { + 'sauce:options': { + appiumVersion: '2.0.0', + name: 'Your Test Name', + build: process.env.BUILD_NUMBER || 'local-build', + tags: ['codeceptjs', 'appium', 'android'], + recordVideo: false, + recordScreenshots: false, + idleTimeout: 300, + newCommandTimeout: 300, + }, + browserName: '', + platformName: 'Android', + platformVersion: '10.0', // Use stable versions (9.0-11.0) + deviceName: 'Android GoogleAPI Emulator', // Use standard names + automationName: 'UiAutomator2', + autoGrantPermissions: true, + noReset: false, // Clean state for reliable tests + } + } + } +} +``` + +**Updated iOS Configuration:** +```javascript +{ + helpers: { + Appium: { + host: 'ondemand.us-west-1.saucelabs.com', + port: 443, + protocol: 'https', + user: process.env.SAUCE_USERNAME, + key: process.env.SAUCE_ACCESS_KEY, + app: 'storage:filename=your-ios-app.zip', // Use Sauce Storage reference + desiredCapabilities: { + 'sauce:options': { + appiumVersion: '2.0.0', + name: 'Your iOS Test', + build: process.env.BUILD_NUMBER || 'local-build', + tags: ['codeceptjs', 'appium', 'ios'], + recordVideo: false, + recordScreenshots: false, + idleTimeout: 300, + newCommandTimeout: 300, + }, + browserName: '', + platformName: 'iOS', + platformVersion: '15.5', // Use stable versions (14.x-15.x) + deviceName: 'iPhone 13 Simulator', // Use widely available devices + automationName: 'XCUITest', + autoAcceptAlerts: true, + noReset: false, // Clean state for reliable tests + } + } + } +} +``` + +### 3. Regional Endpoint Issues +Use specific regional endpoints instead of the generic one: +- US West: `ondemand.us-west-1.saucelabs.com` +- US East: `ondemand.us-east-1.saucelabs.com` +- EU: `ondemand.eu-central-1.saucelabs.com` + +### 4. App Upload Issues +Ensure your app is properly uploaded to Sauce Storage: +```bash +# Upload app to Sauce Labs +curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \ + -X POST "https://api.us-west-1.saucelabs.com/rest/v1/storage/upload" \ + -H "Content-Type: application/octet-stream" \ + --data-binary @your-app.apk +``` + +### 5. Capability Validation +Before running tests, validate your capabilities using Sauce Labs Platform Configurator: +https://saucelabs.com/platform/platform-configurator + +### 6. Alternative Workarounds + +**Local Testing Fallback:** +```javascript +const sauceConfig = { + // Sauce Labs configuration +}; + +const localConfig = { + host: 'localhost', + port: 4723, + protocol: 'http', + desiredCapabilities: { + platformName: 'Android', + platformVersion: '11.0', + deviceName: 'Android Emulator', + automationName: 'UiAutomator2', + app: '/path/to/your/app.apk' + } +}; + +// Use local config if Sauce Labs credentials are missing +const config = (process.env.SAUCE_USERNAME && process.env.SAUCE_ACCESS_KEY) + ? sauceConfig + : localConfig; +``` + +### 7. Debugging Steps + +1. **Check Sauce Labs service status:** + ```bash + curl -s "https://saucelabs.com/rest/v1/info/status" + ``` + +2. **Verify account limits:** + - Check concurrent session limits + - Verify account is not suspended + - Ensure sufficient credits/minutes + +3. **Test with basic capabilities:** + Start with minimal capabilities and add complexity gradually. + +4. **Monitor Sauce Labs dashboard:** + Check real-time test execution in the Sauce Labs dashboard for detailed error messages. + +### 8. Best Practices + +- Use HTTPS protocol for better security +- Set appropriate timeouts (300+ seconds for mobile tests) +- Use specific device names rather than generic ones +- Keep platform versions current (within 2-3 major versions) +- Use `sauce:options` for Sauce Labs specific configurations +- Enable `noReset: true` to speed up test execution +- Set meaningful test names and builds for better organization + +### 9. Contact Support + +If issues persist after following this guide: +- Check Sauce Labs status page: https://status.saucelabs.com/ +- Contact Sauce Labs support with your session ID and error details +- Consider using alternative cloud providers (BrowserStack, etc.) \ No newline at end of file diff --git a/lib/helper/Appium.js b/lib/helper/Appium.js index a092e1b31..94427e849 100644 --- a/lib/helper/Appium.js +++ b/lib/helper/Appium.js @@ -10,6 +10,8 @@ const { truth } = require('../assert/truth') const recorder = require('../recorder') const Locator = require('../locator') const ConnectionRefused = require('./errors/ConnectionRefused') +const { dontSeeElementError, seeElementError } = require('./errors/ElementAssertion') +const { getElementId } = require('../utils') const mobileRoot = '//*' const webRoot = 'body' @@ -1523,7 +1525,36 @@ class Appium extends Webdriver { */ async dontSeeElement(locator) { if (this.isWeb) return super.dontSeeElement(locator) - return super.dontSeeElement(parseLocator.call(this, locator)) + + // For mobile apps, use native display check instead of JavaScript execution + const parsedLocator = parseLocator.call(this, locator) + const res = await this._locate(parsedLocator, false) + + if (!res || res.length === 0) { + return truth(`elements of ${new Locator(parsedLocator)}`, 'to be seen').negate(false) + } + + // Use native isDisplayed() method without JavaScript execution for mobile + const selected = [] + for (let i = 0; i < res.length; i++) { + try { + // Get element ID using utility function + const elementId = getElementId(res[i]) + + // Use the native WebDriver isDisplayed method directly + const isDisplayed = await this.browser.isElementDisplayed(elementId) + selected.push(isDisplayed) + } catch (err) { + // If native method fails, element is not displayed + selected.push(false) + } + } + + try { + return truth(`elements of ${new Locator(parsedLocator)}`, 'to be seen').negate(selected) + } catch (e) { + seeElementError(parsedLocator) + } } /** @@ -1656,7 +1687,37 @@ class Appium extends Webdriver { */ async seeElement(locator) { if (this.isWeb) return super.seeElement(locator) - return super.seeElement(parseLocator.call(this, locator)) + + // For mobile apps, use native display check instead of JavaScript execution + const parsedLocator = parseLocator.call(this, locator) + const res = await this._locate(parsedLocator, true) + + // Check if elements exist + if (!res || res.length === 0) { + throw new AssertionFailedError(`Element ${new Locator(parsedLocator)} was not found`) + } + + // Use native isDisplayed() method without JavaScript execution for mobile + const selected = [] + for (let i = 0; i < res.length; i++) { + try { + // Get element ID using utility function + const elementId = getElementId(res[i]) + + // Use the native WebDriver isDisplayed method directly + const isDisplayed = await this.browser.isElementDisplayed(elementId) + selected.push(isDisplayed) + } catch (err) { + // If native method fails, element is not displayed + selected.push(false) + } + } + + try { + return truth(`elements of ${new Locator(parsedLocator)}`, 'to be seen').assert(selected) + } catch (e) { + dontSeeElementError(parsedLocator) + } } /** diff --git a/lib/helper/WebDriver.js b/lib/helper/WebDriver.js index 2c511a0d1..06e8368a9 100644 --- a/lib/helper/WebDriver.js +++ b/lib/helper/WebDriver.js @@ -21,6 +21,7 @@ const { focusElement } = require('./scripts/focusElement') const { blurElement } = require('./scripts/blurElement') const { dontSeeElementError, seeElementError, seeElementInDOMError, dontSeeElementInDOMError } = require('./errors/ElementAssertion') const { dontSeeTraffic, seeTraffic, grabRecordedNetworkTraffics, stopRecordingTraffic, flushNetworkTraffics } = require('./network/actions') +const { getElementId } = require('../utils') const WebElement = require('../element/WebElement') const SHADOW = 'shadow' @@ -998,7 +999,7 @@ class WebDriver extends Helper { * {{ react }} */ async click(locator, context = null) { - const clickMethod = this.browser.isMobile && this.browser.capabilities.platformName !== 'android' ? 'touchClick' : 'elementClick' + const clickMethod = this.browser.isMobile && this.browser.capabilities.platformName !== 'android' ? 'touchClick' : 'elementClick' const locateFn = prepareLocateFn.call(this, context) const res = await findClickable.call(this, locator, locateFn) @@ -1217,7 +1218,7 @@ class WebDriver extends Helper { * {{> checkOption }} */ async checkOption(field, context = null) { - const clickMethod = this.browser.isMobile && this.browser.capabilities.platformName !== 'android' ? 'touchClick' : 'elementClick' + const clickMethod = this.browser.isMobile && this.browser.capabilities.platformName !== 'android' ? 'touchClick' : 'elementClick' const locateFn = prepareLocateFn.call(this, context) const res = await findCheckable.call(this, field, locateFn) @@ -1237,7 +1238,7 @@ class WebDriver extends Helper { * {{> uncheckOption }} */ async uncheckOption(field, context = null) { - const clickMethod = this.browser.isMobile && this.browser.capabilities.platformName !== 'android' ? 'touchClick' : 'elementClick' + const clickMethod = this.browser.isMobile && this.browser.capabilities.platformName !== 'android' ? 'touchClick' : 'elementClick' const locateFn = prepareLocateFn.call(this, context) const res = await findCheckable.call(this, field, locateFn) @@ -2985,21 +2986,6 @@ function usingFirstElement(els) { return els[0] } -function getElementId(el) { - // W3C WebDriver web element identifier - // https://w3c.github.io/webdriver/#dfn-web-element-identifier - if (el['element-6066-11e4-a52e-4f735466cecf']) { - return el['element-6066-11e4-a52e-4f735466cecf'] - } - // (deprecated) JsonWireProtocol identifier - // https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#webelement-json-object - if (el.ELEMENT) { - return el.ELEMENT - } - - return null -} - // List of known key values to unicode code points // https://www.w3.org/TR/webdriver/#keyboard-actions const keyUnicodeMap = { diff --git a/lib/utils.js b/lib/utils.js index 408600000..19e5da04a 100644 --- a/lib/utils.js +++ b/lib/utils.js @@ -8,6 +8,28 @@ const { convertColorToRGBA, isColorProperty } = require('./colorUtils') const Fuse = require('fuse.js') const { spawnSync } = require('child_process') +// WebDriver constants +// W3C WebDriver web element identifier +// https://w3c.github.io/webdriver/#dfn-web-element-identifier +const W3C_ELEMENT_ID = 'element-6066-11e4-a52e-4f735466cecf' + +/** + * Get element ID from WebDriver element response + * Supports both W3C and legacy JsonWireProtocol formats + */ +function getElementId(el) { + // W3C WebDriver web element identifier + if (el[W3C_ELEMENT_ID]) { + return el[W3C_ELEMENT_ID] + } + // (deprecated) JsonWireProtocol identifier + // https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#webelement-json-object + if (el.ELEMENT) { + return el.ELEMENT + } + return el +} + function deepMerge(target, source) { const merge = require('lodash.merge') return merge(target, source) @@ -659,3 +681,7 @@ module.exports.markdownToAnsi = function (markdown) { }) ) } + +// WebDriver constants and utilities +module.exports.W3C_ELEMENT_ID = W3C_ELEMENT_ID +module.exports.getElementId = getElementId diff --git a/test/helper/AppiumWeb_test.js b/test/helper/AppiumWeb_test.js index bdda9685a..4bb8ee381 100644 --- a/test/helper/AppiumWeb_test.js +++ b/test/helper/AppiumWeb_test.js @@ -16,15 +16,25 @@ describe('Appium Web', function () { desiredCapabilities: { 'sauce:options': { appiumVersion: '2.0.0', + name: 'CodeceptJS Appium Web Test', + build: process.env.BUILD_NUMBER || 'local-build', + tags: ['codeceptjs', 'appium', 'web', 'android'], + recordVideo: false, + recordScreenshots: false, + idleTimeout: 300, + newCommandTimeout: 300, }, - recordVideo: 'false', - recordScreenshots: 'false', + browserName: 'Chrome', platformName: 'Android', - platformVersion: '6.0', - deviceName: 'Android Emulator', + platformVersion: '10.0', // Use very stable Android 10.0 + deviceName: 'Android GoogleAPI Emulator', // Standard emulator + automationName: 'UiAutomator2', + autoGrantPermissions: true, + noReset: true, }, - host: 'ondemand.saucelabs.com', - port: 80, + protocol: 'https', + host: 'ondemand.us-west-1.saucelabs.com', + port: 443, // port: 4723, // host: 'localhost', user: process.env.SAUCE_USERNAME, diff --git a/test/helper/Appium_ios_test.js b/test/helper/Appium_ios_test.js index 68b64073f..90788cab1 100644 --- a/test/helper/Appium_ios_test.js +++ b/test/helper/Appium_ios_test.js @@ -11,7 +11,7 @@ global.codeceptjs = require('../../lib') let app // iOS test app is built from https://github.com/appium/ios-test-app and uploaded to Saucelabs -const apk_path = 'storage:filename=TestApp-iphonesimulator.zip' +const ios_app_path = 'storage:filename=TestApp-iphonesimulator.zip' const smallWait = 3 describe('Appium iOS Tests', function () { @@ -20,24 +20,32 @@ describe('Appium iOS Tests', function () { before(async () => { global.codecept_dir = path.join(__dirname, '/../data') app = new Appium({ - app: apk_path, + app: ios_app_path, desiredCapabilities: { 'sauce:options': { appiumVersion: '2.0.0', + name: 'CodeceptJS Appium iOS Test', + build: process.env.BUILD_NUMBER || 'local-build', + tags: ['codeceptjs', 'appium', 'ios'], + recordVideo: false, + recordScreenshots: false, + idleTimeout: 300, + newCommandTimeout: 300, }, browserName: '', - recordVideo: 'false', - recordScreenshots: 'false', platformName: 'iOS', - platformVersion: '12.2', - deviceName: 'iPhone 8 Simulator', + platformVersion: '15.5', // Use more stable iOS version + deviceName: 'iPhone 13 Simulator', // Use widely available device + automationName: 'XCUITest', androidInstallTimeout: 90000, appWaitDuration: 300000, + autoAcceptAlerts: true, + noReset: true, }, restart: true, - protocol: 'http', - host: 'ondemand.saucelabs.com', - port: 80, + protocol: 'https', + host: 'ondemand.us-west-1.saucelabs.com', + port: 443, user: process.env.SAUCE_USERNAME, key: process.env.SAUCE_ACCESS_KEY, }) diff --git a/test/helper/Appium_test.js b/test/helper/Appium_test.js index 5d7f47f6d..fd7363378 100644 --- a/test/helper/Appium_test.js +++ b/test/helper/Appium_test.js @@ -19,30 +19,59 @@ describe('Appium', function () { before(async () => { global.codecept_dir = path.join(__dirname, '/../data') - app = new Appium({ - app: apk_path, - desiredCapabilities: { - 'sauce:options': { - appiumVersion: '2.0.0', + + // Check if Sauce Labs credentials are available + if (!process.env.SAUCE_USERNAME || !process.env.SAUCE_ACCESS_KEY) { + console.warn('Sauce Labs credentials not found. Skipping Sauce Labs tests.') + console.warn('Set SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables to run Sauce Labs tests.') + app = new Appium({ + app: apk_path, + desiredCapabilities: { + browserName: '', + platformName: 'Android', + platformVersion: '11.0', + deviceName: 'Android Emulator', + automationName: 'UiAutomator2', + androidInstallTimeout: 90000, + appWaitDuration: 300000, }, - browserName: '', - recordVideo: 'false', - recordScreenshots: 'false', - platformName: 'Android', - platformVersion: '7.0', - deviceName: 'Android GoogleAPI Emulator', - androidInstallTimeout: 90000, - appWaitDuration: 300000, - }, - restart: true, - protocol: 'http', - host: 'ondemand.saucelabs.com', - port: 80, - // port: 4723, - // host: 'localhost', - user: process.env.SAUCE_USERNAME, - key: process.env.SAUCE_ACCESS_KEY, - }) + restart: true, + protocol: 'http', + host: 'localhost', + port: 4723, + }) + } else { + app = new Appium({ + app: 'storage:filename=selendroid-test-app-0.17.0.apk', // Use Sauce Storage reference + desiredCapabilities: { + 'sauce:options': { + appiumVersion: '2.0.0', + name: 'CodeceptJS Appium Test', + build: process.env.BUILD_NUMBER || 'local-build', + tags: ['codeceptjs', 'appium', 'android'], + recordVideo: false, + recordScreenshots: false, + idleTimeout: 300, + newCommandTimeout: 300, + }, + browserName: '', + platformName: 'Android', + platformVersion: '10.0', // Use very stable Android 10.0 + deviceName: 'Android GoogleAPI Emulator', // Standard emulator + automationName: 'UiAutomator2', + androidInstallTimeout: 90000, + appWaitDuration: 300000, + autoGrantPermissions: true, + noReset: false, // Clean state for each test + }, + restart: true, + protocol: 'https', + host: 'ondemand.us-west-1.saucelabs.com', + port: 443, + user: process.env.SAUCE_USERNAME, + key: process.env.SAUCE_ACCESS_KEY, + }) + } await app._beforeSuite() app.isWeb = false await app._before() diff --git a/test/helper/WebDriver.noSeleniumServer_test.js b/test/helper/WebDriver.noSeleniumServer_test.js index 622953f3b..742dcf7c0 100644 --- a/test/helper/WebDriver.noSeleniumServer_test.js +++ b/test/helper/WebDriver.noSeleniumServer_test.js @@ -12,6 +12,7 @@ const WebDriver = require('../../lib/helper/WebDriver') const AssertionFailedError = require('../../lib/assert/error') const Secret = require('../../lib/secret') global.codeceptjs = require('../../lib') +const { W3C_ELEMENT_ID } = require('../../lib/utils') const siteUrl = TestHelper.siteUrl() let wd @@ -41,7 +42,7 @@ describe('WebDriver - No Selenium server started', function () { }, }, customLocatorStrategies: { - customSelector: selector => ({ 'element-6066-11e4-a52e-4f735466cecf': `${selector}-foobar` }), + customSelector: selector => ({ [W3C_ELEMENT_ID]: `${selector}-foobar` }), }, }) }) @@ -382,7 +383,6 @@ describe('WebDriver - No Selenium server started', function () { }) }) - describe('#seeTitleEquals', () => { it('should check that title is equal to provided one', async () => { await wd.amOnPage('/') diff --git a/test/helper/WebDriver_test.js b/test/helper/WebDriver_test.js index 78851d6a3..4d249ece2 100644 --- a/test/helper/WebDriver_test.js +++ b/test/helper/WebDriver_test.js @@ -11,6 +11,7 @@ const AssertionFailedError = require('../../lib/assert/error') const webApiTests = require('./webapi') const Secret = require('../../lib/secret') global.codeceptjs = require('../../lib') +const { W3C_ELEMENT_ID } = require('../../lib/utils') const siteUrl = TestHelper.siteUrl() let wd @@ -44,7 +45,7 @@ describe('WebDriver', function () { }, }, customLocatorStrategies: { - customSelector: selector => ({ 'element-6066-11e4-a52e-4f735466cecf': `${selector}-foobar` }), + customSelector: selector => ({ [W3C_ELEMENT_ID]: `${selector}-foobar` }), }, }) })