Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 121/run fix tests #371

Merged
merged 10 commits into from
Aug 4, 2022
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion gradle/libs.versions.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ allure = "2.4.0"
compose = "1.0.5"
activityCompose = "1.4.0"
androidXTest = "1.4.0"
testOrchestrator = "1.4.1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need testOrchestrator?
I've (and not only me) observed a lot of cases when Orchestrator "doesn't see" tests while you try to run some bunch of them. Temporary removing of Orchestrator from the dependencies and usage of base AndroidTestRun resolved the issue for a lot of cases. Let's try to remove Orchestrator because anyway we use Marathon or something similar for more smart tests execution.

Copy link
Collaborator

@eakurnikov eakurnikov Jul 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Nikitae57 Were there any problems with the app state being shared between tests inside the test run?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matzuk I've encountered this behavior too, but it got resolved by upgrading orchestrator from 1.4.0 to 1.4.1.
@eakurnikov Yes, without orchestrator some tests worked fine when I launched them in scope of their module (gradlew moduleName:connectedDebugAndoirdTest) but failed when I run them in all modules (gradlew connectedAndroidDebugTest)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.4.0 doesn't work with 31 api at all. Even after the update orchestrator remains flaky especially when it comes to local test run. But I think it is fine to have it here for now. We can add other runner in future.

lifecycle = "2.4.0"

[libraries]
Expand Down Expand Up @@ -49,7 +50,7 @@ androidXTestCore = { module = "androidx.test:core", version.ref = "androidXTest"
androidXTestRules = { module = "androidx.test:rules", version.ref = "androidXTest" }
androidXTestFragmentTesting = { module = "androidx.fragment:fragment-testing", version.ref = "androidXTest" }
androidXTestRunner = { module = "androidx.test:runner", version.ref = "androidXTest" }
androidXTestOrchestrator = { module = "androidx.test:orchestrator", version.ref = "androidXTest" }
androidXTestOrchestrator = { module = "androidx.test:orchestrator", version.ref = "testOrchestrator" }
androidXTestExtJunit = "androidx.test.ext:junit:1.1.3"
androidXTestExtJunitKtx = "androidx.test.ext:junit-ktx:1.1.3"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ internal class SystemLanguage(
val configuration = amService.javaClass.getDeclaredMethod("getConfiguration").invoke(amService) as Configuration
configuration.javaClass.getDeclaredField("userSetLocale").setBoolean(configuration, true)
configuration.javaClass.getDeclaredField("locale").set(configuration, locale)
amService.javaClass.getDeclaredMethod("updateConfiguration", Configuration::class.java).invoke(amService, configuration)
amService.javaClass.getMethod("updateConfiguration", Configuration::class.java).invoke(amService, configuration)
logger.i("SystemLanguage: Installing new system language=$locale completed")
} catch (error: Throwable) {
logger.e("SystemLanguage: Installing new system language=$locale failed with error=$error")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import android.annotation.SuppressLint
import android.app.Instrumentation
import android.content.Context
import android.os.Build
import android.os.Environment
import com.kaspersky.kaspresso.internal.extensions.other.createDirIfNeeded
import java.io.File

Expand All @@ -17,7 +16,7 @@ class DefaultDirsProvider(
@SuppressLint("WorldReadableFiles", "ObsoleteSdkInt")
override fun provideNew(dest: File): File {
val dir: File = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
Environment.getExternalStorageDirectory().resolve(dest)
instrumentation.targetContext.applicationContext.filesDir.resolve(dest)
} else {
instrumentation.targetContext.applicationContext.getDir(
dest.canonicalPath,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import com.kaspersky.kaspresso.files.resources.ResourceFileNamesProvider
import com.kaspersky.kaspresso.files.resources.ResourceFilesProvider
import com.kaspersky.kaspresso.files.resources.ResourcesDirsProvider
import com.kaspersky.kaspresso.files.resources.ResourcesRootDirsProvider
import com.kaspersky.kaspresso.internal.extensions.other.createDirIfNeeded
import com.kaspersky.kaspresso.internal.extensions.other.createFileIfNeeded
import java.io.File

Expand Down Expand Up @@ -44,7 +45,7 @@ class DefaultResourceFilesProvider(
return resourcesDirsProvider.provide(
resourcesRootDirsProvider.videoRootDir,
subDir
).resolve(resFileName).createFileIfNeeded()
).resolve(resFileName).createDirIfNeeded().createFileIfNeeded()
}

override fun provideViewHierarchyFile(tag: String, subDir: String?): File {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,8 @@ data class Kaspresso(
internal val objectBehaviorInterceptors: List<ObjectBehaviorInterceptor>,
internal val deviceBehaviorInterceptors: List<DeviceBehaviorInterceptor>,
internal val stepWatcherInterceptors: List<StepWatcherInterceptor>,
internal val testRunWatcherInterceptors: List<TestRunWatcherInterceptor>
internal val testRunWatcherInterceptors: List<TestRunWatcherInterceptor>,
internal val resourcesRootDirsProvider: ResourcesRootDirsProvider
Copy link
Collaborator

@eakurnikov eakurnikov Jul 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it is more useful to have an access to ResourceFilesProvider? If we need only root dirs we still can get them from the files paths.

) {

companion object {
Expand Down Expand Up @@ -930,6 +931,7 @@ data class Kaspresso(
),

instrumentalDependencyProvider = instrumentalDependencyProviderFactory.getTestProvider(instrumentation),
resourcesRootDirsProvider = resourcesRootDirsProvider,

params = Params(
flakySafetyParams = flakySafetyParams,
Expand Down Expand Up @@ -957,7 +959,7 @@ data class Kaspresso(
deviceBehaviorInterceptors = deviceBehaviorInterceptors,

stepWatcherInterceptors = stepWatcherInterceptors,
testRunWatcherInterceptors = testRunWatcherInterceptors
testRunWatcherInterceptors = testRunWatcherInterceptors,
)

configurator.waitForIdleTimeout = kautomatorWaitForIdleSettings.waitForIdleTimeout
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package com.kaspersky.kaspresso.testcases.api.testcase
import com.kaspersky.kaspresso.device.Device
import com.kaspersky.kaspresso.device.server.AdbServer
import com.kaspersky.kaspresso.enricher.MainSectionEnricher
import com.kaspersky.kaspresso.files.resources.ResourcesDirsProvider
import com.kaspersky.kaspresso.files.resources.ResourcesRootDirsProvider
import com.kaspersky.kaspresso.kaspresso.Kaspresso
import com.kaspersky.kaspresso.logger.UiTestLogger
import com.kaspersky.kaspresso.params.Params
Expand Down Expand Up @@ -39,6 +41,7 @@ abstract class BaseTestCase<InitData, Data>(
override val device: Device = testAssistantsProvider.device
override val testLogger: UiTestLogger = testAssistantsProvider.testLogger
override val params: Params = testAssistantsProvider.params
override val resourcesRootDirsProvider: ResourcesRootDirsProvider = testAssistantsProvider.resourcesRootDirsProvider

/**
* Starts the building of a test, sets the [BeforeTestSection] actions and returns an existing instance of
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ import org.junit.Rule
* @param locales comma-separated string with locales to run test with.
*/
abstract class DocLocScreenshotTestCase(
private val resourcesRootDirsProvider: ResourcesRootDirsProvider =
override val resourcesRootDirsProvider: ResourcesRootDirsProvider =
DefaultResourcesRootDirsProvider(),
private val resourcesDirsProvider: ResourcesDirsProvider =
DefaultResourcesDirsProvider(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ package com.kaspersky.kaspresso.testcases.core.testassistants

import com.kaspersky.kaspresso.device.Device
import com.kaspersky.kaspresso.device.server.AdbServer
import com.kaspersky.kaspresso.files.resources.ResourcesDirsProvider
import com.kaspersky.kaspresso.files.resources.ResourcesRootDirsProvider
import com.kaspersky.kaspresso.logger.UiTestLogger
import com.kaspersky.kaspresso.params.Params

Expand All @@ -15,4 +17,5 @@ interface TestAssistantsProvider {
val adbServer: AdbServer
val testLogger: UiTestLogger
val params: Params
val resourcesRootDirsProvider: ResourcesRootDirsProvider
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package com.kaspersky.kaspresso.testcases.core.testassistants

import com.kaspersky.kaspresso.device.Device
import com.kaspersky.kaspresso.device.server.AdbServer
import com.kaspersky.kaspresso.files.resources.ResourcesRootDirsProvider
import com.kaspersky.kaspresso.kaspresso.Kaspresso
import com.kaspersky.kaspresso.logger.UiTestLogger
import com.kaspersky.kaspresso.params.Params
Expand All @@ -12,4 +13,5 @@ internal class TestAssistantsProviderImpl(kaspresso: Kaspresso) : TestAssistants
override val adbServer: AdbServer = kaspresso.adbServer
override val testLogger: UiTestLogger = kaspresso.testLogger
override val params: Params = kaspresso.params
override val resourcesRootDirsProvider: ResourcesRootDirsProvider = kaspresso.resourcesRootDirsProvider
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ import androidx.test.ext.junit.runners.AndroidJUnit4
import com.kaspersky.components.composesupport.config.withComposeSupport
import com.kaspersky.components.composesupport.interceptors.behavior.impl.systemsafety.SystemDialogSafetySemanticsBehaviorInterceptor
import com.kaspersky.kaspresso.composesupport.sample.MainActivity
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeMainScreen
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeSimpleFlakyScreen
import com.kaspersky.kaspresso.kaspresso.Kaspresso
import com.kaspersky.kaspresso.params.FlakySafetyParams
import com.kaspersky.kaspresso.testcases.api.testcase.TestCase
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeMainScreen
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeSimpleFlakyScreen
import io.github.kakaocup.compose.node.element.ComposeScreen
import org.junit.Rule
import org.junit.Test
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.test.ext.junit.runners.AndroidJUnit4
import com.kaspersky.components.composesupport.config.withComposeSupport
import com.kaspersky.kaspresso.composesupport.sample.MainActivity
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeMainScreen
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeSimpleFlakyScreen
import com.kaspersky.kaspresso.kaspresso.Kaspresso
import com.kaspersky.kaspresso.testcases.api.testcase.TestCase
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeMainScreen
import com.kaspersky.kaspresso.composesupport.sample.screen.ComposeSimpleFlakyScreen
import io.github.kakaocup.compose.node.element.ComposeScreen.Companion.onComposeScreen
import org.junit.Rule
import org.junit.Test
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,76 +39,53 @@ class WebComposeTest : TestCase() {
}
}

step("Find \"Sign in\" button and \"Protect your data\" title") {
step("Find \"Interceptors\" and \"Logs\" title") {
WebViewScreen {

webView {
withElement(
Locator.XPATH,
"/html/body/div[1]/section[1]/div/div/h2"
) {
containsText("Protect your data")
withElement(Locator.ID, "interceptors") {
containsText("Interceptors")
}

withElement(
Locator.XPATH,
"/html/body/div[1]/header/section/div[3]/div[2]/button"
) {
hasText("Sign in")
withElement(Locator.ID, "writing-readable-logs") {
containsText("Writing readable logs")
}
}
}
}

step("Click \"Contacts\" button") {
step("Find \"Kaspresso wiki\" link") {
WebViewScreen {
webView {
withElement(
Locator.XPATH,
"/html/body/div[1]/footer/div/div/div[1]/nav/div[1]/a"
) {
withElement(Locator.XPATH, "/html/body/p[39]/a") {
compose(this@webView) {
or {
containsText("fffuuuuu")
hasText("fuuuu")
}
or {
containsText("Ask questiop")
hasText("Ask questiop")
}
or {
containsText("Ask question")
hasText("Ask question")
containsText("Kaspresso kiwi")
hasText("kiwi")
}
or {
containsText("Contacts")
hasText("Contacts")
containsText("Kaspresso wiki")
hasText("Kaspresso wiki")
}
}
}

compose {
orWithElement(
Locator.XPATH,
"/html/body/div[1]/footer/div/div/div[1]/nav/div[1]/a"
) {
orWithElement(Locator.XPATH, "/html/body/p[39]/a") {
hasText("TRATATATA")
}
orWithElement(
Locator.XPATH,
"/html/body/div[1]/footer/div/div/div[1]/nav/div[1]/a"
) {
hasText("Ask question")
orWithElement(Locator.XPATH, "/html/body/p[39]/a") {
hasText("Kaspresso kiwi")
} thenContinue {
click()
}
orWithElement(
Locator.XPATH,
"/html/body/div[1]/footer/div/div/div[1]/nav/div[1]/a"
) {
hasText("Contacts")
orWithElement(Locator.XPATH, "/html/body/p[39]/a") {
hasText("Kaspresso wiki")
} thenContinue {
click()
// click()
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ class UiContinuouslyTest : TestCase() {

@Test
fun testDialogPresentUntilAndroidO() {
// Don`t allow to run this test on Android >= Oreo
Assume.assumeTrue(Build.VERSION.SDK_INT < Build.VERSION_CODES.O)

before {
activityTestRule.launchActivity(null)
}.after {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ class DeviceAccessibilitySampleTest : TestCase() {

step("Disable accessibility service") {
device.accessibility.disable()
assertFalseSafely { isAccessibilityServiceEnabled() }
assertFalseSafely(timeoutMs = 30_000L, intervalMs = 1_000L) { isAccessibilityServiceEnabled() }
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import com.kaspersky.kaspressample.screen.MainScreen
import com.kaspersky.kaspressample.utils.SafeAssert.assertTrueSafely
import com.kaspersky.kaspresso.device.exploit.Exploit
import com.kaspersky.kaspresso.testcases.api.testcase.TestCase
import io.github.kakaocup.kakao.screen.Screen
import org.junit.Assert.assertNull
import org.junit.Rule
import org.junit.Test
Expand Down Expand Up @@ -54,9 +55,10 @@ class DeviceExploitSampleTest : TestCase() {
Configuration.ORIENTATION_LANDSCAPE ==
activityTestRule.activity.resources.configuration.orientation
}
Screen.idle() // kaspresso misses the button in case we won't wait
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Kaspresso should handle all possible stuff in related Interceptors. Are you sure that everything is fine here?

Copy link
Collaborator

@eakurnikov eakurnikov Jul 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Nikitae57 Can you share more context on this? I am not able to reproduce this issue, for me test passes. And I don't see how is it possible actually to miss a button as flaky safety should solve this problem.

I also changed the button id to wrong one to make sure the flaky safety works and it works:
Screenshot 2022-07-20 at 1 28 55 PM

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kaspresso didn't handle this in a "flaky safety" way because it kinda worked right. I encountered a scenario when kaspresso didn't wait for orientation change, tried to press button but missed as orientation change performed at this moment. I assume that changing orientation command sends signal to OS and returns control immediately before actual orientation change happens

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't understand how is it possible that flaky safety can't solve this. Any PerformException that was thrown in case of click failure would be handled anyway. Maybe there was some other exception that was not handled by flaky safety intentionally? For me this test passed successfully any time I launched it. I don't think that we should add Screen.idle() here. If you will be able to reproduce the problem and will provide more details, we could try to solve this in a proper way.

}

step("Press back button") {
step("Open web view and press back button") {
MainScreen {
webViewButton {
click()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,11 @@ import android.os.Bundle
import android.os.Looper
import androidx.test.rule.ActivityTestRule
import androidx.test.rule.GrantPermissionRule
import io.github.kakaocup.kakao.screen.Screen
import com.kaspersky.kaspressample.MainActivity
import com.kaspersky.kaspresso.testcases.api.testcase.TestCase
import io.github.kakaocup.kakao.screen.Screen
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertTrue
import org.junit.Rule
import org.junit.Test
Expand Down Expand Up @@ -84,20 +83,17 @@ class DeviceLocationSampleTest : TestCase() {
Looper.getMainLooper()
)

flakySafely(timeoutMs = 10_000, intervalMs = 500) {
flakySafely(timeoutMs = 30_000, intervalMs = 500) {
val location = manager.getLastKnownLocation(LocationManager.GPS_PROVIDER)
assertNotNull(location)
assertEquals(
MUNICH_LOCATION_LAT, location!!.latitude,
DELTA
)
assertEquals(
MUNICH_LOCATION_LAT, location.latitude,
DELTA
)
}

val location = manager.getLastKnownLocation(LocationManager.GPS_PROVIDER)
assertEquals(
MUNICH_LOCATION_LAT, location!!.latitude,
DELTA
)
assertEquals(
MUNICH_LOCATION_LAT, location.latitude,
DELTA
)
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
package com.kaspersky.kaspressample.device_tests

import android.os.Build
import androidx.test.rule.ActivityTestRule
import com.kaspersky.kaspressample.MainActivity
import com.kaspersky.kaspresso.device.logcat.LogcatBufferSize
import com.kaspersky.kaspresso.testcases.api.testcase.TestCase
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Assume
import org.junit.Rule
import org.junit.Test

Expand All @@ -16,6 +18,11 @@ class DeviceLogcatSampleTest : TestCase() {

@Test
fun logcatTest() {
Assume.assumeTrue(
"Due to Android 8 bug 'logcat -c' fails. To run this test please use another device",
Build.VERSION.SDK_INT != Build.VERSION_CODES.O
)

before {
device.logcat.setBufferSize(LogcatBufferSize(8, LogcatBufferSize.Dimension.MEGABYTES))
device.logcat.disableChatty()
Expand Down Expand Up @@ -119,11 +126,11 @@ class DeviceLogcatSampleTest : TestCase() {
step("Using reader block") {
repeat(100) { testLogger.i("Test6Row$it") }

var fullLogcatList = device.logcat.readLogcatRows()
val fullLogcatList = device.logcat.readLogcatRows()

var inneContainsSize = 0
val isContainsBreaked = device.logcat.readLogcatRows { logcatRow ->
inneContainsSize++
var innerContainsSize = 0
val doesContainBreaked = device.logcat.readLogcatRows { logcatRow ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is instead of does?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? "Does" is a modal verb as an "is" and fits English rules

innerContainsSize++
logcatRow.contains("Test6Row42")
}

Expand All @@ -139,7 +146,7 @@ class DeviceLogcatSampleTest : TestCase() {
logcatRow.contains("beginning of")
}

assertTrue(isContainsBreaked && fullLogcatList.size > inneContainsSize)
assertTrue(doesContainBreaked && fullLogcatList.size > innerContainsSize)
assertTrue(!isNotContainsBreaked && fullLogcatList.size <= innerNotContainsSize)

assertTrue(isBreakedOnBeginningRow && indexOfBeginningRow == 1)
Expand Down
Loading