Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed order dependent flakiness in SensitiveValueTest.java in cloudslang-entities module #1427

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

yijut2
Copy link

@yijut2 yijut2 commented Nov 29, 2023

Description

Fixed the order dependent flaky test in SensitiveValueTest.java in cloudslang-entities module.

Root Cause

The order dependent flakiness was detected in the cloudslang-entities module. If the EncryptorConfigTest.java was run before SensitiveValueTest.java, there will be an error.
The error message was below:

[ERROR] Failures: 
[ERROR]   SensitiveValueTest.testEncryptedStringSensitiveValue:71->verifyEncrypted:116->verifySensitiveValue:127 expected:<[{Encrypted}foo]> but was:<[Encrypted]>
[ERROR]   SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted:95->verifyEncrypted:116->verifySensitiveValue:128 expected:<[bar]> but was:<[Decrypted]>
[ERROR]   SensitiveValueTest.testSensitiveValueEncryptDecrypt:47->verifyEncrypted:116->verifySensitiveValue:127 expected:<[{Encrypted}rO0ABXQAFk9yaWdpbmFsU2Vuc2l0aXZlVmFsdWU=]> but was:<[Encrypted]>
[INFO] 
[ERROR] Tests run: 5, Failures: 3, Errors: 0, Skipped: 0

The flakiness is due to the shared state of the EncryptionProvider an Encryption beans between the two test classes. Both EncryptorConfigTest and SensitiveValueTest test define beans for the Encryption interface (getEncryption() and getTestEncryption()). Depending on the order of execution, it might retain one definition over the other. In EncryptorConfigTest, we manipulating the EncryptionProvider in multiple tests, and those can affect the state of the EncryptionProvider and Encryption. However, they were not reseted properly after this test class completes. And then the SensitiveValueTest depends on those shared state, so it cause failures when run after EncryptorConfigTest.

Fix log

To fix this issue, I add the helper function afterTest() that calls cleanup() to be run after each test method, to clean up the state of EncryptionProvider and Encryption.

After the fix, if we run EncryptorConfigTest.java before SensitiveValueTest.java, the order dependent flakiness no longer exists.
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

How this has been tested

Java: openjdk version "11.0.20.1"
Maven: Apache Maven 3.9.5

How to reproduce the failure

  • mvn install -pl cloudslang-entities -am -DskipTests
  • create the CustomTestSuite.java under the cloudslang-entities/src/test/java/io/cloudslang/fortest to run two test classes in order, the code looks like
package io.cloudslang.fortest;

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import io.cloudslang.lang.entities.encryption.EncryptorConfigTest;

@RunWith(Suite.class)

@Suite.SuiteClasses({
        EncryptorConfigTest.class,
        SensitiveValueTest.class,
})

public class CustomTestSuite {  
}

Signed-off-by: Yi-Ju Tsai [yijujt2@illinois.edu]

fix order dependent flakiness

Signed-off-by: Yi-Ju Tsai yijujt2@illinois.edu
yijut2 added a commit to yijut2/idoft that referenced this pull request Nov 29, 2023
Opened PR in cloudslang project

CloudSlang/cloud-slang#1427
@yijut2
Copy link
Author

yijut2 commented Nov 30, 2023

@QueueUpX Could you please review my PR if you have time? Really appreciate!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants