-
Notifications
You must be signed in to change notification settings - Fork 24.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure global test seed is used for all random testing tasks #38991
Conversation
This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work.
Pinging @elastic/es-core-infra |
@elasticmachine retest this please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, one nit
import org.gradle.api.tasks.TaskContainer | ||
|
||
class RandomizedTestingPlugin implements Plugin<Project> { | ||
|
||
void apply(Project project) { | ||
setupSeed(project) | ||
def seed = setupSeed(project) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we try to use concrete types, as it makes it easier to port these classes to java (which we have slowly been doing).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then when we port from Java -> Kotlin we'll have to change back to val
😉
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
☝️ joking about porting to Kotlin FYI 😆
…#38991) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
* elastic/master: Ensure global test seed is used for all random testing tasks (elastic#38991) re-mutes SmokeTestWatcherWithSecurityIT (elastic#38995) Rollup jobs should be cleaned up before indices are deleted (elastic#38930) relax ML Info Docs expected response (elastic#38993) Re-enable single node tests (elastic#38852) ClusterClientIT refactor (elastic#38872) Fix typo in Index API doc Edits to text & formatting in Term Suggester doc (elastic#38963) (elastic#38989) Migrate Streamable to Writeable for WatchStatus (elastic#37390)
…#38991) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
…#38991) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
…#38991) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
…#38991) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
…#39196) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
…#39197) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work. * Use explicit type
…#39195) This commit fixes a bug which resulted in every RandomizedTestingTask generating its own unique test seed rather than using the global test seed generated by the the RandomizedTestingPlugin. The issue didn't affect build invocations which explicitly ran with -Dtest.seed. In those cases, the Ant junit task correctly uses the global seed. Since the Ant random testing target looks for an Ant project property for determining the existing seed we now explicitly set this inside RandomizedTestingPlugin. It just so happens that, incidentally, Ant will inherit system properties, thus wy the -Dtest.seed mechanism continued to work.
Recently it was noticed that the global generated test seed reported at the beginning of every Gradle build was not always used for subsequent test tasks. For example:
As you can see, the global seed
DA01B1B09A1CDA37
is not the same as the seed4804DC280933D0C4
used by the:server:integTest
Task.It turns out the reason for this is an implementation detail in how the Ant
junit4
task works.In this case it looks for an Ant project property, and if not defined, falls back to generating a new seed.
We currently rely on explicitly setting the
test.seed
system property on the test task here but this actually gets ignored when the forked test VMs are created and the randomly generated seed (created above) is used instead.While investigating it was noticed that invoking the build with an explicit seed (ex:
./gradlew -Dtest.seed=foo
) does result in the behavior we want, which is that all random test tasks in the build use the global seed. The reason for this is that all build system properties are inherited by Ant as project properties.This PR simply unifies these behaviors such that when the global seed is generated as part of the build (not specified on the commandline) we explicitly set the appropriate Ant project property to ensure this propagates to any and all Ant tasks.