Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pre-stage test libs on the machine #4902

Merged
merged 1 commit into from
Dec 6, 2023
Merged

Conversation

llxia
Copy link
Contributor

@llxia llxia commented Dec 5, 2023

  • test libs will be staged under ${WORKSPACE}/../../testDependency/lib on the machine if it does not exist

need to merge with adoptium/TKG#476

resolves: #4500

@llxia
Copy link
Contributor Author

llxia commented Dec 5, 2023

  1. first run on a machine that does not pre-staged test libs (Grinder)
    a. stage all test libs (30secs)
00:00:56.011  + perl ./aqa-tests/TKG/scripts/getDependencies.pl -path /home/jenkins/workspace/Grinder/../../testDependency/lib -task default -customUrl https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/
00:00:56.011  /home/jenkins/workspace/Grinder/../../testDependency/lib does not exist, creating one.
00:00:56.011  mkdir /home/jenkins/workspace/Grinder/../../testDependency
00:00:56.011  mkdir /home/jenkins/workspace/Grinder/../../testDependency/lib
00:00:56.011  --------------------------------------------
00:00:56.011  path is set to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:00:56.011  task is set to default
00:00:56.011  dependencyList is set to all
00:00:56.011  --------------------------------------------
00:00:56.011  Starting download third party dependent jars
00:00:56.011  --------------------------------------------
00:00:56.011  downloading dependent third party jars to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:00:56.011  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:56.430  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:56.430  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//openj9jtregtimeouthandler.jar
00:00:56.803  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar
00:00:56.803  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:57.610  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:57.610  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//jaxb-api.jar
00:00:58.414  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/jaxb-api.jar
00:00:58.414  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//asm-all.jar
00:00:59.485  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/asm-all.jar
...
00:01:24.801  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//commons-exec.jar
00:01:25.621  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/commons-exec.jar
00:01:25.621  downloaded dependent third party jars successfully

b. during test compilation, only re-download the jar if SHA does not match. Otherwise, do nothing.

00:01:41.188  getDependentLibs:
00:01:41.188       [exec] --------------------------------------------
00:01:41.188       [exec] path is set to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:01:41.188       [exec] task is set to default
00:01:41.188       [exec] dependencyList is set to junit4
00:01:41.188       [exec] --------------------------------------------
00:01:41.188       [exec] Starting download third party dependent jars
00:01:41.188       [exec] --------------------------------------------
00:01:41.188       [exec] downloading dependent third party jars to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:01:41.188       [exec] /home/jenkins/workspace/Grinder/../../testDependency/lib/junit4.jar exists with correct hash, not downloading
00:01:41.188       [exec] downloaded dependent third party jars successfully
  1. second run on the same machine (pre-stage test libs exist, no downloads) (Grinder) (less than 1sec)
00:07:22.073  + perl ./aqa-tests/TKG/scripts/getDependencies.pl -path /home/jenkins/workspace/Grinder/../../testDependency/lib -task default -customUrl https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/
00:07:22.880  --------------------------------------------
00:07:22.880  path is set to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:07:22.880  task is set to default
00:07:22.880  dependencyList is set to all
00:07:22.880  --------------------------------------------
00:07:22.880  Starting download third party dependent jars
00:07:22.880  --------------------------------------------
00:07:22.880  downloading dependent third party jars to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jaxb-api.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/asm-all.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jcommander.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jtreg_6_1.tar.gz exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/org.eclipse.osgi-3.16.100.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/junit4.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jtreg_7_3_1_1.tar.gz exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/json-simple.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jcstress-tests-all-20220908.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jtreg_5_1_b01.tar.gz exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/testng.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/asm.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/asmtools.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/commons-cli.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jython-standalone.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/javassist.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/commons-exec.jar exists with correct hash, not downloading
00:07:22.880  downloaded dependent third party jars successfully

@llxia
Copy link
Contributor Author

llxia commented Dec 5, 2023

jdk_math xlinux: Grinder
floatSanityTests windows: Grinder
floatSanityTests zos: Grinder

@llxia llxia force-pushed the mytest branch 7 times, most recently from 29cc92e to 667d7d2 Compare December 6, 2023 14:49
@llxia llxia marked this pull request as ready for review December 6, 2023 15:53
@llxia
Copy link
Contributor Author

llxia commented Dec 6, 2023

FYI @AdamBrousseau @pshipton

Copy link
Contributor

@LongyuZhang LongyuZhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@smlambert smlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor typo in comment, but otherwise LGTM.

}
// check pre-stage test libs on the machine
// check for each lib. If lib does not exist, donwload it.
// If lib exists, SHA will be check. Redlownload if SHA does not match.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// If lib exists, SHA will be check. Redlownload if SHA does not match.
// If lib exists, SHA will be checked. Re-download if SHA does not match.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated. Thanks

Signed-off-by: Lan Xia <Lan_Xia@ca.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Jenkins copyArtifacts can be very slow
3 participants