Multiple JTReg JDI Tests failing on Windows #1711

M-Davies · 2020-04-02T09:35:08Z

Describe the bug
We currently have approximately 36 JTReg tests consistently failing on JDK8/hs/x32_windows and JDK8/hs/x64_windows&&x32_windows nightlies. It's the same group every night since the 31st March and they're all com/sun/jdi/*.sh tests.

FAILING TESTS

com/sun/jdi/ArrayLengthDumpTest.sh.ArrayLengthDumpTest
com/sun/jdi/BreakpointWithFullGC.sh.BreakpointWithFullGC
com/sun/jdi/CatchAllTest.sh.CatchAllTest
com/sun/jdi/CatchCaughtTest.sh.CatchCaughtTest
com/sun/jdi/CatchPatternTest.sh.CatchPatternTest
com/sun/jdi/CommandCommentDelimiter.sh.CommandCommentDelimiter
com/sun/jdi/DeoptimizeWalk.sh.DeoptimizeWalk
com/sun/jdi/EvalArgs.sh.EvalArgs
com/sun/jdi/EvalArraysAsList.sh.EvalArraysAsList
com/sun/jdi/EvalInterfaceStatic.sh.EvalInterfaceStatic
com/sun/jdi/GetLocalVariables3Test.sh.GetLocalVariables3Test
com/sun/jdi/GetLocalVariables4Test.sh.GetLocalVariables4Test
com/sun/jdi/JdbLockTest.sh.JdbLockTest
com/sun/jdi/JdbMethodExitTest.sh.JdbMethodExitTest
com/sun/jdi/JdbMissStep.sh.JdbMissStep
com/sun/jdi/JdbVarargsTest.sh.JdbVarargsTest
com/sun/jdi/MixedSuspendTest.sh.MixedSuspendTest
com/sun/jdi/NotAField.sh.NotAField
com/sun/jdi/NullLocalVariable.sh.NullLocalVariable
com/sun/jdi/Redefine-g.sh.Redefine-g
com/sun/jdi/RedefineAbstractClass.sh.RedefineAbstractClass
com/sun/jdi/RedefineAddPrivateMethod.sh.RedefineAddPrivateMethod
com/sun/jdi/RedefineAnnotation.sh.RedefineAnnotation
com/sun/jdi/RedefineChangeClassOrder.sh.RedefineChangeClassOrder
com/sun/jdi/RedefineClasses.sh.RedefineClasses
com/sun/jdi/RedefineClearBreakpoint.sh.RedefineClearBreakpoint
com/sun/jdi/RedefineException.sh.RedefineException
com/sun/jdi/RedefineFinal.sh.RedefineFinal
com/sun/jdi/RedefineImplementor.sh.RedefineImplementor
com/sun/jdi/RedefineIntConstantToLong.sh.RedefineIntConstantToLong
com/sun/jdi/RedefineMulti.sh.RedefineMulti
com/sun/jdi/RedefinePop.sh.RedefinePop
com/sun/jdi/RedefineStep.sh.RedefineStep
com/sun/jdi/RedefineTTYLineNumber.sh.RedefineTTYLineNumber
com/sun/jdi/StringConvertTest.sh.StringConvertTest
com/sun/jdi/WatchFramePop.sh.WatchFramePop

The text was updated successfully, but these errors were encountered:

M-Davies · 2020-04-02T09:35:48Z

@smlambert can you take a look? It seems like most are thread & permission failures

sophia-guo · 2020-04-02T11:58:33Z

Those tests were moved from extended bucket to sanity on March 31 as jdk_jdi is part of tier1 openjdk tests. #1695

smlambert · 2020-04-02T12:17:22Z

Yes, so its likely not a new failure, they used to be in extended.sanity, where they may have been failing for some time now, though I see that I had not enabled weekly runs of extended.openjdk on windows like the other platforms, as we were low on machines (so I can't look at a history).

The interesting part is that its tests that are .sh scripts that fail. We should check if only on certain machines, are there permissions that are not allowing the shell scripts to run on all or some our windows machines.

M-Davies · 2020-04-02T15:26:46Z

I've kicked off a few sanity.openjdk grinder runs on several different machines that could be affected.

Machine	Grinder Link	Result
test-aws-win2019-x64-2	https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2739/	FAIL
test-godaddy-win2016-x64-1	https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2740/	FAIL
test-godaddy-win2016-x64-2	https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2741/	FAIL
test-godaddy-win2016-x64-4	https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2742/	FAIL
test-azure-win2012r2-x64-1	https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2743/	FAIL

smlambert · 2020-04-02T15:34:09Z

~~sanity.openjdk will no longer show those failures as the tests have been moved to the extended level~~ ignore that... it will show them!

however, there is no need to run the top-level target (which takes a long time), it is sufficient to run the sub-target, jdk_jdi in the Grinder

(and thanks for launching some Grinders @M-Davies!)

M-Davies · 2020-04-02T15:43:20Z

I aborted them all just before saw your edit @smlambert 😆. No harm done, I've edited the table above with links to new grinders. Should be a little quicker than from the top level target (thanks for that hint)

M-Davies · 2020-04-02T15:55:40Z

Looks like it's all of them. Possibly a silly question, but do the jenkins users have permission to make their own folders/files? I noticed a lot of the tests are spitting out these sort of errors

16:52:54  mkdir: cannot create directory '/aa7364': Permission denied

smlambert · 2020-04-02T18:00:32Z

I believe you have hit upon the root cause of these failures. I know we have had permissions problems on windows very early on in this project, though do not know that these are the same, and not sure specifically what infra actions resolved them, but think this may need to become an infra issue.

M-Davies · 2020-04-03T08:55:45Z

@sxa555 Could you take a look at this? IIRC we've been having perm failures recently with some boxes? Or is that my imagination

sxa · 2020-04-03T15:34:16Z

Looks like it's all of them. Possibly a silly question, but do the jenkins users have permission to make their own folders/files? I noticed a lot of the tests are spitting out these sort of errors
16:52:54  mkdir: cannot create directory '/aa7364': Permission denied

Why would anything be trying to write into the root of the filesystem? The jobs shoudn't have access to write to that location on any system. If it does I'd consider it a bug ...

smlambert · 2020-04-03T16:48:51Z

https://github.com/openjdk/jdk11u/blob/master/test/jdk/com/sun/jdi/ShellScaffold.sh#L287-L330

sxa · 2020-04-03T17:12:24Z

https://github.com/openjdk/jdk11u/blob/master/test/jdk/com/sun/jdi/ShellScaffold.sh#L287-L330

That is an odd piece of logic but suggests the TESTCLASSES variable isn't set.

smlambert · 2020-04-03T17:14:53Z

yes, indeed its not set... not sure what happens on other platforms from this line in that code:
TESTCLASSES=cat $tmpFile | sed -e 's@\\\\@/@g'

M-Davies · 2020-04-08T11:25:37Z

Exclude PR #1724

It should be possible to reproduce this in a grinder. I could investigate this. See what the test is doing further up

M-Davies · 2020-04-08T11:46:24Z

~~Although ShellScaffold.sh is not one of the failing tests and on a JDK11u repo. I'll see if any of the failing tests in the list mess around with paths at all~~

Ah I see https://github.com/M-Davies/openjdk-jdk8u/blob/baec42c5baec170b8da5a2fae4388104530b6262/jdk/test/com/sun/jdi/ArrayLengthDumpTest.sh#L92-L104 and other tests use ShellScaffold.sh to set itself up.

M-Davies · 2020-04-08T14:53:47Z

I ran a grinder with some debug output from https://github.com/M-Davies/openjdk-jdk8u/blob/5715a65e47dac6060cf925c26d408f2bbd5db457/jdk/test/com/sun/jdi/ShellScaffold.sh#L277-L284 to see what the values of some key variables are.

15:51:04  STDOUT:
15:51:04  osname before build break is: 
15:51:04  CYGWIN_NT-10.0
15:51:04  SystemRoot before build break is:
15:51:04  C:\Windows
15:51:04  tmpFile before build break is:
15:51:04  C:\Windows/tmp.1117
15:51:04  TESTCLASSES before build break is:
15:51:04  C:/Users/jenkins.EC2AMAZ-T7I06S9/workspace/Grinder/openjdk-tests/TKG/test_output_15863574524135/jdk_custom_0/work/classes/0/com/sun/jdi

$TESTCLASSES appears to be set correctly but $tmpFile is the one pointing at the root dir. Generated above, the variable is a tmp. double dollar sign. Double dollar sign returns the current process id (according to this SF article anyway) which is what results in the weird numbered file name it's trying to create.

I also see that tmpFile has a forward slash in there when it should really have a backslash to comply with windows file path syntax's. Im not sure if this matters here? @sxa ?

sxa · 2020-04-08T16:45:47Z

In your code the blog that sets tmpfile before the debug messages is:

         if [ -w "$SystemRoot" ] ; then
            tmpFile=$SystemRoot/tmp.$$
         elif [ -w "$SYSTEMROOT" ] ; then
            tmpFile=$SYSTEMROOT/tmp.$$
         else
            tmpFile=tmp.$$
         fi

Whereas the original code that Shelley pointed at was:

            if [ -w "$Temp" ] ; then
                tmpFile=$Temp/tmp.$$
            elif [ -w "$TEMP" ] ; then
                tmpFile=$TEMP/tmp.$$
            else
                tmpFile=tmp.$$
            fi

Has something changed? $TEMP should be /tmp in a cygwin shell which would make tmpFile be /tmp/tmp.$$ which should be ok. Using the Windows SYSTEMROOT(Defaults to C:\Windows) doesn't seem like a good idea (if for no other reason than because the jenkins user, as a non-Administrator, wouldn't generally be able to write to it

M-Davies · 2020-04-09T10:34:12Z

This is a discrepancy between 8 and 9+. This was fixed in https://hg.openjdk.java.net/jdk-updates/jdk11u/rev/3ea9823cb503 (Bug ID https://bugs.openjdk.java.net/browse/JDK-8172188) but was not backported to jdk8 for some reason??? Seeing as it is affecting a large number of tests, this should be probably be contributed upstream

@sxa @smlambert do you agree?

sophia-guo · 2020-04-09T12:27:43Z

@M-Davies Also those shell scripts have been rewritten to java from jdk12+. https://bugs.openjdk.java.net/browse/JDK-8201652

I've talked with @jerboaa in jbs channel https://adoptopenjdk.slack.com/archives/CF3CGHGHW/p1586294477011100 and believe the reason it's not backported to jdk8 is just help-needed:) https://adoptopenjdk.slack.com/archives/CF3CGHGHW/p1586335628012900

M-Davies · 2020-04-09T12:39:17Z

Ah ok. Thanks @sophia-guo . I've opened up a request to the team here in Hursley to see if anyone can pick this up as there's quite a lot on my plate at the moment

M-Davies · 2020-04-09T12:51:42Z

Same story for #1718

M-Davies · 2021-03-12T14:12:58Z

I haven't seen any indication of these tests failing in the last few days. Closing...

sophia-guo · 2021-03-12T14:59:35Z

same as #1737. Reopen.

M-Davies mentioned this issue Apr 2, 2020

Nightly Build&Test Triage Report adoptium/temurin-build#1634

Closed

karianna added the bug label Apr 2, 2020

karianna added this to TODO in aqa-tests via automation Apr 2, 2020

M-Davies mentioned this issue Apr 17, 2020

Multple JDI tests failing on JDK8-s390x_Linux (scaffold issue?) #1737

Closed

M-Davies changed the title ~~Multiple JTReg Sun Tests failing on Windows~~ Multiple JTReg JDI Tests failing on Windows Apr 27, 2020

M-Davies closed this as completed Mar 12, 2021

aqa-tests automation moved this from TODO to Done Mar 12, 2021

sophia-guo reopened this Mar 12, 2021

aqa-tests automation moved this from Done to In Progress Mar 12, 2021

M-Davies mentioned this issue Mar 12, 2021

Trial unexclusions #2376

Merged

smlambert closed this as completed in #2376 Mar 18, 2021

aqa-tests automation moved this from In Progress to Done Mar 18, 2021

karianna added this to the March 2021 milestone Mar 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple JTReg JDI Tests failing on Windows #1711

Multiple JTReg JDI Tests failing on Windows #1711

M-Davies commented Apr 2, 2020 •

edited

M-Davies commented Apr 2, 2020

sophia-guo commented Apr 2, 2020

smlambert commented Apr 2, 2020

M-Davies commented Apr 2, 2020 •

edited

smlambert commented Apr 2, 2020 •

edited

M-Davies commented Apr 2, 2020 •

edited

M-Davies commented Apr 2, 2020

smlambert commented Apr 2, 2020

M-Davies commented Apr 3, 2020

sxa commented Apr 3, 2020

smlambert commented Apr 3, 2020

sxa commented Apr 3, 2020

smlambert commented Apr 3, 2020

M-Davies commented Apr 8, 2020 •

edited

M-Davies commented Apr 8, 2020 •

edited

M-Davies commented Apr 8, 2020

sxa commented Apr 8, 2020

M-Davies commented Apr 9, 2020 •

edited

sophia-guo commented Apr 9, 2020

M-Davies commented Apr 9, 2020 •

edited

M-Davies commented Apr 9, 2020

M-Davies commented Mar 12, 2021

sophia-guo commented Mar 12, 2021

Multiple JTReg JDI Tests failing on Windows #1711

Multiple JTReg JDI Tests failing on Windows #1711

Comments

M-Davies commented Apr 2, 2020 • edited

M-Davies commented Apr 2, 2020

sophia-guo commented Apr 2, 2020

smlambert commented Apr 2, 2020

M-Davies commented Apr 2, 2020 • edited

smlambert commented Apr 2, 2020 • edited

M-Davies commented Apr 2, 2020 • edited

M-Davies commented Apr 2, 2020

smlambert commented Apr 2, 2020

M-Davies commented Apr 3, 2020

sxa commented Apr 3, 2020

smlambert commented Apr 3, 2020

sxa commented Apr 3, 2020

smlambert commented Apr 3, 2020

M-Davies commented Apr 8, 2020 • edited

M-Davies commented Apr 8, 2020 • edited

M-Davies commented Apr 8, 2020

sxa commented Apr 8, 2020

M-Davies commented Apr 9, 2020 • edited

sophia-guo commented Apr 9, 2020

M-Davies commented Apr 9, 2020 • edited

M-Davies commented Apr 9, 2020

M-Davies commented Mar 12, 2021

sophia-guo commented Mar 12, 2021

M-Davies commented Apr 2, 2020 •

edited

M-Davies commented Apr 2, 2020 •

edited

smlambert commented Apr 2, 2020 •

edited

M-Davies commented Apr 2, 2020 •

edited

M-Davies commented Apr 8, 2020 •

edited

M-Davies commented Apr 8, 2020 •

edited

M-Davies commented Apr 9, 2020 •

edited

M-Davies commented Apr 9, 2020 •

edited