git fetch openjdk-tests fails on aarch64

[adamfarley]

Problem:
The failure occurs right after we start performing sanity.system and extended.system tests on the aarch64 test node.

 > git fetch --tags --force --progress -- https://github.com/AdoptOpenJDK/openjdk-tests.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://github.com/AdoptOpenJDK/openjdk-tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 21, done.        
remote: Counting objects:   4% (1/21)        
etc etc    
remote: Counting objects: 100% (21/21), done.        
remote: Compressing objects:   4% (1/21)        
etc etc     
remote: Compressing objects: 100% (21/21), done.        
Receiving objects:   0% (1/11853)
etc etc
Receiving objects:  15% (1778/11853)
error: RPC failed; curl 56 GnuTLS recv error (-24): Decryption has failed.
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

• test suite/name?
  JDK8 j9 extended.system - test-packet-ubuntu1604-armv8-1
  JDK8 j9 sanity.system - test-docker-ubuntu2004-armv8-1
  JDK16 j9 sanity.system](https://trss.adoptopenjdk.net/output/build?id=601b904dddf93e76e9ff1013) - build-docker-ubuntu1804-armv8-2
  JDK16 j9 sanity.system (xl)](https://trss.adoptopenjdk.net/output/build?id=601b903eddf93e76e9ff0ff0) - test-packet-ubuntu1604-armv8-1
  JDK16 j9 extended.system (xl) - test-docker-ubuntu2010-armv8-1

• Is there an existing issue elsewhere covering this?
  Not that I found.

• Which machine(s) does it work on?
  build-packet-ubuntu1804-armv8l-1f2
  build-packet-ubuntu1804-armv8l-1a
  test-packet-ubuntu1604-armv8-2
  build-packet-ubuntu1804-armv8l-1c
  build-packet-ubuntu1804-armv8l-1b
  build-packet-ubuntu1804-armv8l-1a

• Which machine(s) does it fail on?
  build-docker-ubuntu1804-armv8-2
  test-docker-ubuntu2004-armv8-1
  test-docker-ubuntu2010-armv8-1
  test-packet-ubuntu1604-armv8-1

• Do you have a link to a Grinder re-run if the test with the failure?

Any other details:
In passing runs, we see the message "Avoid second fetch" after the git fetch command.

Could be an infra problem based on the node names. May need to open secondary issue after further investigation.

[adamfarley]

Using grinders on two of the "failing" machines results in the same "Avoid second fetch" messages, and no defect reproduction.

Rebuilding the pipeline job to rule out limitations with the grinder.

https://ci.adoptopenjdk.net/job/Test_openjdk16_j9_sanity.system_aarch64_linux/22/console

[adamfarley]

Git fetch appears to have passed.

Could this be a temporary issue? Some sort of strange networking bug on aarch64 that's passed?

Unknown. Will put this on the backburner and review the nightlies tomorrow and Monday.

If the issue does not reoccur by Monday (8/2/2021), I'll close this issue.

[adamfarley]

Didn't even need to wait that long, here's a reproduction on test-docker-ubuntu1804-armv8-1.

Investigation resumed.

[sxa]

A couple of things:

• This may be related to the issues we've seen on some builds on the ThunderX aarch64 systems (aarch64 build failure, Hash of java.rmi (...) differs to expected hash  temurin-build#1804 (comment))
• A number of the machines listed above have now been renamed: Proposal - increase proportion of testing performed in docker containers infrastructure#1809 (comment)

So the hosts that it

Which machine(s) does it work on?
build-packet-ubuntu1804-armv8l-1f2 (D05)
build-packet-ubuntu1804-armv8l-1a (D05)
test-packet-ubuntu1604-armv8-2 (ThunderX host)
build-packet-ubuntu1804-armv8l-1c (D05)
build-packet-ubuntu1804-armv8l-1b (D05)
build-packet-ubuntu1804-armv8l-1a (D05)

Which machine(s) does it fail on?
build-docker-ubuntu1804-armv8-2 (Hosted on build-packet-ubuntu1804-armv8-1 - ThunderX)
test-docker-ubuntu2004-armv8-1 (ThunderX)
test-docker-ubuntu2010-armv8-1 (ThunderX)
test-packet-ubuntu1604-armv8-1 (ThunderX)

[adamfarley]

Closing this issue as the underlying issue appears to be ThunderX-specific, and the machines may now be disabled until they are removed.

Further actions on the underlying issue will take place in this infra issue.