arm jobs failing with corrupted workspace

[richardlau]

e.g. https://ci.nodejs.org/job/node-test-commit-arm-debug/nodes=ubuntu1804-arm64/2813/console

11:02:25 Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --progress -- git@github.com:nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/heads/v12.x-staging:refs/remotes/origin/_jenkins_local_branch" returned status code 128:
11:02:25 stdout: 
11:02:25 stderr: error: object file .git/objects/7f/6fe877ea2c2c0726ed1b10c6f946d0176e5c39 is empty
11:02:25 error: object file .git/objects/7f/6fe877ea2c2c0726ed1b10c6f946d0176e5c39 is empty
11:02:25 fatal: loose object 7f6fe877ea2c2c0726ed1b10c6f946d0176e5c39 (stored in .git/objects/7f/6fe877ea2c2c0726ed1b10c6f946d0176e5c39) is corrupt
11:02:25 

Looks to have been happening on

• test-equinix-ubuntu1804_sharedlibs_container-arm64-1
• test-equinix-ubuntu1804_sharedlibs_container-arm64-3

Although it looks like we haven't had a build on test-equinix-ubuntu1804_sharedlibs_container-arm64-2 (the other container on the first Altra docker host) since yesterday so that could possibly also be affected.

[richardlau]

https://ci.nodejs.org/job/node-test-commit-arm/42455/nodes=rhel8-arm64/console also failed in a similar way on test-equinix-rhel8_container-arm64-1.

[sxa]

Do you think it's possible that this has happened when the machine has fallen over at some point?

[richardlau]

Do you think it's possible that this has happened when the machine has fallen over at some point?

Yes. The likeliest explanation is the machine restarted while jobs were executing a git checkout/fetch operation.

[richardlau]

Also these containers are all on the machine that I had to rescue (i.e. unstick from the UEFI prompt) yesterday: #2894 (comment)

[richardlau]

bash-fu:

iojs@test-equinix-ubuntu2004-docker-arm64-1:~$ find . -type d -name ".git" -exec bash -c "export ws=\`dirname {}\` && cd \$ws && if ! git status >/dev/null;
 then echo \$ws ; fi " \;
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
fatal: loose object f050af1aba95dbdfcf15b2338d3b59415381345f (stored in .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f) is corrupt
./test-equinix-ubuntu2004_container-armv7l-1/build/workspace/node-test-commit-arm
error: object file .git/objects/dd/4174dff1f80d5a492441f2a666aacfff9d2444 is empty
error: object file .git/objects/dd/4174dff1f80d5a492441f2a666aacfff9d2444 is empty
error: object file .git/objects/dd/4174dff1f80d5a492441f2a666aacfff9d2444 is empty
fatal: loose object dd4174dff1f80d5a492441f2a666aacfff9d2444 (stored in .git/objects/dd/4174dff1f80d5a492441f2a666aacfff9d2444) is corrupt
./test-equinix-ubuntu1804_sharedlibs_container-arm64-3/build/workspace/node-test-commit-arm-debug
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
fatal: loose object f050af1aba95dbdfcf15b2338d3b59415381345f (stored in .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f) is corrupt
./test-equinix-rhel8_container-arm64-1/build/workspace/node-test-commit-arm
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
error: object file .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f is empty
fatal: loose object f050af1aba95dbdfcf15b2338d3b59415381345f (stored in .git/objects/f0/50af1aba95dbdfcf15b2338d3b59415381345f) is corrupt
./test-equinix-ubuntu1804_sharedlibs_container-arm64-1/build/workspace/node-test-commit-arm-debug
iojs@test-equinix-ubuntu2004-docker-arm64-1:~$

Removing above workspaces:

 find . -type d -name ".git" -exec bash -c "export ws=\`dirname {}\` && cd \$ws && if ! git status >/dev/null; then cd - && rm -rf \$ws ; fi " \;

After:

iojs@test-equinix-ubuntu2004-docker-arm64-1:~$ find . -type d -name ".git" -exec bash -c "export ws=\`dirname {}\` && cd \$ws && if ! git status >/dev/null; then echo \$ws ; fi " \;
iojs@test-equinix-ubuntu2004-docker-arm64-1:~$

[richardlau]

Test builds:

host	job
test-equinix-ubuntu1804_sharedlibs_container-arm64-1	https://ci.nodejs.org/job/node-test-commit-arm-debug/2815/nodes=ubuntu1804-arm64/console
test-equinix-rhel8_container-arm64-1	https://ci.nodejs.org/job/node-test-commit-arm/42460/nodes=rhel8-arm64/console