Builds are failing due to timeout

[novemberkilo]

Jobs are timing out after the tests run - example

Investigation shows that dune runtest does not exit cleanly:

❯ dune runtest
File "plugins/git/test/dune", line 1, characters 0-52:
1 | (mdx
2 |  (packages current_git)
3 |  (package current_git))
ocaml-mdx-test: internal error, uncaught exception:
                Unix.Unix_error(Unix.EINTR, "waitpid", "")

[maiste]

Investigating. It seems to be related to mdx internal issue.

[maiste]

We have two commits here to compare the working and broken behaviour:

• Working: https://ci.ocamllabs.io/github/ocurrent/ocurrent/commit/cd9156e800915aac3149affaebd58518ee03f653/variant/alpine-3.15-4.14_opam-2.1
• Broken: https://ci.ocamllabs.io/github/ocurrent/ocurrent/commit/b75be989d67034e77e4a77316d342972d75ecc61/variant/alpine-3.15-4.14_opam-2.1

[maiste]

After digging into the investigation, the issue seems unrelated tomdx. Indeed, if you look at the last commit that works and the first one that breaks, you can see the mdx version is precisely the same, and so are the other packages.

However, one point that seems to change is the git version in the docker image. This is the patch that shows the difference between the package in the two ubove version:

 curl-7.80.0-r3 x86_64 {curl} (MIT) [installed]
< expat-2.4.9-r0 x86_64 {expat} (MIT) [installed]
---
> curl-7.80.0-r4 x86_64 {curl} (MIT) [installed]
> expat-2.5.0-r0 x86_64 {expat} (MIT) [installed]
19c19
< git-2.34.4-r0 x86_64 {git} (GPL-2.0-or-later) [installed]
---
> git-2.34.5-r0 x86_64 {git} (GPL-2.0-or-later) [installed]
29c29
< libcurl-7.80.0-r3 x86_64 {curl} (MIT) [installed]
---
> libcurl-7.80.0-r4 x86_64 {curl} (MIT) [installed]
41,42c41,42
< libx11-1.7.2-r0 x86_64 {libx11} (custom:XFREE86) [installed]
< libx11-dev-1.7.2-r0 x86_64 {libx11} (custom:XFREE86) [installed]
---
> libx11-1.7.3.1-r0 x86_64 {libx11} (custom:XFREE86) [installed]
> libx11-dev-1.7.3.1-r0 x86_64 {libx11} (custom:XFREE86) [installed]

An easy way to reproduce is to take the non-working Dockerfile and just replace the image sha with the sha from the working image.

Considering (in the case of alpine), the git patch do this, we still need to figure out why it makes ocurrent have an infinite loop.

[maiste]

Thanks to @art-w and @MisterDA, one solution has been found here: https://bugs.launchpad.net/ubuntu/+source/git/+bug/1993586.

This idea authorises the file on the command line.

[tmcgilchrist]

https://ci.ocamllabs.io/github/ocurrent/ocurrent/commit/2a710c0fe7448632adbff9aeed1323203d61d60f/variant/macos-homebrew-4.14.0_arm64_opam-2.1

This is not fully fixed on macOS @maiste @MisterDA

[tmcgilchrist]

I am able to reproduce the issue with git 2.36.1 on macOS home-brew. Upgrading to 2.38.1 resolves the issue for me locally on macOS 12.6.1 (Monterey). The macOS worker m1-worker-03 is using git version 2.37.1 (Apple Git-137.1) also on Monterey.

Running the tests with the git version 2.37.1 (Apple Git-137.1) fails around this point in the test https://github.com/ocurrent/ocurrent/blob/master/plugins/git/test/test_submodules.md?plain=1#L108-L121

[mtelvers]

I have updated the build script to include brew install git as part of the base image. Homebrew is first in $PATH; therefore, the Homebrew version of git will now be used in preference to the Apple-supplied version. I have taken the opportunity to update from beta1 to beta2. The updates are running, but only two machines have been completed.

[tmcgilchrist]

Fixed in realworldocaml/mdx#409