Build script does not terminate properly when detecting concurrent build

[saivann]

I've worked on this issue for a few hours with incomplete conclusions.

The current build script on the build VM should (and previously did) stop to build as soon as it detects another build script starting on a more recent commit. It turns out that the build script still detects concurrent builds and terminate accordingly. However, persistent resource intensive ruby processes indicate that the child processes called by the script in fact do not terminate.

The following lines should in theory kill all sub-processes as soon as the script is exiting. But it turns out that they seem to be somehow ignored for some reason.

trap "rm -rf $WORKDIR; kill 0; exit 1;" EXIT
trap "rm -rf $WORKDIR; kill 0;" SIGINT

As a result, it suffices for a few commits within a short timeframe to start multiple simultaneous build scripts, all of which are uselessly slowing down each other for no reason by eating RAM and CPU. In the worst case scenario, this can lead to extremely long delay for website updates or crashing the VM, although this seems unlikely in our case as we don't push commits constantly.

/tmp also gets polluted to a point that it risks filling up all disk space, leading to impredictable behavior from the build script. For instance, 9522 folders (most being empty) did accumulate leading to using 100% of the disk space. It seems like periodic reboot (clearing /tmp) is what prevented that issue to be noticed.

Confusingly, this issue didn't happen before (and still isn't happening locally on my PC). Upon further testing, I noticed a conclusive fact; when the following line is commented, the build script deletes his temporary directory as expected when the script exits (although it fails to build due to RVM not being configured properly), which suggests that a solution could be close.

source /etc/profile.d/rvm.sh

@harding You've completed a fair amount of work with configuring the VMs with RVM and bundler. I was wondering if you had interest for investigating that issue further and see if you could find a workaround?

[harding]

Working on it now. I've always wanted to improve that part of the script to use more typical locking anyway. Thanks!

[saivann]

@harding Thanks!