sporadic issue with background cache operations when running pants in a loop

[cosmicexplorer]

The following was described in #general when running pants 1.14.0 in a loop:

## Exception when compiling 44 sources to /data/var/lib/jenkins/workspace/pr-mediarithmics-platform-pants/.pants.d/compile/zinc/a53b6931478d/mediarithmics-api.core-api.src.test.scala.scala/current/classes
                       [error] java.io.IOException: Analysis at (d6aa1af9a111c49918f3f1e79451c82b: /data/var/lib/jenkins/workspace/pr-mediarithmics-platform-pants/.pants.d/compile/zinc/a53b6931478d/mediarithmics-api.common-play-api.src.main.scala.scala/9520fc387886/z.analysis) has changed since startup!

The script leading to this was based on https://www.pantsbuild.org/orgs.html#continuous-integration, leading to #7478 to attempt to fix it, before realizing that running pants in a loop was actually necessary. The script was:

#!/usr/bin/env bash

set -eEux

function clean_pants() {
	./pants ng-killall
}

trap clean_pants ERR

remote=${1:-"origin"}
changedParent="${remote}/${2:-"master"}"

# Disable Zinc incremental compilation to ensure no historical cruft pollutes the current build.
export PANTS_COMPILE_ZINC_INCREMENTAL=false


changed=$(./pants --changed-parent=${changedParent} list)
dependees=$(./pants dependees --dependees-transitive --dependees-closed ${changed})
filtered=$(mktemp)
./pants filter --filter-type=junit_tests \
               --filter-tag-regex=-integration,test-dev,suite-by-suite ${dependees} | sort > ${filtered}

for target in $(cat ${filtered}); do
  ./pants test.junit --output-mode=ALL --no-fast --parallel-threads=12 ${target}
done

./pants filter --filter-type=junit_tests \
               --filter-tag-regex=suite-by-suite ${dependees} | sort > ${filtered}

for target in $(cat ${filtered}); do
    ./pants test.junit --batch-size=1 --output-mode=ALL ${target%:*}
done

rm ${filtered}

For the user with the issue, running pants without a loop both worked and stopped them from seeing this issue, but it was explained that running in a loop is actually necessary right now. The error to me seems to imply that some of our background cache operations aren't completing quickly enough before the next pants run starts somehow.

[chrisbenincasa]

Our org also just started getting this error on a consistent basis. We're also running a CI script similar to the one from the docs, which includes running Pants in a loop. Are there any workarounds to this?

[stuhood]

Hey folks! As described in https://blog.pantsbuild.org/automatically-unlocking-concurrent-builds-and-fine-grained-caching-on-the-jvm-with-dependency-inference/ and https://blog.pantsbuild.org/pants-2-9/, per-file dependency inference means that Pants v2 no longer needs to use Zinc.

We are very happy to help folks upgrade from Pants 1.30.x to 2.9+ in order to use the new support: please check out https://www.pantsbuild.org/docs/jvm-overview and visit Slack if you have questions.