[ant-276] copy project with rsync

[hazelbit]

Motivation

Small PR to copy the project using rsync to remote macOS devices to improve stability and reduce walltime.

The two main benefits are increased resilience (especially in regards to retries) and skipping . files/folders.

Quick note: rsync can compress files for transfer (-z, trading cpu utilization and network bandwidth) which led to a walltime penalty in our scenarios (see ticket for more details) so it is not part of the PR.

Draft PR with more pipelines to be scheduled, for success we should not see regressions in walltime and all further steps should find their required files in the same location (we should not see any pipeline failures related to this change). See ant-276.

[theopnv]

It's unclear whether the retry mechanism in case the connection was reset is built-in rsync or not. My understanding is that it's not, so if we want to have one maybe we'll have to have to call it inside some while loop? See this SO reply https://superuser.com/a/302843

[hazelbit]

That is right, I was relying on the retries: 2 mechanism (initial + up to two attempts). The -P implies --partial which is what we want for the retries (though arguably I should pass that explicitly, the second implied arg, --progress, is unused due to -q/--quiet, it's a leftover). Parts of the description were inaccurate as I've mentioned in the ticket which contains more background information ✔️