Continuous sync from remote to local machine during remote command execution.

[artem-zinnatullin]

Currently Mainframer has 3 main steps that run in serialized manner one after another:

1. sync from local to remote machine
2. remote command execution
3. sync from remote to local machine

Step №1 is typically fast enough and not much we can do here, optimization of step №2 is not something we can do in Mainframer, but step №3 actually looks like something we can optimize in Mainframer.

Instead of waiting for remote command to finish, we can try to run step №3 in parallel to step №2, that can significantly reduce overall execution time.

Quick lookup suggests that rsync doesn't support "continuous" sync.
We can try hooking into filesystem changes on remote machine and trigger rsync. Although I feel like that would require 2 additional ssh connections because remote machine can't directly ssh into local machine.

@ming13 thoughts?

[arturdryomov]

It might work, but there is an issue regarding a black box nature of the execution phase. The execution itself might produce some intermediate files that are being discarded on completion. In the suggested scenario we are going to copy them anyway, i. e. introducing redundancy of unknown amounts (since we don’t know if it is going to be GiB or KiB). Plus an IDE or an another client-side indexing thingy potentially can go nuts trying to reindex potentially not valid state (since the copy operation is not sync, i. e. intermediate files can be out of order).

[artem-zinnatullin]

These are valid concerns for sure, but I think it still might worth it.

Unnecessary intermediate files most likely can be ignored through remoteignore config.

Afaik IDEs (IntelliJ) don't hook into filesystem notifications, so if you keep using Mainframer IntelliJ Plugin it will only trigger IDE file sync after Mainframer exit which we will do only after final rsync run after remote command execution.

[artem-zinnatullin]

rsync warning: some files vanished before 
they could be transferred (code 24) at main.c(1679) [generator=3.1.3]

Something I'll have to deal with…

[artem-zinnatullin]

I guess warning is fine and actually points to intermediate files that should be ignored via config, but this one is error:

Sync remote → local machine failed: rsync exit code '24',
rsync stdout 'cannot delete non-empty directory: buck-out'

(Noice error output btw, 😽myself)

[artem-zinnatullin]

mhm, this problems seems to be consistently appearing during clean tasks like

mainframer ./gradlew clean

Doesn't happen during clean/incremental builds so far.

[arturdryomov]

Hey-ho, say hello to races 🚗