You can clone with
HTTPS or Subversion.
I am frequently working with remote file-systems by mounting them using SSHFS while working. Obviously the delay caused by the connection slows down CtrlP's caching. But the CtrlPBuffer command is delayed as well. Why is this? Does it even interact the the file-system? The delay is well above three seconds which makes CtrlP unusable (shouldn't the delay also be diminish by caching when using the CtrlP command?).
It's probably the root finding feature. You can try turning it off with g:ctrlp_working_path_mode.
The root finding/zooming feature is the only thing that interacts with the file system when starting in Buffer mode. In File mode, the sorting compares modified times, so it also interacts with the file system regardless of the file list being cached.