Visualize progress of potentially longer operations

[dumblob]

Currently lf does not show any feedback for pasting a copy or similar operations. This gets frustrating especially in situations when the files/directories are on a remote storage or any other less reliable storage (USB flash drives, etc.) and one doesn't know whether the operation progresses normally or not.

E.g. copying even a small files takes time which freezes lf and the user does not know what's the reason (it might be a timed-out connection of the remote drive, it might be just a big file, it might be lf getting stuck because of bug, etc.).

Could you please add some progress indication (either something like a very simple "modal" dialog in the form of a red progress bar overlapping the status bar and showing percentage or even a job queue view independent from the main process)?

[gokcehan]

@dumblob This is something I'm really missing from ranger. Two separate things we need is to do operations asynchronously and add a progress bar. First one may not be difficult but I don't have an idea how to tackle on the second one yet.

Currently there is no file copying function provided in go standard library since portability is not very easy to achieve:

https://go-review.googlesource.com/#/c/1591/

I try to avoid implementing it in lf for the same reason. That's why lf uses cp and mv for file operations and they do not provide a standard way to indicate progress as far as I'm aware. I suspect ranger uses the file copying functions in python standard library which may be providing a progress indication somehow.

Another possibility is to customize paste command which was discussed somewhere in another issue. This way, it is possible to use rsync or pv to display the progress somehow. The problem with this approach is that we can only do this synchronously without involving the ui.

[dumblob]

Well, there is a half-way solution I would say. Namely the trick as used by progress. This should work on most systems unless they don't provide file descriptor information anywhere independently. On these "problematic" systems this feature would be simply substituted by changing the status message to Copying in progress... (no worries, any errors are reported ASAP) and corresponding Copying successfully done.

[KenjiTakahashi]

No, Python's stdlib does not include any way to do this. Ranger has implemented its' own set of functions, which yield every 16KB of transferred data (here).

The cv (now progress) util was the first thing that came to my mind as well. But this "trick" is not portable at all, procfs is a strictly Linux thing nowadays (you can install/enable it on most BSDs, but most (all?) people don't, and so we shouldn't rely on it). If you look in the code, you'll see that progress is actually using a completely different mechanism for macOS support.

[dumblob]

I'm aware of the portability issue (that's why I called it a half-way solution).

We could also step aside and just implement a tiny wrapper for a user-defined command to show progress (and do the desired operation), so that user can decide himself, whether he'll use e.g. progress or not (in which case only the information strings I mentioned above would be shown).

[gokcehan]

Sorry for late response.

@KenjiTakahashi thanks for the direct link for ranger's copy functions.

Instead of using half portable solutions as in progress we might as well implement our own half portable copy functions. But ranger's code shows that things get hairy rather quickly and in our case we don't have a reference implementation in the standard library to copy from. So I think it is best to avoid this as much as possible.

In the meantime, I have been trying something along the line (which does not work yet):

cmd paste-async &{{
    resp=$(echo 'load' | nc -U /tmp/lf.${USER}.sock)
    mode=$(echo $resp | cut -d' ' -f1)
    list=$(echo $resp | cut -d' ' -f2-)
    if [ $mode = 'copy' ]; then
        lf -remote "send $id echo copying.."
        IFS=':'; rsync -av --progress $list . | awk '
            BEGIN { RS="\r" }
            { system("lf -remote \"send " $ENVIRON["id"] " echo " $0 "\"") }
        '
        lf -remote "send $id renew"
        lf -remote "send $id echo copying finished."
    fi
}}
map P paste-async

This seems to work as I expected when I paste this in a shell with a hand written $id but does not work as a command somehow. It is likely related to subshells or buffers when a process is executed in go or maybe a bug I can't see at the moment. What it does is to read progress of rsync command line by line and displays it in the executing client as a message asynchronously every second. There are two complications in the above code:

• nc is neccessary for now since lf -remote does not support reading back responses yet.
• awk was necessary to read output line by line using carriage returns. I have also tried rsync .. | tr '\r' '\n' | while read line.. but there is an issue about buffering which I can't get around. There is probably an easier way to do this.

This approach have the following advantages/disadvantages:

• It is completely optional, it does not require implementing something new, and does not add an extra dependency
• It is a separate command than a regular paste so users need to remember using async version for long running operations (although this is similar to shell use, it may not be as good as ranger's way for some)
• It is not trivial to write such commands (although we can provide these examples)
• Messages can get lost if the user is moving at the same time

For the last point I think we can add a builtin command such as progress to be used instead of echo to display a colored progress bar as in ranger so that messages do not overlap with progress. Or maybe we can change the ui a little bit to display such progress messages (e.g. dividing the statusline into two parts).

[dumblob]

@gokcehan I'm fine with the "write your own copy/move/... function" solution. Once lf -remote supports reading responses, I'll provide my scripts which will be highly portable across many systems (including e.g. systems completely lacking awk etc.) and safe (e.g. avoiding echo, avoiding unquoted variables, etc.).

[gokcehan]

I have now added builtin copy and move operations with progress bars which is available in r11. Closing this issue now.