Unexpected path edits for SFTP remotes prevents successful runs

[jayjlawrence]

First I wish to start with a thanks to Chris et al for their work on this code. I am not a python programmer so I do not have a pull request.

I am using rclonesync to sync between a few local machines but ran into unexpected issues. I am already successful in using rclone to do unidirectional sync's like this:

rclone Projects remote:Projects -l

rclonesync.py called like this fails:

rclonesync.py --first-sync Projects remote:Projects --rclone-args -l

Complaining about an unknown directory. (not sure which - the logging does not tell us) Also not sure about what rclone command is being run I added some logging to tell me what's going on.

I discover that the rclone command being invoked is:

rclone -l Projects/ remote:/Projects/

What's with those slashes!?! This is very different than I would expect given I have invoked both programs similarly. Perhaps for other remotes require the leading and trailing slashes. IMHO the "pathparse" routine transforms my paths in unexpected and undesirable ways. My vote would be to defeat these transformations and use the exact same path syntax as rclone. Alternatively this needs to be clearly documented at the outset to advise users that there are differences and what to expect. Finally the current set up prevents me from using the "home" directory of the remote and I forced to use an absolute path.

Additionally I would recommend adding to the logging for rclone_cmd and rclone_lsl. I needed an additional logging statement to output the actual command being invoked.

Although a Ruby programmer I whacked in the following line after line 186 which reads logging.error(print_msg("ERROR", "rclone lsl failed. Specified path invalid? (Line {})".format(linenum)))

added statement:

     logging.error(print_msg("ERROR", "rclone command: {}".format(process_args)))

I'm sure I didn't count the number of spaces correctly so I may have this in the wrong scope. It is meant to be part of the rclone_lsl function and not between the definitions of rclone_lsl and rclone_cmd. :-}

[cjnaz]

Hello JayJ,
Let's try to get to the heart of the actual problem. I see that you are passing the symbolic link switch to rclone. Please run a test between your machines without the --rclone-args -l to confirm this isn't part of the issue. From your issue post I don't have enough info to help. Please post a shortened/sanitized version of the output with -v --rc-verbose --rc-verbose that gives a clue what the misbehavior is.

• Background RE the slashes... The 'transformations' should be benign. rclonesync takes absolute or relative paths to the Path1 and Path2. These paths are to the roots of the sync points. If you specify remote: then you are synching to the top level (your home?) of the remote, which is equivalent to remote:/. remote:Projects is equivalent to remote:/Projects/. There is no way to reference a relative path on a remote (or am I missing something?). The semantics are the same with rclone itself. rclonesync does these transformations to keep the code path handling simpler.

In your invocation rclonesync.py --first-sync Projects remote:Projects --rclone-args -l, Projects must be a subdirectory of the cwd, since this is relative path. If Projects is elsewhere then specify a relative path from the cwd, or specify the full/absolute path.

If I'm not understanding what path you are trying to specify, please clarify.

Also, please see the TROUBLESHOOTING.md, which might have a few clues of use.

[cjnaz]

There is no way to reference a relative path on a remote (or am I missing something?).

An rclone STFP remote is new to me. I see that the default root is the user's home path, and I see that the / that rclonesync inserts after the remote colon will screw that up. Let me think about if that slash can be cleanly removed for all cases.

[cjnaz]

Please check out branch v2.10. I've removed the force leading slash on remote, and implemented the keyboard interrupt handler.

• Tested on Linux with Python 3.7.3 and Dropbox.
• I do not currently have a SFTP remote to test with.
• Not tested on Windows or Python 2.7.

[cjnaz]

@jayjlawrence - any feedback?

[jayjlawrence]

(rats - had a response and forgot to commit it)

One point about adding the trailing slashes. I do not believe they are a material concern with rclone. What I can off is that with rsync the trailing slashes are very significant. Because of that experience I am personally apprehensive with the idea of amending the user's supplied paths. I mention this strictly as an informational note that may come in handy when a type of remote is very sensitive to slashes and where they are placed.

Thank you again and I hope my Ruby-centric viewpoint (and occasional uncalled-for comment) is seen as a friendly joke ... I find Python programmers as a group so serious I j! ;-)

[cjnaz]

Thanks Jay. I trust that this is now working well for your SFTP remotes? Be aware that rclonesync works not so well for active filesystems. See issue #24 . I see you're syncing a SQL db. Your results may vary.

regards,
cjn

[cjnaz]

v2.10 published