Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First-sync optimization #66

Open
eric-void opened this issue Nov 4, 2020 · 2 comments
Open

First-sync optimization #66

eric-void opened this issue Nov 4, 2020 · 2 comments
Labels
enhancement New feature or request

Comments

@eric-void
Copy link

eric-void commented Nov 4, 2020

Command line help for flag "--first-sync" says: "First run setup. WARNING: Path2 files may overwrite path1 versions. Consider using with --dry-run first. Also asserts --verbose."

Looking at full documentation (and by checking the code) this is wrong.
Path1 file may overwrite path2 versions!

Another thing: looking at the code there is a lot of work to copy Path2 files not in Path1, and then do an "rclone sync" from Path1 to Path2. Why don't you just do and "rclone copy" from Path1 to Path2, and then an "rclone copy" from Path2 to Path1?
Something like this:

        logging.info(">>>>> --first-sync copying Path1 to Path2")
        # NOTE:  --min-size 0 added to block attempting to overwrite Google Doc files which have size -1 on Google Drive.  180729
        if rclone_cmd('copy', path1_base, path2_base, filter_file=user_filter_file, options=switches + ['--min-size', '0']):
            return RTN_CRITICAL
        logging.info(">>>>> --first-sync copying Path2 to Path1")
        # NOTE:  --min-size 0 added to block attempting to overwrite Google Doc files which have size -1 on Google Drive.  180729
        if rclone_cmd('copy', path2_base, path1_base, filter_file=user_filter_file, options=switches + ['--min-size', '0']):
            return RTN_CRITICAL
        logging.info(">>>>> --first-sync generating lsl files")
        if rclone_lsl(path1_base, path1_lsl_file, filter_file=user_filter_file):
            return RTN_CRITICAL
        if rclone_lsl(path2_base, path2_lsl_file, filter_file=user_filter_file):
            return RTN_CRITICAL

This is much simpler, there is no need to read 2 times LSL files and to loop for missing Path2 files, and the result should be the same.
Am i missing something?

@cjnaz
Copy link
Owner

cjnaz commented Nov 6, 2020

Thanks Eric. The command line doc error is a remnant from the algo rework in V3.0. With regard to the first-sync algo, you are probably correct, and as noted this should eliminate two LSL reads. I'll incorporate this change in the next rev, after I think carefully about any corner cases.

@cjnaz cjnaz added enhancement New feature or request documentation Doc clarification or fix labels Nov 6, 2020
@cjnaz
Copy link
Owner

cjnaz commented Dec 2, 2020

V3.2 released. Corrected the command line doc info, but deferred the first-sync optimization for now, since its not broke and should be rarely run. I'll leave this open, tagged for future enhancement.

@cjnaz cjnaz removed the documentation Doc clarification or fix label Dec 2, 2020
@cjnaz cjnaz changed the title --first-sync command line help is wrong (and a note on first-sync code) First-sync optimization Mar 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants