Add support for batched local monitor processing

[JC-comp]

This PR replaces the original one-by-one processing of the local file system monitor with batched processing for a single update, improving speed and consistency during periods of rapid changes.

Changes

• Batch local operations until no operations occur within one second into a single batch
• Reorder processing sequence in a batch to prevent data loss
• Upload changed local files using multithreading

Use case examples

1. multithreaded uploading
   $ touch 1 2 3 4 5
   • old version

     New items to upload to OneDrive: 1
     Uploading new file ./1 ... done.
     New items to upload to OneDrive: 1
     Uploading new file ./2 ... done.
     New items to upload to OneDrive: 1
     Uploading new file ./3 ... done.
     New items to upload to OneDrive: 1
     Uploading new file ./4 ... done.
     New items to upload to OneDrive: 1
     Uploading new file ./5 ... done.
     

   • PR version

     [M] Total number of local file changed: 5
     New items to upload to OneDrive: 5
     Uploading new file ./4 ... done.
     Uploading new file ./1 ... done.
     Uploading new file ./3 ... done.
     Uploading new file ./2 ... done.
     Uploading new file ./5 ... done.
     

2. Prevent data loss
   touch 1.txt && sleep 5 && echo "123" > 1.txt && mv 1.txt 2.txt
   • old version

     New items to upload to OneDrive: 1
     Uploading new file ./1.txt ... done.
     // sleeping
     Changed local items to upload to OneDrive: 1
     Cannot upload file changes/creation: ./2.txt: No such file or directory
     Cannot upload file changes/creation: ./1.txt: No such file or directory
     Moving ./1.txt to ./2.txt
     

   • PR version

     [M] Total number of local file changed: 1
     New items to upload to OneDrive: 1
     Uploading new file ./1.txt ... done.
     // sleeping
     Moving ./1.txt to ./2.txt
     [M] Total number of local file changed: 1
     Uploading modified file ./2.txt ... done.
     

[abraunegg]

@JC-comp
Please re-base / fix conflicts

[JC-comp]

Conflicts fixed

[abraunegg]

@JC-comp
This PR is producing odd application output.

All directories should be created first (typically right away, not batched) so that they are 100% available online.

This current PR, makes it appear that they are potentially made, but the inotify event for the directory creation is being handled last.

Please look into this and potentially change your ordering / processing so that directories are processed first, before files are being uploaded.

Evidence:

Sync with Microsoft OneDrive is complete
Monitoring directory: ./random_100_files
Monitoring directory: random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx
Monitoring directory: random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD
Monitoring directory: random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj
Monitoring directory: random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz
Monitoring directory: random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp
Monitoring directory: random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3
Monitoring directory: random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w
Monitoring directory: random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL
Monitoring directory: random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf
Monitoring directory: random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD
[M] Local directory created: ./random_100_files
Scanning the local file system './random_100_files' for new data to upload
OneDrive Client requested to create this directory online: ./random_100_files
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files
Successfully created the remote directory ./random_100_files on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx
Successfully created the remote directory ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD
Successfully created the remote directory ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj
Successfully created the remote directory ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz
Successfully created the remote directory ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp
Successfully created the remote directory ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3
Successfully created the remote directory ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3 on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w
Successfully created the remote directory ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL
Successfully created the remote directory ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf
Successfully created the remote directory ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD
The requested directory to create was not found on OneDrive - creating remote directory: ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD
Successfully created the remote directory ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD on Microsoft OneDrive
New items to upload to OneDrive: 100
Total New Data to Upload:        53 MB
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file3.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file0.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file4.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file5.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file2.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file7.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file1.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file6.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file2.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file5.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file4.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file9.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file3.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file0.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file1.data ... done.
Uploading new file ./random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx/file8.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file9.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file1.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file2.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file6.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file0.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file7.data ... done.
Uploading new file ./random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD/file8.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file3.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file1.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file9.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file5.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file8.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file4.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file0.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file6.data ... done.
Uploading new file ./random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj/file7.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file6.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file2.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file7.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file5.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file9.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file8.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file3.data ... done.
Uploading new file ./random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz/file4.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file1.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file0.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file7.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file2.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file3.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file4.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file5.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file6.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file3.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file0.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file1.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file9.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file5.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file2.data ... done.
Uploading new file ./random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp/file8.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file4.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file8.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file2.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file9.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file1.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file3.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file6.data ... done.
Uploading new file ./random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3/file7.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file0.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file1.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file6.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file0.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file5.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file4.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file8.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file7.data ... done.
Uploading new file ./random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w/file9.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file2.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file9.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file7.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file6.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file8.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file5.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file3.data ... done.
Uploading new file ./random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL/file4.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file1.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file7.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file2.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file6.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file5.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file0.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file3.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file4.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file1.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file8.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file5.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file2.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file3.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file4.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file0.data ... done.
Uploading new file ./random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf/file9.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file6.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file9.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file7.data ... done.
Uploading new file ./random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD/file8.data ... done.
[M] Local directory created: random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx
Scanning the local file system 'random_100_files/Ga15swtPfTdtqT9jSnQHImYI5afUPfHx' for new data to upload
[M] Local directory created: random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD
Scanning the local file system 'random_100_files/edtVCv40Lwb3ePXtKyfClbBd7coQ5DCD' for new data to upload
[M] Local directory created: random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj
Scanning the local file system 'random_100_files/ciWtQY7vMxjFiZ0BoQDls2hpQ2PXDDkj' for new data to upload
[M] Local directory created: random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz
Scanning the local file system 'random_100_files/s3vTwyfPC6gRnbL3iji2Tg0IK8ekvBjz' for new data to upload
[M] Local directory created: random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp
Scanning the local file system 'random_100_files/hxoaYQWVjUTXm1X07g90OAS2gV0SsIbp' for new data to upload
[M] Local directory created: random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3
Scanning the local file system 'random_100_files/xGdnInAiHc0BZiuk56gmDrspfmZtk5y3' for new data to upload
[M] Local directory created: random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w
Scanning the local file system 'random_100_files/SOdmKPdweVcamTo1KD4fSUotsd8TWC2w' for new data to upload
[M] Local directory created: random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL
Scanning the local file system 'random_100_files/WZx1HDCuvOyAQwtPKpNPLtpwE601scgL' for new data to upload
[M] Local directory created: random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf
Scanning the local file system 'random_100_files/i1dXmMeKJuwcT2hB2VtHYbVa7Ww7L1sf' for new data to upload
[M] Local directory created: random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD
Scanning the local file system 'random_100_files/45oL09hYGp17ft7bsaG6oX3020cfRbgD' for new data to upload
[M] Total number of local file changed: 9

[JC-comp]

This behavior aligns with the original version's expectations, where some subfolders are processed last. There is no need to rearrange all folders to be processed first, as it will not cause any issues. Folders will still be guaranteed to be created before files are uploaded.

Testing command: mkdir 1 && touch 1/1 && mkdir 1/3/ && touch 1/3/3 && mkdir 1/4/ && touch 1/3/4
alpha-5 result

Monitoring directory: ./1
[M] Local directory created: ./1
Scanning the local file system './1' for new data to upload
OneDrive Client requested to create this directory online: ./1
The requested directory to create was not found on OneDrive - creating remote directory: ./1
Successfully created the remote directory ./1 on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./1/4
The requested directory to create was not found on OneDrive - creating remote directory: ./1/4
Successfully created the remote directory ./1/4 on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./1/3
The requested directory to create was not found on OneDrive - creating remote directory: ./1/3
Successfully created the remote directory ./1/3 on Microsoft OneDrive
New items to upload to OneDrive: 3
Total New Data to Upload:        0 Bytes
Uploading new file ./1/1 ... done.
Uploading new file ./1/3/3 ... done.
Uploading new file ./1/3/4 ... done.
Monitoring directory: 1/3
[M] Local directory created: 1/3
Scanning the local file system '1/3' for new data to upload
Monitoring directory: 1/4
[M] Local directory created: 1/4
Scanning the local file system '1/4' for new data to upload

[abraunegg]

Why scan the entire filesystem, rather than pushing and using the uploadNewFile function directly?

[JC-comp]

The scanLocalFilesystemPathForNewData function can handle file uploads as well. In this new version, for cases like mkdir new_folder && mv new_folder new_loc, new_folder will not trigger a redundant online creation (as local dir is gone). Consequently, it will not be in sync and will require a filesystem scan.

[abraunegg]

@JC-comp
Currently in this PR, you have the following calculation (from main.d):

addLogEntry("[M] Total number of local file changed: " ~ to!string(changedLocalFilesToUploadToOneDrive.length));

One of the whiteboard items I have is how to integrate 'classify_as_big_delete' when using --monitor mode to prevent accidental massive deletion.

Is there a way for you to potentially integrate checking this config option against this total number of files|folders changed, so that the application stops the processing of the massive online delete ?

Again this comes back to the processing order - directories should always be first, thus you can calculate the children of the path being deleted and negate that delete from being sent online if required.

Please have a think about this on how this can be done with this PR.

[JC-comp]

Given that each file/folder deletion triggers a separate inotify event, a naive implementation is to sum up these events and compare them with the classify_as_big_delete flag.
Then what actions should be taken when hitting the limit? Is it appropriate to sleep in a while loop and wait for users to take actions and restart in monitor mode?

[abraunegg]

Given that each file/folder deletion triggers a separate inotify event, a naive implementation is to sum up these events and compare them with the classify_as_big_delete flag. Then what actions should be taken when hitting the limit? Is it appropriate to sleep in a while loop and wait for users to take actions and restart in monitor mode?

At the moment, in v2.4.x here is no action - the delete is processed, but, v2.4.x and v2.5.x (before this PR) processes each delete action one at a time, making 'classify_as_big_delete flag' rather impossible.

The only thought I had was - could something be done here? In Windows, a local delete of volumes of data, deletes online without any prevention.

For the moment, unless some other idea can be found, I am OK to have the existing situation carried forward for --monitor mode and just need to make sure that the documentation specifies that option does not apply to --monitor operations.

[abraunegg]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.