OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait #2582

abraunegg · 2024-01-07T22:46:26Z

Summary

This is an implementation that resolves #2518

Changes

Replace polling monitor loop with blocking wait
Check inotify events in separate threads and send signals to main thread when changes are available
Add blocking wait support for webhooks
Only handle changes from /delta when webhook events are received
Handle all inotify events and never discard them at any time to prevent missing local changes

How Has This Been Tested?

build and run in monitor loop
test local change handling trigger: create/delete/move local file/folder
test remote sync trigger: wait for next monitor_interval
test shutdown capability: monitor_max_loop set to 1 and 2
test integration:
- sync mode
- local first
- upload only
- download only
test stability: run 6 300-sec-interval monitor loop

What is still missing?

Checking if the interrupt triggered at src/monitor.d is mandatory (does not affect functionality)

Effect
Remove the following inotify polling and sync checking, sleep every second

clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
....

commit 1eff2d7 Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 18 19:15:27 2023 +1100 Update PR * Add --source-directory 'path/as/source/' --destination-directory 'path/as/destination' functionality commit ad3ddee Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 18 17:32:24 2023 +1100 Update PR * Add --create-directory * Add --remove-directory commit 7dfe6b6 Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 18 12:27:03 2023 +1100 Update PR * Update PR commit 75c071e Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 18 10:12:05 2023 +1100 Update PR * Update PR commit 6db484c Author: abraunegg <alex.braunegg@gmail.com> Date: Mon Oct 16 17:01:25 2023 +1100 Update PR * Update PR commit d893ea5 Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 11 10:43:50 2023 +1100 Update PR * Update PR commit 82bd593 Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 11 09:14:17 2023 +1100 Update PR * Validate and document --auth-files operation commit c551203 Author: abraunegg <alex.braunegg@gmail.com> Date: Wed Oct 11 05:48:22 2023 +1100 Update PR * Add --create-share-link commit fbf6399 Author: abraunegg <alex.braunegg@gmail.com> Date: Tue Oct 10 18:39:21 2023 +1100 Update PR * Update PR commit 72a4680 Author: abraunegg <alex.braunegg@gmail.com> Date: Tue Oct 10 17:43:15 2023 +1100 Update PR * Add --get-file-link * Add --modified-by commit 0d3fc3e Author: abraunegg <alex.braunegg@gmail.com> Date: Tue Oct 10 14:28:10 2023 +1100 Add --display-sync-status * Add --display-sync-status commit 1f183ca Author: abraunegg <alex.braunegg@gmail.com> Date: Mon Oct 9 08:18:13 2023 +1100 Update PR * Update PR with doc updates commit b0628d7 Author: abraunegg <alex.braunegg@gmail.com> Date: Sun Oct 8 10:52:52 2023 +1100 Update PR * Update PR commit 7e3df95 Author: abraunegg <alex.braunegg@gmail.com> Date: Sat Oct 7 05:31:26 2023 +1100 Update PR * Update PR commit c69f2ab Author: abraunegg <alex.braunegg@gmail.com> Date: Sat Oct 7 05:28:28 2023 +1100 Update PR * Update PR commit ea1ca33 Author: abraunegg <alex.braunegg@gmail.com> Date: Fri Oct 6 14:57:51 2023 +1100 Update PR * Update PR commit 1503f96 Author: abraunegg <alex.braunegg@gmail.com> Date: Fri Oct 6 09:19:04 2023 +1100 Update PR * Update PR commit 5127464 Author: abraunegg <alex.braunegg@gmail.com> Date: Fri Oct 6 06:48:20 2023 +1100 Change when the integrity check is performed * Change when the integrity check is performed commit c7cc45d Author: abraunegg <alex.braunegg@gmail.com> Date: Thu Oct 5 19:40:05 2023 +1100 Update maxInotifyWatches location * Update maxInotifyWatches location commit c44ad96 Author: abraunegg <alex.braunegg@gmail.com> Date: Thu Oct 5 17:41:31 2023 +1100 Update main.d * Fix --version segfault commit 51f0ffc Author: abraunegg <alex.braunegg@gmail.com> Date: Thu Oct 5 17:24:30 2023 +1100 Uplift to v2.5.0-alpha-2 * Uplift to v2.5.0-alpha-2 commit cbe3e6e Author: abraunegg <alex.braunegg@gmail.com> Date: Thu Oct 5 17:17:26 2023 +1100 Clean up before onedrive-v2.5.0-alpha-2 * Clean up before onedrive-v2.5.0-alpha-2

* Uplift to v2.5.0-alpha-3

* Add webhook functionality back in

* Fix that the '.' were not being printed in --verbose mode for fetching the /delta response

* Update webhook feature with #2516 changes

* Add documentation notes for webhooks * Fix variable naming

* --download-only and --local-first cannot be used together

* Fix onedrive -s --download-only * Fix onedrive -s --download-only --local-first

* Add a check when using --download-only to test if the file already exists locally, and if it does, has the local file been modified since the file was last downloaded. If it has, then rename the existing file to preserve it to prevent local data loss

* Implement --display-quota

* Add missing curlEngine.http.clearRequestHeaders(); from 'patch' and 'post'

@JC-comp

* Cherry pick 5a4a351 Fix by @JC-comp

* Fix curl reusing socket

* Update podman documentation

* Update doc

* Wait for all parallel jobs that depend on the database to complete - added from f86c4b9

* Add debug output to determine DB access threadsafe value

* Simplify existing shutdown cleanup before #2519 is merged

* Update doc, add getenforce check validation

* Update exit scope handling, ensure that oneDriveApiInstance is shutdown otherwise memory leak occurs on exit PRECHANGE: ==442773== LEAK SUMMARY: ==442773== definitely lost: 0 bytes in 0 blocks ==442773== indirectly lost: 0 bytes in 0 blocks ==442773== possibly lost: 32 bytes in 1 blocks ==442773== still reachable: 175,810 bytes in 13 blocks ==442773== suppressed: 0 bytes in 0 blocks POSTCHANGE: ==450812== LEAK SUMMARY: ==450812== definitely lost: 0 bytes in 0 blocks ==450812== indirectly lost: 0 bytes in 0 blocks ==450812== possibly lost: 32 bytes in 1 blocks ==450812== still reachable: 153,856 bytes in 5 blocks ==450812== suppressed: 0 bytes in 0 blocks

* Fix exit coredump when using -s -v

Update --display-quota output to: Deleted: 8.19 GB (8793627814 bytes) Remaining: 25580.69 GB (27467060611614 bytes) State: normal Total: 25600.00 GB (27487790694400 bytes) Used: 11.12 GB (11936454972 bytes)

* Import Lyncredible@d9fee18 manually

* Fix memory leak on exit when using webhooks

* When --syncdir is being used, and no initial config file exists, write the --syncdir value to a new default configuration file, so that when the client is run again, the --syncdir value is used. If an existing config file is present, update the sync_dir config value that was passed in, but only if it is different to the existing set value

* Update PR to add a test for entrypoint.sh to exist, to assist with Docker first run scenario

* Update Podman and Docker documentation

* Update doc

* Add 'Using 'Client Side Filtering' rules to determine what should be synced with Microsoft OneDrive'

* Add TOC link

* Add note regarding Client Side Filtering

* Update usage docs

* Update function comments * Define a better function and use for checking for HTML URL encoded items in API response JSON elements

* Update doc

* Add back in missing note when application is authorised

* Fix up large file handling output * Fix calculation error for upload fragments

* Catch an error when the file being uploaded disappears during session upload

* Update doc

* Comment updates

* Remove CurlOption.ssl_verifypeer configuration which was added when developing the multi-threaded transfer handling and OpenSSL was causing issues due to insufficient file handles to open the read of the CA Certificate. If the SSL CA Certificate cannot be read, we should hard exit.

* Add JC-comp #2519 changes to 'alpha-4' minus the changes to 'sync.d' and 'util.d' as these are irrelevant

Pull in e32d32e

* Add missing log output for new file adds

abraunegg · 2024-01-08T19:50:44Z

@cholzer79 , @Lyncredible

Given the current maturity of 'alpha-4' - are you potentially able to test this specific PR which has changes specifically impacting the current 'WebHooks' feature?

It would be great if you are able to test this and provide some feedback based on using Webhooks with this PR.

* Use variable as set

Lyncredible · 2024-01-09T14:39:55Z

@abraunegg Sorry but I am quite busy in Jan. Will give it a try when I find some free time but no promises...

abraunegg · 2024-01-13T20:22:29Z

@Lyncredible

@abraunegg Sorry but I am quite busy in Jan. Will give it a try when I find some free time but no promises...

When you do get to review, please review 'alpha-5' as I am going to close off this PR.

The 'alpha-5' PR is: #2584

abraunegg · 2024-01-13T20:23:18Z

Typo: The 'alpha-5' PR is: #2584

abraunegg · 2024-01-27T01:26:01Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

abraunegg and others added 30 commits October 19, 2023 05:31

Uplift to v2.5.0-alpha-3

35edb4c

* Uplift to v2.5.0-alpha-3

Add webhook functionality back in

3a3c954

* Add webhook functionality back in

Update sync.d

d653ff3

* Fix that the '.' were not being printed in --verbose mode for fetching the /delta response

Update webhook feature with #2516 changes

158a571

* Update webhook feature with #2516 changes

Update PR

1449855

* Add documentation notes for webhooks * Fix variable naming

Update config.d

25015da

* --download-only and --local-first cannot be used together

Update PR

2257f0b

* Fix onedrive -s --download-only * Fix onedrive -s --download-only --local-first

Update sync.d

d7ea797

* Add a check when using --download-only to test if the file already exists locally, and if it does, has the local file been modified since the file was last downloaded. If it has, then rename the existing file to preserve it to prevent local data loss

Implement --display-quota

77c294b

* Implement --display-quota

Update onedrive.d

5fddd74

* Add missing curlEngine.http.clearRequestHeaders(); from 'patch' and 'post'

Update onedrive.d

5dedd26

* Cherry pick 5a4a351 Fix by @JC-comp

Fix curl reusing socket (#2526)

1d29ca0

* Fix curl reusing socket

Update podman.md

7ffbd98

* Update podman documentation

Update podman.md

9837e59

* Update doc

Update main.d

b9f20cd

* Wait for all parallel jobs that depend on the database to complete - added from f86c4b9

Add debug output to determine DB access threadsafe value

b3a1e4a

* Add debug output to determine DB access threadsafe value

Update main.d

58d17d5

* Simplify existing shutdown cleanup before #2519 is merged

Update podman.md

a22e4e8

* Update doc, add getenforce check validation

Update main.d

8979bf7

* Fix exit coredump when using -s -v

Update FR #2359

16595ef

Update --display-quota output to: Deleted: 8.19 GB (8793627814 bytes) Remaining: 25580.69 GB (27467060611614 bytes) State: normal Total: 25600.00 GB (27487790694400 bytes) Used: 11.12 GB (11936454972 bytes)

Fix webhook application hang on exit

4df36ff

* Import Lyncredible@d9fee18 manually

Fix memory leak on exit when using webhooks

3798e1a

* Fix memory leak on exit when using webhooks

Update PR

677f076

* Update PR to add a test for entrypoint.sh to exist, to assist with Docker first run scenario

Update Podman and Docker documentation

f7aa1e8

* Update Podman and Docker documentation

Update docker.md

7fde0f4

* Update doc

Update docker.md

5ba96d2

* Update doc

Update docker.md

65cafa6

* Update doc

abraunegg added 14 commits January 4, 2024 08:46

Update usage.md

9dc4e54

* Add 'Using 'Client Side Filtering' rules to determine what should be synced with Microsoft OneDrive'

Update usage.md

6fbf7fa

* Add TOC link

Update readme.md

76706f5

* Add note regarding Client Side Filtering

Update usage.md

b5d4d47

* Update usage docs

Update code

019deda

* Update function comments * Define a better function and use for checking for HTML URL encoded items in API response JSON elements

Update usage.md

6bca81e

* Update doc

Update code

abe4439

* Add back in missing note when application is authorised

Fix up large file handling output

8a7a850

* Fix up large file handling output * Fix calculation error for upload fragments

Update sync.d

8f0a6d1

* Catch an error when the file being uploaded disappears during session upload

Update usage.md

8a075a2

* Update doc

Update usage.md

0b73ea6

* Update doc

Update sync.d

d05a8f5

* Comment updates

Add JC-comp #2519 changes to 'alpha-4'

7b2e59f

* Add JC-comp #2519 changes to 'alpha-4' minus the changes to 'sync.d' and 'util.d' as these are irrelevant

abraunegg mentioned this pull request Jan 8, 2024

Support blocking monitor loop and full scan separation #2519

Closed

abraunegg added More Information Needed On Hold On Hold labels Jan 8, 2024

abraunegg added 2 commits January 8, 2024 15:04

Fix 'Unexpected end of input ..' output message

9ec8e73

Pull in e32d32e

Update sync.d

4dd2f63

* Add missing log output for new file adds

abraunegg changed the title ~~Onedrive v2.5.0 alpha 4 add jc 2519v2~~ OneDrive Client Version v2.5.0-alpha-4 + JC-comp #2519: Replace polling monitor loop with blocking wait Jan 8, 2024

abraunegg changed the title ~~OneDrive Client Version v2.5.0-alpha-4 + JC-comp #2519: Replace polling monitor loop with blocking wait~~ OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait Jan 8, 2024

Update onedrive.d

e4552dd

* Use variable as set

abraunegg closed this Jan 13, 2024

abraunegg deleted the onedrive-v2.5.0-alpha-4-add-jc-2519v2 branch January 19, 2024 18:10

Repository owner locked and limited conversation to collaborators Jan 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait #2582

OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait #2582

abraunegg commented Jan 7, 2024 •

edited

Loading

abraunegg commented Jan 8, 2024

Lyncredible commented Jan 9, 2024

abraunegg commented Jan 13, 2024 •

edited

Loading

abraunegg commented Jan 13, 2024

abraunegg commented Jan 27, 2024

OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait #2582

OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait #2582

Conversation

abraunegg commented Jan 7, 2024 • edited Loading

Summary

abraunegg commented Jan 8, 2024

Lyncredible commented Jan 9, 2024

abraunegg commented Jan 13, 2024 • edited Loading

abraunegg commented Jan 13, 2024

abraunegg commented Jan 27, 2024

abraunegg commented Jan 7, 2024 •

edited

Loading

abraunegg commented Jan 13, 2024 •

edited

Loading