Support blocking monitor loop and full scan separation #2519

JC-comp · 2023-10-18T09:40:24Z

Summary

This is an implementation ~~for replacing polling monitor loop~~ that resolves ~~part of~~ #2518

Changes

Replace polling monitor loop with blocking wait
Check inotify events in separate threads and send signals to main thread when changes are available

How Has This Been Tested?

build and run in monitor loop
test local change handling trigger: create/delete/move local file/folder
test remote sync trigger: wait for next monitor_interval
test shutdown capability: monitor_max_loop set to 1 and 2
test integration:
- sync mode
- local first
- upload only
- download only
test stability: run 6 300-sec-interval monitor loop

What is still missing?

Error handling logs (out of scope)
Checking if the interrupt triggered at src/monitor.d is mandatory (does not affect functionality)

Effect
Remove the following inotify polling and sync checking, sleep every second

clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda2aa2490) = 0
poll([{fd=6, events=POLLIN}], 1, 0)     = 0 (Timeout)
....

abraunegg · 2023-10-18T18:51:19Z

@JC-comp
Many thanks for your efforts here, however I have just tried to clone this PR as it is, and compile to test locally and I am unable to do so:

Cloning into 'onedrive-pr2519'...
remote: Enumerating objects: 6907, done.
remote: Counting objects: 100% (1953/1953), done.
remote: Compressing objects: 100% (288/288), done.
remote: Total 6907 (delta 1754), reused 1802 (delta 1663), pack-reused 4954
Receiving objects: 100% (6907/6907), 4.63 MiB | 10.03 MiB/s, done.
Resolving deltas: 100% (4886/4886), done.
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Total 5 (delta 4), reused 5 (delta 4), pack-reused 0
Unpacking objects: 100% (5/5), 1.77 KiB | 908.00 KiB/s, done.
From https://github.com/abraunegg/onedrive
 * [new ref]         refs/pull/2519/head -> pr2519
Switched to branch 'pr2519'
checking for a BSD-compatible install... /usr/bin/install -c
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for dmd... ldc2
checking version of D compiler... 1.18.0
checking for curl... yes
checking for sqlite... yes
checking for notify... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating contrib/pacman/PKGBUILD
config.status: creating contrib/spec/onedrive.spec
config.status: creating onedrive.1
config.status: creating contrib/systemd/onedrive.service
config.status: creating contrib/systemd/onedrive@.service
rm -f onedrive onedrive.o version
rm -rf autom4te.cache
rm -f config.log config.status
if [ -f .git/HEAD ] ; then \
        git describe --tags > version ; \
else \
        echo v2.5.0-alpha-2 > version ; \
fi
ldc2  -w -g -O -J. -L-lcurl -L-lsqlite3  -L-ldl src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/progress.d src/monitor.d -ofonedrive
src/monitor.d(139): Error: shared method monitor.MonitorBackgroundWorker.watch is not callable using a non-shared object
src/monitor.d(143): Error: shared method monitor.MonitorBackgroundWorker.shutdown is not callable using a non-shared object
src/monitor.d(191): Error: shared method monitor.MonitorBackgroundWorker.initialise is not callable using a non-shared object
src/monitor.d(207): Error: shared method monitor.MonitorBackgroundWorker.shutdown is not callable using a non-shared object
src/monitor.d(278): Error: shared method monitor.MonitorBackgroundWorker.add is not callable using a non-shared object
src/monitor.d(322): Error: shared method monitor.MonitorBackgroundWorker.remove is not callable using a non-shared object
src/monitor.d(333): Error: shared method monitor.MonitorBackgroundWorker.remove is not callable using a non-shared object
/home/alex/dlang/ldc-1.18.0/bin/../import/std/concurrency.d(461): Error: static assert:  "Aliases to mutable thread-local data not allowed."
src/monitor.d(498):        instantiated from here: spawn!(void function(MonitorBackgroundWorker, Tid), MonitorBackgroundWorker, Tid)

Please can you look into this so that this can be merged into 'alpha-3'

JC-comp · 2023-10-18T22:38:43Z

I have just tried to clone this PR as it is, and compile to test locally and I am unable to do so

PR Updated.
Fixed for LDC compiler error.

checking for a BSD-compatible install... /usr/bin/install -c
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for dmd... ldc2
checking version of D compiler... 1.35.0
checking for curl... yes
checking for sqlite... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating contrib/pacman/PKGBUILD
config.status: creating contrib/spec/onedrive.spec
config.status: creating onedrive.1
config.status: creating contrib/systemd/onedrive.service
config.status: creating contrib/systemd/onedrive@.service
if [ -f .git/HEAD ] ; then \
        git describe --tags > version ; \
else \
        echo v2.5.0-alpha-2 > version ; \
fi
ldc2 -w -g -O -J. -L-lcurl -L-lsqlite3  -L-ldl src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/progress.d src/monitor.d -ofonedrive
done.

abraunegg · 2023-10-19T22:42:28Z

@JC-comp
Please can you rebase this PR against https://github.com/abraunegg/onedrive/tree/onedrive-v2.5.0-alpha-3 so that it can be merged into 'alpha-3' ?

Right now, there would be 10 conflicts to deal with.

JC-comp · 2023-10-20T10:52:03Z

Changes:

Add blocking wait support for webhook
Refine cleanup process when client exit or failure

JC-comp · 2023-10-22T08:47:12Z

Changes

Only handle changes from /delta when webhook events are received
Handle all inotify events and never discard them at any time to prevent missing local changes

now it resolves all features requested in #2518

abraunegg · 2023-10-24T21:39:01Z

@JC-comp
Could you please pull this commit from this PR and make this a separate PR: 7d11a8d

abraunegg · 2023-10-24T21:41:54Z

@JC-comp
This commit: 51163d5

Whilst valid, is on my whiteboard for a wider 'fix'.

In developing '2.5.x' I put in a lot of extra code to help ensure when things were being re-written, the code would 'break', then I would update various functions.

safeRename | safeBackup was two of the main ones.

If you could pull this commit from this PR that would be greatly appreciated.

* Simplify existing shutdown cleanup before #2519 is merged

JC-comp · 2023-10-25T04:57:22Z

@abraunegg

@JC-comp This commit: 51163d5

Whilst valid, is on my whiteboard for a wider 'fix'.

In developing '2.5.x' I put in a lot of extra code to help ensure when things were being re-written, the code would 'break', then I would update various functions.

safeRename | safeBackup was two of the main ones.

If you could pull this commit from this PR that would be greatly appreciated.

This commit is neccesary for 0d1b044 to work, we have to mark a local backup as copy instead of move, so that new items to be downloaded are not moved to the backup location remotely before downloading.

* Add JC-comp #2519 changes to 'alpha-4' minus the changes to 'sync.d' and 'util.d' as these are irrelevant

abraunegg · 2024-01-08T00:57:00Z

@JC-comp
This current PR (in it's existing state) is rather broken.

When this PR is used to upload files online, it constantly attempts to create the folder online. This was an issue identified in 'alpha-3' which is fixed in 'alpha-4':

No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
Monitoring directory: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
[M] Local directory created: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
Scanning the local file system 'random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe' for new data to upload ...
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
OneDrive Client requested to create this directory online: random_large_files/35CLWBm9Zc2hd0NRdn4m2jTBeHAwHyCe
^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data

Second point, your changes to 'sync.d' and 'util.d' are 100% irrelevant to the technical change you are suggesting with this PR.

When I take your PR (minus the changes to 'sync.d' and 'util.d') - this is what is PR #2582 essentially is.

Please either re-work this PR or re-work #2582

abraunegg · 2024-01-08T20:18:51Z

@JC-comp
Closing this in favour of #2582

abraunegg · 2024-01-16T01:34:41Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

JC-comp added 3 commits October 20, 2023 09:19

Replace inotify polling

116a7b9

Add webhook support for blocking wait

90c6200

Refine shutdown cleanup

7d11a8d

JC-comp force-pushed the blocking_loop branch from 8f74f40 to 7d11a8d Compare October 20, 2023 10:49

JC-comp changed the base branch from onedrive-v2.5.0-alpha-2 to onedrive-v2.5.0-alpha-3 October 20, 2023 10:50

JC-comp added 2 commits October 22, 2023 07:55

Seperate webhook callback

3db932e

Fix missing inotify event

0d1b044

JC-comp changed the title ~~Replace polling inotify with blocking version~~ Support blocking monitor loop and full scan separation Oct 22, 2023

Fix backup action

51163d5

abraunegg mentioned this pull request Oct 22, 2023

Fix api bug and curl issue #2523

Closed

abraunegg added a commit that referenced this pull request Oct 25, 2023

Update main.d

58d17d5

* Simplify existing shutdown cleanup before #2519 is merged

JC-comp and others added 2 commits October 25, 2023 04:45

Merge 58d17d5

8ac4038

Merge branch 'onedrive-v2.5.0-alpha-3' into blocking_loop

9f0019b

Fix retry interval wait

4aa32d2

abraunegg added a commit that referenced this pull request Jan 8, 2024

Add JC-comp #2519 changes to 'alpha-4'

7b2e59f

* Add JC-comp #2519 changes to 'alpha-4' minus the changes to 'sync.d' and 'util.d' as these are irrelevant

abraunegg added More Information Needed On Hold On Hold and removed More Information Needed On Hold On Hold labels Jan 8, 2024

abraunegg closed this Jan 8, 2024

Repository owner locked and limited conversation to collaborators Jan 16, 2024

JC-comp deleted the blocking_loop branch January 20, 2024 14:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support blocking monitor loop and full scan separation #2519

Support blocking monitor loop and full scan separation #2519

JC-comp commented Oct 18, 2023 •

edited

Loading

abraunegg commented Oct 18, 2023

JC-comp commented Oct 18, 2023

abraunegg commented Oct 19, 2023

JC-comp commented Oct 20, 2023

JC-comp commented Oct 22, 2023

abraunegg commented Oct 24, 2023

abraunegg commented Oct 24, 2023

JC-comp commented Oct 25, 2023

abraunegg commented Jan 8, 2024 •

edited

Loading

abraunegg commented Jan 8, 2024

abraunegg commented Jan 16, 2024

Support blocking monitor loop and full scan separation #2519

Support blocking monitor loop and full scan separation #2519

Conversation

JC-comp commented Oct 18, 2023 • edited Loading

Summary

abraunegg commented Oct 18, 2023

JC-comp commented Oct 18, 2023

abraunegg commented Oct 19, 2023

JC-comp commented Oct 20, 2023

JC-comp commented Oct 22, 2023

abraunegg commented Oct 24, 2023

abraunegg commented Oct 24, 2023

JC-comp commented Oct 25, 2023

abraunegg commented Jan 8, 2024 • edited Loading

abraunegg commented Jan 8, 2024

abraunegg commented Jan 16, 2024

JC-comp commented Oct 18, 2023 •

edited

Loading

abraunegg commented Jan 8, 2024 •

edited

Loading