OneDrive Client Version v2.5.0-alpha-2 is now available for testing

[abraunegg]

As part of the ongoing development of v2.5.0, client version v2.5.0-alpha-2 is now available for testing.

Changelog from alpha-1:

• Added support for OneDrive SharePoint Libraries

Whats still missing?

• Logging output is still broken
• Websocket support is not yet added back in
• Some minor functionality is still being added back in
• Documentation updates are still being done

You can use your normal account if you wish (this is your choice), however this is 100% alpha quality software at the moment. If you experience data loss or something else with your real data - whilst not good, please take every effort to not use this code with your real data.

There will be functionality differences, there will be bugs or things that are not working right.

My ask is that you create a test report - what works, what does not work, what could be potentially changed | improved - basically I am soliciting some feedback on where this total re-write is heading as it matures to a v2.5.0 release.

How can you get v2.5.0-alpha-2 ?

You need to build this client from source. Follow the high level directions here to get the development requirements installed first: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#building-from-source---high-level-requirements

Next clone and checkout the PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2505/head:pr2505
git checkout pr2505
./configure; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

When running the PR, your version should be as per below or greater:

v2.5.0-alpha-2 GitHub version: v2.4.25-20-g7e3df95

The important point is v2.5.0-alpha-2. The v2.4.25-XX-abcdefg will change as new commits are put into the PR branch, whilst things are added/fixed/updated.

What is known to be not functioning right at this stage:

• Application logging output when downloading & uploading files is jumbled. The reason for this is all the parallel threads writing to terminal at the same time. This is even more evident when downloading or uploading files >4MB. This will not be fixed in alpha-2 but will need to be focused upon for future releases once feature parity with v2.4.25 is reached. Please do not raise an issue ticket because of this.
• Existing documentation does not match what v2.5.0 does or supports at present. It will be updated as development progresses.

I have no test files or test data - what can I use?

In developing this application I use a script as per below to generate random data:

#!/bin/bash
FILECOUNT=0
DIRCOUNT=0
DESTDIR="/my/path/to/my/sync_dir/folder/OneDrive/random_100_files"
FILES=10
FOLDERS=10
BLOCKSIZE=4

mkdir -p $DESTDIR

while [ $DIRCOUNT -lt $FOLDERS ] ; do
        DIRNAME=`cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1`
        DIRPATH=$DESTDIR/$DIRNAME
        mkdir -p $DIRPATH
        while [ $FILECOUNT -lt $FILES ] ; do
                FILESIZE=$RANDOM
                FILESIZE=$(($FILESIZE+1024))
                FILENAME=$DIRPATH/file${FILECOUNT}.data
                echo "Creating $FILENAME"
                dd if=/dev/urandom of=${FILENAME} bs=$BLOCKSIZE count=$FILESIZE > /dev/null 2>&1
                ((FILECOUNT++))
        done
        FILECOUNT=0
        ((DIRCOUNT++))
done

Update DESTDIR with the right directory and adjust FILES and FOLDERS as desired. The above settings will create 10 folders, with each folder having 10 files. with each file size between 6KB and 125KB.

To increase (or decrease) the random file size, change BLOCKSIZE, generally use ^2 values: 1,2,4,8,16,32,64,128,256 and so on.

At 128 you will get files from 100KB -> ~3.8MB, at 256 most files will be between 500KB --> 7.5MB

[tdcockers]

Testing now.

Platform is Nethserver 7 (Centos 7) that has previously had 2.4 running from source.

[root@dc01 system]# ~/onedrive/onedrive --version
v2.5.0-alpha-2 GitHub version: v2.4.25-14-gc44ad96

Installation of the was fine, it didn't add the symlinks for the systemwide variable but from the instructions above that looks to be correct.

I haven't been able to run it yet, getting the following errors:

[root@dc01 system]# ~/onedrive/onedrive --monitor --confdir="/root/.od-config/sf-scans"
Reading configuration file: /root/.od-config/sf-scans/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
The item database is incompatible, re-creating database table structures
Sync Engine Initialised with new Onedrive API instance
ERROR: OneDrive quota information is being restricted. Please fix by speaking to your OneDrive / Office 365 Administrator.
All application operations will be performed in: /var/lib/nethserver/ibay/sf-scans

ERROR: Unable to perform a database vacuum: out of memory

std.file.FileException@std/file.d(370): /proc/sys/user/max_inotify_watches: No such file or directory
----------------
/root/dlang/dmd-2.102.0/linux/bin64/../../src/phobos/std/file.d:247 @safe bool std.file.cenforce!(bool).cenforce(bool, scope lazy const(char)[], immutable(char)[], ulong) [0x65b34c]
??:? @trusted void[] std.file.readImpl(scope const(char)[], scope const(char)*, ulong) [0x714554]
/root/dlang/dmd-2.102.0/linux/bin64/../../src/phobos/std/file.d:319 @safe void[] std.file.read!(immutable(char)[]).read(immutable(char)[], ulong) [0x5d0d06]
/root/dlang/dmd-2.102.0/linux/bin64/../../src/phobos/std/file.d:516 @safe immutable(char)[] std.file.readText!(immutable(char)[], immutable(char)[]).readText(immutable(char)[]) [0x5d0ae1]
src/main.d:530 _Dmain [0x597cfe]

[abraunegg]

I haven't been able to run it yet, getting the following errors:

hrmm ... let me work on that ... as this exists on CentOS 8

[abraunegg]

@tdcockers
Could you rebuild the PR - the PR version should now be: v2.5.0-alpha-2 GitHub version: v2.4.25-15-gc7cc45d

You may need to delete your PR download location to ensure you get the updated build.

[abraunegg]

@tdcockers
As per running on CentOS 7:

Application version:  v2.5.0-alpha-2 GitHub version: v2.4.25-15-gc7cc45d
Account Type:         business
Default Drive ID:     <redacted>
Default Root ID:      <redacted>
Remaining Free Space: 1058325873 KB
All application operations will be performed in: /home/alex/OneDrive
OneDrive synchronisation interval (seconds): 300
Maximum allowed open files:                  183662
Maximum allowed inotify user watches:        8192

[alex@localhost onedrive-pr2505]$ uname -a
Linux localhost.localdomain 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[alex@localhost onedrive-pr2505]$ 

[tdcockers]

(dmd-2.102.0)[root@dc01 onedrive]# ./onedrive --version
v2.5.0-alpha-2 GitHub version: v2.4.25-15-gc7cc45d

Sync is running. I created an empty text file, which gave an odd error but did in fact upload the file:

[root@dc01 sf-scans]# touch timtest.txt
[root@dc01 sf-scans]# journalctl -f --unit="od-sf-scans"
-- Logs begin at Thu 2023-10-05 06:36:02 AEST. --
Oct 05 20:29:55 dc01.office onedrive[40437]: Fetching items from the OneDrive API for Drive ID: b!a-d--g1wJU6mXF5bI9JI9b1rkXQELydIjbr02G7uceI78Sr_Lbn3So8pdIcDfWEt ..
Oct 05 20:29:55 dc01.office onedrive[40437]: Processing changes and items received from Microsoft OneDrive ...
Oct 05 20:29:55 dc01.office onedrive[40437]: Performing a database consistency and integrity check on locally stored data ...
Oct 05 20:29:55 dc01.office onedrive[40437]: Scanning the local file system '/var/lib/nethserver/ibay/sf-scans' for new data to upload ...
Oct 05 20:29:55 dc01.office onedrive[40437]: Performing a final true-up scan of online data from Microsoft OneDrive
Oct 05 20:29:56 dc01.office onedrive[40437]: Fetching items from the OneDrive API for Drive ID: b!a-d--g1wJU6mXF5bI9JI9b1rkXQELydIjbr02G7uceI78Sr_Lbn3So8pdIcDfWEt ..
Oct 05 20:29:56 dc01.office onedrive[40437]: Processing changes and items received from Microsoft OneDrive ...
Oct 05 20:29:56 dc01.office onedrive[40437]: Sync with Microsoft OneDrive is complete
Oct 05 20:31:16 dc01.office onedrive[40437]: Scanning the local file system '/var/lib/nethserver/ibay/sf-scans' for new data to upload ...
Oct 05 20:31:17 dc01.office onedrive[40437]: Uploading new file ./timtest.txt ... done.
Oct 05 20:31:18 dc01.office onedrive[40437]: Line: 2428
Oct 05 20:31:18 dc01.office onedrive[40437]: Cannot upload file changes/creation: JSONValue is not an object

And my first attempt at renaming a folder has also been successful. Will attempt starting up another library with --download-only --cleanup-local-files and report separately shortly

[tdcockers]

Switching to --sync --download-only --cleanup-local-files got hung up here for 10 mins:

[root@dc01 system]# journalctl -f --unit="od-sf-clients"
-- Logs begin at Thu 2023-10-05 06:36:02 AEST. --
Oct 05 20:47:36 dc01.office.superfactory.com.au systemd[1]: Started OneDrive Free Client.
Oct 05 20:47:36 dc01.office.superfactory.com.au onedrive[43327]: Reading configuration file: /root/.od-config/sf-clients/config
Oct 05 20:47:36 dc01.office.superfactory.com.au onedrive[43327]: Configuration file successfully loaded
Oct 05 20:47:37 dc01.office.superfactory.com.au onedrive[43327]: Configuring Global Azure AD Endpoints
Oct 05 20:47:37 dc01.office.superfactory.com.au onedrive[43327]: WARNING: Application has been configured to cleanup local files that are not present online.
Oct 05 20:47:37 dc01.office.superfactory.com.au onedrive[43327]: WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.
Oct 05 20:47:37 dc01.office.superfactory.com.au onedrive[43327]: Sync Engine Initialised with new Onedrive API instance
Oct 05 20:47:37 dc01.office.superfactory.com.au onedrive[43327]: ERROR: OneDrive quota information is being restricted. Please fix by speaking to your OneDrive / Office 365 Administrator.
Oct 05 20:47:38 dc01.office.superfactory.com.au onedrive[43327]: All application operations will be performed in: /var/lib/nethserver/ibay/sf-clients
Oct 05 20:47:38 dc01.office.superfactory.com.au onedrive[43327]: Generating a /delta compatible JSON response from the OneDrive API ...

The server was not showing any activity on CPU or NICs, I'm not sure how long this would normally be expected to take (drive has about 90,000 files in it).

Switching to --monitor without --download-only --cleanup-local-files seemed to get it going. Took just over 10 mins to complete the 'Fetching items' step, but I could see network traffic on the server the whole time. Files that were incorrectly removed as discussed earlier (#2499) have been correctly recovered after about 15mins of processing changes, that seems pretty good for that many files. Monday when the office is back in full swing will be the real test.

Out of curiosity, Is the ability to use '--monitor --download-only` one of the features that will be implemented in due course, or is that not possible with the new base?

[tdcockers]

Client just did a crash and restart, but quite possibly environment related as I'm running this on a system with 4GB RAM and it's handling a big number of files. Errors attached
sf-clients error.txt

[tdcockers]

Tried bumping the system RAM to 6GB, just got the same error ERROR: Unable to perform a database vacuum: out of memory while scanning the local file system and uploading changes. Is that possibly to do with not enough watches available, rather than system memory?

[abraunegg]

@tdcockers

Switching to --sync --download-only --cleanup-local-files got hung up here for 10 mins:
...
Generating a /delta compatible JSON response from the OneDrive API ...

OK .. The client is generating a compatible /delta response because --cleanup-local-files is being used. As you are not using --verbose the client would not have been hung up - it was generating a /delta compatible response.

Please ensure you add --verbose so you can see what the client is doing as non-verbose mode will make it look like the client is stuck

Out of curiosity, Is the ability to use '--monitor --download-only` one of the features that will be implemented in due course, or is that not possible with the new base?

The options --monitor and --download-only are technically incompatible as the flags are opposite methods of activity. I know this is possible in v2.4.25 to use them together, however in this alpha build I am trying to limit odd combinations such as that until the client functionality is solid.

Took just over 10 mins to complete the 'Fetching items' step, but I could see network traffic on the server the whole time. Files that were incorrectly removed as discussed earlier (#2499) have been correctly recovered after about 15mins of processing changes, that seems pretty good for that many files. Monday when the office is back in full swing will be the real test.

That should be the correct experience, as you are using --cleanup-local-files thus forcing a /delta to be created. when not using that mode, it should be even faster as there have been significant performance improvements made.

Tried bumping the system RAM to 6GB, just got the same error ERROR: Unable to perform a database vacuum: out of memory while scanning the local file system and uploading changes. Is that possibly to do with not enough watches available, rather than system memory?

Possibly not. If you are running out of watches, the application should be advising that is the case (hence why you got that original error). There are potentially 2 issues going on here:

• some sort of file download|upload failure causing the integrity check to fail by the looks of it
• An exit that is causing the database not to be able to be written out to - the database object has already been purged, thus you get the error that it cant perform the vacuum

As this seems to be 100% repeatable for you (I have not seen these issues in my testing thus far), could you run the client in debug mode - refer to https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support for that process and could you compress & then email the debug log ?

I am going to switch to using a CentOS 7 system for some SharePoint testing and see if I can re-create some of these issues you are seeing.

[abraunegg]

@tdcockers
I am able to reproduce what you are seeing - no need for a debug log

[tdcockers]

OK, thanks. I've just reviewed the output from last night. Possibly related (or not) the client was regularly getting throttled - I'm using 300 second sync intervals - and periodically the vacuum error coincided with that. There was no file changes being made to the drive overnight.

Oct 06 05:41:49 dc01.office onedrive[36892]: Starting a sync with Microsoft OneDrive
Oct 06 05:41:49 dc01.office onedrive[36892]: Syncing changes from Microsoft OneDrive ...
Oct 06 05:42:13 dc01.office onedrive[36892]: ------------------------------------------------------------------
Oct 06 05:42:13 dc01.office onedrive[36892]: getDeltaQueryOneDriveApiInstance.viewChangesByItemId(selectedDriveId, selectedItemId, providedDeltaLink) generated a OneDriveException
Oct 06 05:42:13 dc01.office onedrive[36892]: selectedDriveId:   b!CDAxCQErP0Gwxv9JJcM3G71rkXQELydIjbr02G7uceKMngKcQwLtR5QYLTgth_em
Oct 06 05:42:13 dc01.office onedrive[36892]: selectedItemId:    01W36FZM56Y2GOVW7725BZO354PWSELRRZ
Oct 06 05:42:13 dc01.office onedrive[36892]: providedDeltaLink: https://graph.microsoft.com/v1.0/drives/b!CDAxCQErP0Gwxv9JJcM3G71rkXQELydIjbr02G7uceKMngKcQwLtR5QYLTgth_em/items/01W36FZM56Y2GOVW7725BZO354PWSELRRZ/delta?token=NDslMjM0OyUyMzE7Mzs5YzAyOWU4Yy0wMjQzLTQ3ZWQtOTQxOC0yZDM4MmQ4N2Y3YTY7NjM4MzIxMzE3MDg0MDAwMDAwOzM2ODQ4MzE5MjslMjM7JTIzOyUyMzA7JTIz
Oct 06 05:42:13 dc01.office onedrive[36892]: ------------------------------------------------------------------
Oct 06 05:42:13 dc01.office onedrive[36892]: Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Oct 06 05:42:13 dc01.office onedrive[36892]: Sleeping for 5 seconds
Oct 06 05:42:21 dc01.office onedrive[36892]: ERROR: Unable to perform a database vacuum: out of memory
Oct 06 05:42:21 dc01.office onedrive[36892]: onedrive.OneDriveException@src/onedrive.d(1349): HTTP request returned status code 429 ()
Oct 06 05:42:21 dc01.office onedrive[36892]: {
Oct 06 05:42:21 dc01.office onedrive[36892]: "error": {
Oct 06 05:42:21 dc01.office onedrive[36892]: "code": "activityLimitReached",
Oct 06 05:42:21 dc01.office onedrive[36892]: "innerError": {
Oct 06 05:42:21 dc01.office onedrive[36892]: "client-request-id": "ee4d707a-1b91-4132-8abe-53862f192985",
Oct 06 05:42:21 dc01.office onedrive[36892]: "code": "throttledRequest",
Oct 06 05:42:21 dc01.office onedrive[36892]: "date": "2023-10-05T19:42:18",
Oct 06 05:42:21 dc01.office onedrive[36892]: "innerError": {
Oct 06 05:42:21 dc01.office onedrive[36892]: "code": "quota"
Oct 06 05:42:21 dc01.office onedrive[36892]: },
Oct 06 05:42:21 dc01.office onedrive[36892]: "request-id": "ee4d707a-1b91-4132-8abe-53862f192985"
Oct 06 05:42:21 dc01.office onedrive[36892]: },
Oct 06 05:42:21 dc01.office onedrive[36892]: "message": "The request has been throttled"
Oct 06 05:42:21 dc01.office onedrive[36892]: }
Oct 06 05:42:21 dc01.office onedrive[36892]: }
Oct 06 05:42:21 dc01.office onedrive[36892]: ----------------
Oct 06 05:42:21 dc01.office onedrive[36892]: src/onedrive.d:42 void onedrive.OneDriveApi.checkHttpResponseCode(std.json.JSONValue) [0x6c69c4]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/onedrive.d:1007 std.json.JSONValue onedrive.OneDriveApi.get(immutable(char)[], bool) [0x6c4eab]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/onedrive.d:512 std.json.JSONValue onedrive.OneDriveApi.viewChangesByItemId(immutable(char)[], immutable(char)[], immutable(char)[]) [0x6c2ad2]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/sync.d:2186 std.json.JSONValue syncEngine.SyncEngine.getDeltaChangesByItemId(immutable(char)[], immutable(char)[], immutable(char)[]) [0x6d4536]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/sync.d:672 void syncEngine.SyncEngine.fetchOneDriveDeltaAPIResponse(immutable(char)[], immutable(char)[], immutable(char)[]) [0x6ce340]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/sync.d:455 void syncEngine.SyncEngine.syncOneDriveAccountToLocalDisk() [0x6cd714]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/main.d:884 void main.performStandardSyncProcess(immutable(char)[], monitor.Monitor) [0x599423]
Oct 06 05:42:21 dc01.office onedrive[36892]: src/main.d:723 _Dmain [0x598408]
Oct 06 05:42:21 dc01.office systemd[1]: od-sf-clients.service: main process exited, code=exited, status=1/FAILURE
Oct 06 05:42:21 dc01.office systemd[1]: Unit od-sf-clients.service entered failed state.
Oct 06 05:42:21 dc01.office systemd[1]: od-sf-clients.service failed.
Oct 06 05:42:24 dc01.office systemd[1]: od-sf-clients.service holdoff time over, scheduling restart.
Oct 06 05:42:24 dc01.office systemd[1]: Stopped OneDrive Free Client.
Oct 06 05:42:24 dc01.office systemd[1]: Started OneDrive Free Client.

[abraunegg]

periodically the vacuum error coincided with that

What appears to be happening is that the failure and exit scopes are being called - when the exit scope does its thing, that is what is causing that error message - an error yes, in output only at the moment as the call was done during the failure scope

[abraunegg]

@tdcockers
If you can rebuild the client with the latest update, what was happening was that I was being over ambitious on where an integrity check was being called - thus that led to the error. In alpha-1 that code was 'duplicated' many times, and in alpha-2 I have made this a function, which led to that error.

If you rebuild, you should get now v2.5.0-alpha-2 GitHub version: v2.4.25-16-g5127464

I am testing a 100K fileset against a SharePoint Library at the moment

[tdcockers]

Running v2.5.0-alpha-2 GitHub version: v2.4.25-16-g5127464 and performing initial syncs now. Need to head out of the office for a few hours, will report back when I can

[tdcockers]

OK, a day worth of logs... it came up with the same error (ERROR: Unable to perform a database vacuum: out of memory) three times today. The first two came after 6 consecutive attempts that were throttled, The third one came after 2 consecutive attempts that were throttled. Looks like once all the libraries were synced up, all the issues went away and has been performing without fuss since, with just a handful of file transfers/renames over that time

[abraunegg]

@tdcockers

ERROR: Unable to perform a database vacuum: out of memory

I identified the root cause of that as well - essentially on a failure the exit failure scope was doing its cleanup routine, then the app would exit, which was triggering the exit scope and attempt to do the same DB vacuum on a null object. That should be fixed by the latest version v2.5.0-alpha-2 GitHub version: v2.4.25-18-gea1ca33

[bpozdena]

@abraunegg ,
I also experienced the crashes as others. The latest build v2.4.25-18-gea1ca33 indeed fixed it for me.

[abraunegg]

@tdcockers
As a performance piece - I started an upload of 100K files (mainly small files, total size ~1.7GB) at 7am this morning - 9hrs later (after a few stop and starts due to fixing the things you were highlighting) sitting at approx 71K files uploaded. Normally using the old code this would be a day + overnight excercise.

[tdcockers]

Yes no question performance is better. Were you getting throttled at all during that time?

[abraunegg]

Yes and still am during upload activities:

Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file281.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file280.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file284.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file289.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file290.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file295.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file282.data ... done.
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 61 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 41 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 62 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 44 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 61 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 61 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 50 seconds
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file309.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file305.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file306.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file311.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file310.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file304.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file296.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file302.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file301.data ... done.
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 62 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 60 seconds
Thread sleeping due to 'HTTP request returned status code 429' - The request has been throttled
Sleeping for 61 seconds
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file298.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file299.data ... done.
Uploading new file ./random_100k_files/rrrQBdGAdkTrFuhGRjz1CgWKRvS35LTT/file308.data ... done.

[abraunegg]

@bpozdena , @tdcockers
Any further feedback from the past week ?

[tdcockers]

No issues that I can definitively say are related to 2.5. I've been noting a nearly constant level of disk reads (about 2 MBps) happening on a drive that holds my largest library - looking at the logs it starts a sync, takes about 20mins to sync the changes (none, it's an archive drive so not changed often), then starts again in 5 mins per the instructions in the config file. Might just change the config for that drive to longer intervals.

I wasn't noticing it with 2.4 but I wasn't particularly paying attention either so not sure if this normal behaviour that I'm only seeing because I'm looking for things that might be going wrong.

[abraunegg]

@tdcockers

I've been noting a nearly constant level of disk reads (about 2 MBps) happening on a drive that holds my largest library - looking at the logs it starts a sync, takes about 20mins to sync the changes (none, it's an archive drive so not changed often)

It would be good to get a debug log of this.

In normal circumstances, the client should be always using the /delta call, which only is fetching minimal data as to what has changed online, so if none, then, not much should be send and then processed.

If you are using --cleanup-local-files however, because 'deletes' online are only ever sent once, and can be missed in /delta calls, the client walks the online tree to build a compatible /delta response. It could be this build that is taking 20min to do (which if a large library, would make sense).

The disk reads however is of more interest to me at this point. It could be the integrity check comparing the DB hash and timestamp (the read) with the self developed /delta response to determine what, if anything has changed.

If you are using --cleanup-local-files - can you switch to not using that for a while just to compare the actual experience with the disk reads / 20min checking online changes?

If this is the culprit, then I will have to look at how to potentially handle this without doing a full online walk and build of a /delta compatible response.

[tdcockers]

Hmm. good thought... I had switched to using --cleanup-local-files with the most recent version of 2.4, but I've just checked the config and I did remember (was reminded!) to switch it back off when using --monitor.

Looking now my second biggest library is doing it too, but the gaps between finishing a sync and starting the next round are bigger... makes sense given the smaller library.

Can you please remind me where to find the instructions for generating a debug log?

[abraunegg]

@tdcockers

Can you please remind me where to find the instructions for generating a debug log?

Have a read here: https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support

[abraunegg]

@tdcockers

It might also be worth running a performance profile as well - to understand where the disk reads are coming from

https://github.com/abraunegg/onedrive/wiki/Generate-system-performance-data-for-performance-related-issues

When I was running alpha-2, I have been running this in a CentOS 8 system, with the GUI running, looking at the the performance the OS is giving, whilst also constantly looking at the vSphere disk I/O on the back side of the equation - all whilst using a 100K file/folder set specifically on a documentLibrary (SharePoint). I was doing this to ensure that all the nuances of SharePoint were correctly being looked at and covered off - and I cant recall any constant disk read going on from the graphs.

My test systems are currently turned off (due to external influencing factors) so I cannot at this stage fire them up and get some other testing underway, but once those external factors are resolved, I will do this and post back here - so hopefully there is some sort of correlation that can be seen to at least a pointer to something

[JC-comp]

One performance suggestion is that deltaLink should only be updated when we actually process all changes, as jsonItemsToProcess will actually be lost on the next launch before being processed. Or these raw changes should be stored somehow in somewhere for the next start up if the application exits early. Although I think eventually it will find a way to make up for the missing items in the DB, this would lead to some unnecessary processes. Or probably it will cause some inconsistent issue like if the deletion event is lost?

Although it was mentioned earlier that this was not the intended use case, another thing is that the design of this version will cause a lot of memory usage. The temporary storage of 300k items takes up about 2.5GB of memory, so for me it probably requires at least 10GB of RAM for a startup sync in this version.

[abraunegg]

@JC-comp
Many thanks for your great feedback.

One performance suggestion is that deltaLink should only be updated when we actually process all changes, as jsonItemsToProcess will actually be lost on the next launch before being processed.

Many thanks for calling this out - and yes you are right. The DB should only be updated when the processing of all JSON items to process has been done, so that if there is a failure for whatever reason, the last known deltaLink is then used on next application startup. That is an easy item to fix to add.

Although it was mentioned earlier that this was not the intended use case, another thing is that the design of this version will cause a lot of memory usage. The temporary storage of 300k items takes up about 2.5GB of memory ...

Thankyou for providing this real-world insight. In the v2.4.x the scope of the JSON responses was reduced, so that only the elements that were needed were asked for. In the current v2.5.0-alpha-X code, I have specifically unset that filter so that all JSON elements are being returned - thus this would be inflating the actual memory being used. At some point before getting into a -beta release or -release-candidate release, this was going to be re-looked at to ensure only the actual required items and elements are requested and responded with which then would help with the local memory aspect, but also time to receive object, time to process objects as there is less data being sent|received and processed.

The challenge here is the following:

• OneDrive API JSON events must be processed in the order that they are received
• OneDrive API changeset bundle is hard coded by the API, to only provide ~200 elements (200-205 is what I have seen) at a time

If we process the changeset bundle, when it is received, this leads to longer processing times, and has limitations on how to pipeline|parallel the download or other operations. This is why when re-developing the code, after looking at and testing many different options, quickly validating the JSON (should this be included, excluded), processing root objects first then storing any valid request for further processing worked the best performance wise.

so for me it probably requires at least 10GB of RAM for a startup sync in this version.

Potentially not - even if you are getting a /delta response for your entire 1.7M objects online, whilst you will be receiving many changeset bundles, that initial filtering of inclusion|exclusion is going to discard many objects and these wont be kept in memory - only the objects that are going to be processed will be held in memory until they are processed.

When I was performing a --resync using my test 100K dataset - overall memory usage did not stand out for me as something being aggressively over consumed, as I was also using valgrind to ensure no major memory leak regression was also being introduced. My test systems only have 4GB memory overall (with 4GB swap space configured) - no swapping was being done, but when I have my systems back online I will do some testing and see what drops out of that.

[JC-comp]

This is why when re-developing the code, after looking at and testing many different options, quickly validating the JSON (should this be included, excluded), processing root objects first then storing any valid request for further processing worked the best performance wise.

2023-Oct-13 01:25:38.9152670    Fetching items from the OneDrive API for Drive ID: b!4dsVbk42XUCnmoNmCHuG2jpZi3R68kJOqomyONjIEIJsjA9W_TfzToS5QHo6kjnI
2023-Oct-13 01:30:40.4213182    Processed 41727 OneDrive items
...
2023-Oct-13 03:01:15.3657065    Processed 800099 OneDrive items --> 6.6GB ram
...
2023-Oct-13 04:01:44.7815090    Processed 1352129 OneDrive items --> 11.3GB ram
...

Not sure if the pipeline you mentioned is the future plan or the current solution. I believe in the current version, the whole JSON item will be kept in memory untill all changeset are downloaded, no initial filtering is applied.

[abraunegg]

On this point - you are correct - im missing the initial discard check, that is done when self-generating a /delta response - that needs to be corrected to ensure the process is at least as similar in nature. This will then keep what is in-scope to what should just be processed.

[JC-comp]

As for the filtering, is it possible to check the user defined skip_file & skip_dir first and then check other validations?
Because some failure will be logged and notified for each sync loop, ex. inavlid symlink and invalid names, but only files that I'm not aware of violate these rules should be a problem. If there is no special consideration to use the current designed order, it should be a quick fix.

[JC-comp]

The first 4000 items in early processing stage
2023-Oct-13 09:00:08.9352643    Processing file: xxx
...
2023-Oct-13 09:00:14.8187182    Processing file: xxx
-----------------------------------------------------------
Another 4000 items in later stage
2023-Oct-13 10:48:11.4495490    Processing file: unwanted
...
2023-Oct-13 10:49:09.0402691    Processing file: unwanted

Another performance concern is that the throughput drops off dramatically over time in processJSONItemsInBatch.
I'm not familiar with D, but at first glance, I'm suspecting if the use of the cached skippedItems original target to accelerate the process is actually a linear search, which could be slowing down the overall performance instead?

[JC-comp]

You python comapare is hardly a factual or realistic comparison given all the other items going on with the function that quesies the API

As far as I know, the things happened for this function only includes

• OneDriveApi creation/initialise (X)
• check token expiration (X)
• add auth header (O)
• send request (O)
• parseJSON (O)
• OneDriveApi cleanup (X)

I think it's not too much far away from the python version, as a compiled language, are these overheads reasonable to make it take more than twice as long? On the other hand, if we can somehow reduce these overhead(ex. reuse the api instance), maybe we could save half of the syncing time.

[abraunegg]

I think it's not too much far away from the python version, as a compiled language, are these overheads reasonable to make it take more than twice as long? On the other hand, if we can somehow reduce these overhead(ex. reuse the api instance), maybe we could save half of the syncing time.

As I have indicated before I am not where my systems are at the moment so I cannot give you better answers at the moment.

Now, Python has its own interface (PycURL) to the libcURL Library. In D, there is a very limited interface to access the libcURL library: https://dlang.org/library/std/net/curl.html - so maybe there are performance improvements that could be made there - but thats out of my control.

This potential difference alone could account for performance differences you are seeing. To compare apples with apples you have to compare apples. At the moment you are comparing apples to grapefruit.

On the other hand, if we can somehow reduce these overhead(ex. reuse the api instance), maybe we could save half of the syncing time.

The original code (v2.4.x and below including the 1.x 'skilion' code) used a single API instance. Changing this out to how this alpha code base works, was the only realistic way to correctly support multi-threading.

In this alpha code for each call of 'getDeltaChangesByItemId' a new API instance is created for that invocation. This is to ensure that previous bugs and issues around curl events such as timeout and handling, this was the correct way, for this function to create its own API instance to query the OneDrive API. The performance impact of making a new API instance per call should be highly minuscule and have negligible impact to the query of the API.

You can search all the closed issues|bugs for curl timeout problems: https://github.com/abraunegg/onedrive/issues?q=is%3Aissue+is%3Aclosed

Additionally your Python has no presentation of what handle_error() is doing - so what is happening when there are responses that require request re-submission based on 'retry-after' ... The timing difference could be that the 'getDeltaChangesByItemId' is handling a 429 or some other error leading to the increased time response.

Again - this is alpha code .... I have been trying to obtain feedback (which you are doing so thankyou it is greatly appreciated) - but given some local external influencing factors as previously advised I cannot give you more details on why X or why Y is occurring at this time.

Additionally if you feel you can contribute by improving the performance of the application, patches are always welcome.

[JC-comp]

After a further investigation, since we are not reusing the OneDriveApi now, I believe the overhead comes from acquiring a new accessToken with each call. This also explains why it takes twice as long. Maybe the accessToken and expiration time should be cached somewhere else.

[abraunegg]

After a further investigation, since we are not reusing the OneDriveApi now, I believe the overhead comes from acquiring a new accessToken with each call. This also explains why it takes twice as long. Maybe the accessToken and expiration time should be cached somewhere else.

Your analysis is misguided and incorrect.

What actually occurs is this:

1. At startup, the 'refresh_token' on disk is read into memory and stored as a configuration variable.
2. On each API instance creation, this memory variable is read and used. If the variable entry is empty, the token is again read from disk.
3. If, during the course of using that existing token gets updated, it is first updated to memory, then committed to disk so that, in the event of application failure, the latest token is available.

The only time a new token is acquired is when the initial application authentication is done, which generates the initial 'refresh_token'

If you believe that a memory space access read is accounting for a 50% drop that you are seeing as compared to Python code that on the surface is similar, but is actually vastly different ... again - you are not comparing apples with apples in your analysis.

To be more useful - please start performing performance related data. Please read: https://github.com/abraunegg/onedrive/wiki/Generate-system-performance-data-for-performance-related-issues

In the meantime however it is still a few days before I can make & test changes again - due to external influencing factors, so it may be better to hold of on your analysis or any other ideas that you have until I can make some updates as previously advised.

[JC-comp]

What actually occurs is this: ...

There is no problem with refresh_token, but I think there is a problem with accessToken and accessTokenExpiration stored in OneDriveApi, as they are not stored in memory(ApplicationConfig) or committed to disk(only refresh_token do). Each API call uses accessToken instead of refresh_token, so if we are not resuing OneDriveApi, we lost accessToken for each call and ask for a new one with refresh_token.
If this is the case, it's 100% the cause of the overhead since we actually have two calls in one delta request. So there is nothing to do with the underlying libs/interface/language.

By the way, it's perfectly okay if you don't respond until your system is back and have some tests done, I am in no rush for that.

[JC-comp]

In this release, full scan is deactivated on startup. If this is made by design, perhaps we need something like database consistency check (without touching remote files) before syncing changes from online to local? As the database cannot reflect the true local state anymore.

And when performing a full scan, should we mark all the items in db as out-of-sync first and remove those out-of-sync items at the end? Say a remote item is deleted and the db will not aware of this?

[abraunegg]

it doesn't mean that all cases will not trigger a full scan on startup.

All cases in v2.5.x code, if the deltaLink is valid in the database, by default, a full scan will not be triggered. This is 100% different to v2.4.x. As per #2513 you can explicitly disable the full scan as well.

And the problem of trusting the database and deltalink is, as I stated, it no longer reflects the real local state or the real remote state(deltalink provide changes not state) when the app restart,

No - because if you have a valid deltaLink in the DB and you pass it in, the OneDrive API will reject it, and the code handles this rejection to query without the prior known DB deltaLink.

we could have unexpected data lost because the result of a --resync will be different from an app restart.

Not accurate as, regardless of mode of operation (--sync, --monitor, --download-only, --upload-only, --single-directory) all data is checked in each direction - changes are checked for applicability via date, size, hash - and even then, if the issue is local, the local file in conflict is backed up to a backup file, the online is replaced & can be retrieved from trash.

I think what you need to start doing here is rather than having some theories, please test the code out, test your scenario's and provide detailed logging evidence of what the application is doing vs what your theory is.

So you can understand what has been actually tested Use Case wise, please refer to this list: #2415 (comment)

If we don't mark these out-of-sync items when a full scan is triggered in monitor mode, when items are delete remotely and we are not able to fetch the changeset (because we hit the full scan interval), we will think these items still exist remotely, and leads to an inconsistency.

Not true at all. Because if the item is deleted online, the JSON items show that they are deleted remotely, thus they get deleted locally.

By the way, the design of --local-first mode is really confusing to me, it should be disabled at least if the database is never synced, otherwise it will try to reupload all the files at first. Or better replace with the concept of conflict handling I suggested.

I really think you are starting to over think your use cases here.

Local first means you are trusting your local directory as what must be stored online. Regardless of database state, local first means local first and replicate the local state to online.

[JC-comp]

I will try to set up a testing environment to reproduce the issues and report detailed logging, but for some reason it may take me a few weeks to do that, which is why I can only work on the codebase and come up with some theories.

[JC-comp]

I think I got the point now, the 'source of truth' will only be considered when there is a conflict not for the whole state

Local first means you are trusting your local directory as what must be stored online. Regardless of database state, local first means local first and replicate the local state to online.

Want to make sure if the design of local first is

1. Only care about local items and make sure the local item will exist online and in sync. And don't care about the real consistency between remote drive and local drive.
2. The consistency of the entire disk will be considered. If the file exists on the local machine, it will exist on the remote site. If it does not exist on the local machine, it will not exist on the remote site.

If 1 is the case, then probably it is not the idea of 'source of truth', as it only cares about existing local file, not first. And this will lose the actual synchronization status of the remote and local sides. Remote drive will not looks the same what local drive(source of truth) is when the syn starts.
If 2 is the case, then the app is not working this way, I will report some failed cases._

[abraunegg]

@JC-comp
You also need to be mindful of how OneDrive actually operates:

Use Case:

• File exists locally and online and is in sync (timestamp, hash, size etc)
• I edit the file locally (add new data) and I edit the file online (add totally different data)

When I use --local-first the local file will be used as the 'this is the copy I care about', and be uploaded first. The online version will be retained as a prior version (version history online), and be replaced (latest version online) with that of the file from the local file system.

When not using --local-first the local file should be replaced by the online file, as the prior system state was that both files were in sync.

In the case where --resync is being used (so the entire sync status is not known - thus the sync status of the local file is unknown) the timestamp, hash, size etc are different to that what is online. The local file will then be renamed (a form of local versioning to retain the local file) and the actual local file is then replaced with the online version.

[JC-comp]

Thank you for your follow-up information. Confirm that it was a misunderstanding of the design logic. Now I have a clearer picture of the logic behind the design. Will move on sending PR for confirmed bugs or performance related issue if it can be easily implemented. Some tests to speed up sync times and reduce resource usage of my use case are in progress. Feel free to let me know if this is not a good point to request for performance related modifications, e.g. #2519.

Thank you again for your patience and detailed explanation.

[bpozdena]

I've noticed that the --download-only option does not work in monitor mode. Is this intentional? It used work in v2.4 and older.

 ./onedrive --version                                                                                 
v2.5.0-alpha-2 GitHub version: v2.4.25-15-gc7cc45d

 ./onedrive --monitor --download-only
ERROR: --monitor and --download-only cannot be used together.

[abraunegg]

@bpozdena
That is currently intentional

[abraunegg]

@bpozdena
This restriction has now been removed with v2.5.0-alpha-2 GitHub version: v2.4.25-30-g75c071e

[abraunegg]

@JC-comp
Many thanks for you patience and understanding over the last few days. The fibre has been repaired and I have general access again.

As such, please can you test the following PR version - v2.5.0-alpha-2 GitHub version: v2.4.25-29-g6db484c

Main updates:

• Implement Feature Request: Add support for cron style scheduled full scan #2513 - set 'monitor_fullscan_frequency' to zero to disable any full scan frequency
• Add in the gap of performing initial client side filtering check when performing /delta query as this was missed (due to focus on getting 'azure_ad_endpoint' working)
• Change how /delta query initiates API, reuse for each 'fetch' of next bundle
• Change how 'accessToken' is being compared, as this was triggering a newToken when it was not required
• In non-verbose mode, add some details around how many applicable changes are going to be processed (post fetching /delta) - for example: Processing 145 applicable changes and items received from Microsoft OneDrive ..

Using this updated version, if you are able to gather some performance metrics following https://github.com/abraunegg/onedrive/wiki/Generate-system-performance-data-for-performance-related-issues that would be greatly appreciated.

[JC-comp]

The code changes looks good to me so far. I'm wondering if there is a contribution guide for this project, so that I can correctly send a PR to merge my local changes and get more accurate test results for you.

[abraunegg]

For the time being, please just raise a PR and I will consider or reject - especially given the alpha state of the code.

[JC-comp]

A patch is included in #2515

---

Most of the files are excluded unexpectedly

[DEBUG] This item is not a OneDrive deletion change
[DEBUG] defaultRootId                                        = ..
[DEBUG] 'search id'                                          = ..
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(onedriveJSONItem)                         = false
[DEBUG] onedriveJSONItem['name'].str == 'root'               = false
[DEBUG] itemHasParentReferenceId                             = true
Checkingorigin
[DEBUG] skip_dir path to check (simple):  origin
[DEBUG] Parent details not in database - unable to compute complex path to check
[DEBUG] Performing a simple check only
[DEBUG] skip_dir evaluation for: origin
[DEBUG] No Strict Matching Enforced
[DEBUG] skip_dir exclude result (directory based): false
sync_list item to check: /backup/work/....
[DEBUG] Evaluation against 'sync_list' for this path: /backup/work/....
[DEBUG] [S]exclude           = false
[DEBUG] [S]exludeDirectMatch = false
[DEBUG] [S]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' entry: /backup
[DEBUG] [F]exclude           = false
[DEBUG] [F]exludeDirectMatch = false
[DEBUG] [F]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' final result: EXCLUDED

Please feel free to let me know if more logs are needed

[abraunegg]

Most of the files are excluded unexpectedly

Please check your 'sync_list' configuration. The 'sync_list' file excludes all by *default when you are using it. You must specifically include what you want.

For example, to correctly exclude your 'backup' folder, but sync anything else, your sync list would look like:

# Comments
# Exclude /backup
!/backup/*

# Include everything else
/*

This logic has not changed between v2.4 and v2.5

Please check your 'sync_list' composition.

[abraunegg]

@JC-comp
Further to this, about your 'sync_list' issue - this is the configuration you should be using - for example, if I configure my 'sync_list' as per your configuration:

[DEBUG] This item is not a OneDrive deletion change
[DEBUG] defaultRootId                                        = 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
[DEBUG] 'search id'                                          = 01PTBOWVQQTWRBWB423ZE2TZBR2UH3KDEN
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(onedriveJSONItem)                         = false
[DEBUG] onedriveJSONItem['name'].str == 'root'               = false
[DEBUG] itemHasParentReferenceId                             = true
[DEBUG] skip_file item to check (file name only - parent path not in database): /H2A Research Equivalence v6.docx
[DEBUG] skip_file evaluation for: /H2A Research Equivalence v6.docx
[DEBUG] skip_file evaluation for: /H2A Research Equivalence v6.docx
[DEBUG] Result: false
[DEBUG] sync_list item to check: /Data/4mb_word_doc/H2A Research Equivalence v6.docx
[DEBUG] Evaluation against 'sync_list' for this path: /Data/4mb_word_doc/H2A Research Equivalence v6.docx
[DEBUG] [S]exclude           = false
[DEBUG] [S]exludeDirectMatch = false
[DEBUG] [S]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' entry: /Data
[DEBUG] [F]exclude           = false
[DEBUG] [F]exludeDirectMatch = false
[DEBUG] [F]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' final result: EXCLUDED
Skipping item - excluded by sync_list config: /Data/4mb_word_doc/H2A Research Equivalence v6.docx

I reproduce the problem you are seeing.

However if I configure the 'sync_list' correctly:

# Only include the /Data directory and all sub directories
/Data/*

the application processes this correctly for the same item:

[DEBUG] This item is not a OneDrive deletion change
[DEBUG] defaultRootId                                        = 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
[DEBUG] 'search id'                                          = 01PTBOWVQQTWRBWB423ZE2TZBR2UH3KDEN
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(onedriveJSONItem)                         = false
[DEBUG] onedriveJSONItem['name'].str == 'root'               = false
[DEBUG] itemHasParentReferenceId                             = true
[DEBUG] skip_file item to check (file name only - parent path not in database): /H2A Research Equivalence v6.docx
[DEBUG] skip_file evaluation for: /H2A Research Equivalence v6.docx
[DEBUG] skip_file evaluation for: /H2A Research Equivalence v6.docx
[DEBUG] Result: false
[DEBUG] sync_list item to check: /Data/4mb_word_doc/H2A Research Equivalence v6.docx
[DEBUG] Evaluation against 'sync_list' for this path: /Data/4mb_word_doc/H2A Research Equivalence v6.docx
[DEBUG] [S]exclude           = false
[DEBUG] [S]exludeDirectMatch = false
[DEBUG] [S]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' entry: /Data/*
[DEBUG] Evaluation against 'sync_list' result: wildcard pattern match
[DEBUG] [F]exclude           = false
[DEBUG] [F]exludeDirectMatch = false
[DEBUG] [F]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' final result: included for sync
[DEBUG] Adding this Raw JSON OneDrive Item to jsonItemsToProcess array for further processing

So in your case, your 'sync_list' should be:

# Only include the /backup directory and all sub directories
/backup/*

[ZaxLofful]

Is there an ETA, for this version being feature complete with the stable releases (the listed not working features)?

ETA for when enough testing for this to be live will be?

[abraunegg]

The aim of the 'alpha' series code is to develop the v2.5 code to a point of feature parity, and fix any glaring issues.

It is almost at feature parity (a few small items + one large outstanding) to go, plus more bug scrub and use case testing before tagging as an alpha-3. Post that, I want to start working on fixing the logging output when downloading / uploading data - as this is horribly broken in any operational mode as the threads do not play nicely with console output at present. That is going to take some work.

After than, my intention is to drop a beta for a few months - hopefully have way more people test it / use it / shake it out - I can only test so much.

In terms of an actual release, most likely within the first few months of 2024 - but could be later.

To make this go quicker - more people, more testing given the comprehensive re-write this has been - it needs to be thoroughly tested.

[JC-comp]

@abraunegg
segmentation fault (core dumped) occurs when I send interrupt to the app

1. In monitor mode
2. when fetching items from the OneDrive API
3. Compiled with LDC
4. Ubuntu - 22.04.1
5. Cannot reproduce in v2.4

Steps to reproduce

$ ldc2 --version
LDC - the LLVM D compiler (1.28.0):
  based on DMD v2.098.0 and LLVM 11.1.0
  built with LDC - the LLVM D compiler (1.28.0)
  Default target: x86_64-pc-linux-gnu
  Host CPU: haswell
  http://dlang.org - http://wiki.dlang.org/LDC
$ curl --version
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05

$ ./configure DC=/usr/bin/ldc2
$ make
$ onedrive --version
v2.5.0-alpha-2 GitHub version: v2.4.25-31-g7dfe6b6

$ onedrive  --monitor 
Reading configuration file: xx
Invalid value for key in config file - using default value: monitor_fullscan_frequency
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 86400
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: af2b5e52bea96776 .^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
Segmentation fault (core dumped)

[abraunegg]

My local version does not segfault

Run yours in debug adding -v -v

[abraunegg]

@JC-comp
I cannot reproduce with the PR version of 'alpha-2' - when trying to ^C at any point - there is no unclean exit:

[alex@onedrive-client-dev code-2.5.x]$ ./clone_pr.sh 
Cloning into 'onedrive-pr2505'...
remote: Enumerating objects: 6913, done.
remote: Counting objects: 100% (1959/1959), done.
remote: Compressing objects: 100% (289/289), done.
remote: Total 6913 (delta 1760), reused 1809 (delta 1668), pack-reused 4954
Receiving objects: 100% (6913/6913), 4.63 MiB | 3.42 MiB/s, done.
Resolving deltas: 100% (4892/4892), done.
From https://github.com/abraunegg/onedrive
 * [new ref]         refs/pull/2505/head -> pr2505
Switched to branch 'pr2505'
checking for a BSD-compatible install... /usr/bin/install -c
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for dmd... ldc2
checking version of D compiler... 1.18.0
checking for curl... yes
checking for sqlite... yes
checking for notify... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating contrib/pacman/PKGBUILD
config.status: creating contrib/spec/onedrive.spec
config.status: creating onedrive.1
config.status: creating contrib/systemd/onedrive.service
config.status: creating contrib/systemd/onedrive@.service
rm -f onedrive onedrive.o version
rm -rf autom4te.cache
rm -f config.log config.status
if [ -f .git/HEAD ] ; then \
        git describe --tags > version ; \
else \
        echo v2.5.0-alpha-2 > version ; \
fi
ldc2  -w -g -O -J. -L-lcurl -L-lsqlite3  -L-ldl src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/progress.d src/monitor.d -ofonedrive
v2.5.0-alpha-2 GitHub version: v2.4.25-33-g1eff2d7
[alex@onedrive-client-dev code-2.5.x]$ cd onedrive-pr2505/
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive --version
v2.5.0-alpha-2 GitHub version: v2.4.25-33-g1eff2d7
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive -m 
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data ... 
Scanning the local file system '~/OneDrive' for new data to upload ...
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV .^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive -m 
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data ... 
Scanning the local file system '~/OneDrive' for new data to upload ...
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive -m 
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
^C
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive -m 
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
^C
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive -m 
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data ... 
Scanning the local file system '~/OneDrive' for new data to upload ...
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV .^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
[alex@onedrive-client-dev onedrive-pr2505]$ 
[alex@onedrive-client-dev onedrive-pr2505]$ 
[alex@onedrive-client-dev onedrive-pr2505]$ ./onedrive -m --resync --resync-auth
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Deleting the saved application sync status ...
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ..^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
[alex@onedrive-client-dev onedrive-pr2505]$ 

[JC-comp]

Changed to pr2505, still crashed, and the crash only occurs when it is sending the request

$ onedrive --version
v2.5.0-alpha-2 GitHub version: v2.4.25-33-g1eff2d7
$ onedrive  --monitor -vv
..
[DEBUG] Existing Microsoft OneDrive Access Token Expires: 2023-Oct-18 22:12:10.9052531
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/drives/af2b5e52bea96776/items/AF2B5E52BEA96776!sea8cc6beffdb43d7976fbc7da445c639/delta?token=NDslMjM0OyUyMzE7Mzs1ZTg4M2Q0YS1mYWVmLTQ4YWQtOTA1Ny0yNWJmNTUwNzQ4ZDI7NjM4MzMyNjAzMzM0MDcwMDAwOzIyNTczNTEwNTslMjM7JTIzOyUyMzA7JTIz
^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
[DEBUG] Attempting to perform a database vacuum to merge any temporary data
Segmentation fault (core dumped)

[JC-comp]

Probably could be problems of OneDriveApi curl lib. The affected area is from getDeltaQueryOneDriveApiInstance.initialise(); to getDeltaQueryOneDriveApiInstance.shutdown();

onedrive/src/sync.d

Line 789 in 1eff2d7

getDeltaQueryOneDriveApiInstance.shutdown();

$ onedrive --display-config
Reading configuration file: /home/alan/.config/onedrive/config
Invalid value for key in config file - using default value: monitor_fullscan_frequency
Configuration file successfully loaded
onedrive version                             = v2.5.0-alpha-2 GitHub version: v2.4.25-31-g7dfe6b6
Config path                                  = /home/onedrive/.config/onedrive
Config file found in config path             = true
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 86400
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-alpha-2
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 30
Config option 'data_timeout'                 = 240
Config option 'ip_protocol_version'          = 0

Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false
Selective Business Shared Items configured   = false

Config option 'webhook_enabled'              = false

[abraunegg]

@JC-comp

Ubuntu - 22.04.1

How did you install LDC ?

Ubuntu versions of DMD and LDC cause segfaults .. this is a documented known issue. Please read:

• https://github.com/abraunegg/onedrive/blob/master/docs/ubuntu-package-install.md#1-the-client-may-segfault--core-dump-when-exiting
• https://bugs.launchpad.net/ubuntu/+source/ldc/+bug/1895969

To test this for yourself, uninstall your 'ldc' package, install the package directly from https://dlang.org/download.html and compare.

Or just use a better distribution .. stop using Ubuntu.

[abraunegg]

Probably could be problems of OneDriveApi curl lib. The affected area is from getDeltaQueryOneDriveApiInstance.initialise(); to getDeltaQueryOneDriveApiInstance.shutdown();

However, when I do a ^C in that exact area, when querying the API, getting the data at that same point you are performing a ^C .. there is no segfault:

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Deleting the saved application sync status ...
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ........^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data

[abraunegg]

@JC-comp
Just to 100% close out this 'Segmentation fault (core dumped)' issue you are facing on Ubuntu.

Default Ubuntu 22.04 using 'ldc' Ubuntu Package

alex@ubuntu-22-04:~$ ldc2 --version
LDC - the LLVM D compiler (1.28.0):
  based on DMD v2.098.0 and LLVM 11.1.0
  built with LDC - the LLVM D compiler (1.28.0)
  Default target: x86_64-pc-linux-gnu
  Host CPU: nehalem
  http://dlang.org - http://wiki.dlang.org/LDC

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore
alex@ubuntu-22-04:~$ curl --version
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd
alex@ubuntu-22-04:~$ ./clone_pr.sh 
Cloning into 'onedrive-pr2505'...
remote: Enumerating objects: 6920, done.
remote: Counting objects: 100% (1966/1966), done.
remote: Compressing objects: 100% (296/296), done.
remote: Total 6920 (delta 1765), reused 1808 (delta 1668), pack-reused 4954
Receiving objects: 100% (6920/6920), 4.65 MiB | 4.45 MiB/s, done.
Resolving deltas: 100% (4897/4897), done.
From https://github.com/abraunegg/onedrive
 * [new ref]         refs/pull/2505/head -> pr2505
Switched to branch 'pr2505'
checking for a BSD-compatible install... /usr/bin/install -c
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for dmd... no
checking for ldmd2... ldmd2
checking version of D compiler... 1.28.0
checking for curl... yes
checking for sqlite... yes
checking for notify... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating contrib/pacman/PKGBUILD
config.status: creating contrib/spec/onedrive.spec
config.status: creating onedrive.1
config.status: creating contrib/systemd/onedrive.service
config.status: creating contrib/systemd/onedrive@.service
rm -f onedrive onedrive.o version
rm -rf autom4te.cache
rm -f config.log config.status
if [ -f .git/HEAD ] ; then \
        git describe --tags > version ; \
else \
        echo v2.5.0-alpha-2 > version ; \
fi
ldmd2 -w -g -O -J. -L-lcurl -L-lsqlite3  -L-ldl src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/progress.d src/monitor.d -ofonedrive
v2.5.0-alpha-2 GitHub version: v2.4.25-34-g380e316
alex@ubuntu-22-04:~$ cd onedrive-pr2505/
alex@ubuntu-22-04:~/onedrive-pr2505$ ./onedrive 
Configuring Global Azure AD Endpoints
Authorise this application by visiting:

https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=d50ca740-c83f-4d1b-b616-12c519384f0c&scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access&response_type=code&prompt=login&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient

Enter the response uri from your browser: https://login.microsoftonline.com/common/oauth2/nativeclient?<redacted>
Sync Engine Initialised with new Onedrive API instance

Your command line input is missing either the '--sync' or '--monitor' switches. Please include one (but not both) of these switches in your command line, or refer to 'onedrive --help' for additional guidance.

It is important to note that you must include one of these two arguments in your command line for the application to perform a synchronisation with Microsoft OneDrive

alex@ubuntu-22-04:~/onedrive-pr2505$ ./onedrive -m -v
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alex/.config/onedrive
Application configuration file has been updated, checking if --resync needed
WARNING: no backup config file was found, unable to validate if any changes made
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application version:  v2.5.0-alpha-3 GitHub version: v2.4.25-34-g380e316
Account Type:         documentLibrary
Default Drive ID:     b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV
Default Root ID:      01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 26823301454 KB
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive
OneDrive synchronisation interval (seconds): 300
Maximum allowed open files:                  9223372036854775807
Maximum allowed inotify user watches:        65536
Initialising filesystem inotify monitoring ...
Monitoring directory: .
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV ...^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
Segmentation fault (core dumped)
alex@ubuntu-22-04:~/onedrive-pr2505$ 

Remove Ubuntu 'ldc' package and use 'ldc' from GitHub

• Install LDC 1.28.0 from https://github.com/ldc-developers/ldc/releases/tag/v1.28.0 and extract locally
• Compile using the specified 'ldc2' binary

alex@ubuntu-22-04:~/onedrive-pr2505$ ./configure DC=~/ldc2-1.28.0-linux-x86_64/bin/ldc2; make clean; make;
checking for a BSD-compatible install... /usr/bin/install -c
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for dmd... /home/alex/ldc2-1.28.0-linux-x86_64/bin/ldc2
checking version of D compiler... 1.28.0
checking for curl... yes
checking for sqlite... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating contrib/pacman/PKGBUILD
config.status: creating contrib/spec/onedrive.spec
config.status: creating onedrive.1
config.status: creating contrib/systemd/onedrive.service
config.status: creating contrib/systemd/onedrive@.service
rm -f onedrive onedrive.o version
rm -rf autom4te.cache
rm -f config.log config.status
if [ -f .git/HEAD ] ; then \
        git describe --tags > version ; \
else \
        echo v2.5.0-alpha-2 > version ; \
fi
/home/alex/ldc2-1.28.0-linux-x86_64/bin/ldc2 -w -g -O -J. -L-lcurl -L-lsqlite3  -L-ldl src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/progress.d src/monitor.d -ofonedrive
alex@ubuntu-22-04:~/onedrive-pr2505$ ./onedrive -m -v
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alex/.config/onedrive
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application version:  v2.5.0-alpha-3 GitHub version: v2.4.25-34-g380e316
Account Type:         documentLibrary
Default Drive ID:     b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV
Default Root ID:      01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 26823301454 KB
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive
OneDrive synchronisation interval (seconds): 300
Maximum allowed open files:                  9223372036854775807
Maximum allowed inotify user watches:        65536
Initialising filesystem inotify monitoring ...
Monitoring directory: .
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV .......^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
alex@ubuntu-22-04:~/onedrive-pr2505$ 

Note: There is no segfault when not using the Ubuntu 'ldc' package - this segfault / core creation is the sole problem of Ubuntu provided 'ldc' package.

[JC-comp]

Confirmed that it was a problem with my environment setting, uninstalling the PPA package solved the issue. I did install the compiler from source, but forgot to activate it ... All works as expected now.