OneDrive Client Version v2.5.0-alpha-4 is now available for testing #2568

abraunegg · 2023-12-22T20:58:11Z

abraunegg
Dec 22, 2023
Maintainer

As part of the ongoing development of v2.5.0, client version v2.5.0-alpha-4 is now available for testing.

Changelog from alpha-3:

Implement buffered logging so that all logging from all upload & download activities are handled correctly
Documentation updates

Whats still missing?

Logging output may still broken in some areas due to buffered logging
Documentation updates need to be finished

You can use your normal account if you wish (this is your choice), however this is 100% alpha quality software at the moment. If you experience data loss or something else with your real data - whilst not good, please take every effort to not use this code with your real data.

There will be functionality differences, there will be bugs or things that are not working right.

My ask is that you create a test report - what works, what does not work, what could be potentially changed | improved - basically I am soliciting some feedback on where this total re-write is heading as it matures to a v2.5.0 release.

How can you get v2.5.0-alpha-4 ?

You need to build this client from source. Follow the high level directions here to get the development requirements installed first: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#building-from-source---high-level-requirements

Next clone and checkout the PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2563/head:pr2563
git checkout pr2563
./configure; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

When running the PR, your version should be as per below or greater:

v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1

The important point is v2.5.0-alpha-4. The v2.4.25-XX-abcdefg will change as new commits are put into the PR branch, whilst things are added/fixed/updated.

What is known to be not functioning right at this stage:

Existing GitHub master documentation does not match what v2.5.0 does or supports at present. Please refer to the updated docs here: https://github.com/abraunegg/onedrive/tree/onedrive-v2.5.0-alpha-4/docs

I have no test files or test data - what can I use?

In developing this application I use a script as per below to generate random data:

#!/bin/bash
FILECOUNT=0
DIRCOUNT=0
DESTDIR="/my/path/to/my/sync_dir/folder/OneDrive/random_100_files"
FILES=10
FOLDERS=10
BLOCKSIZE=4

mkdir -p $DESTDIR

while [ $DIRCOUNT -lt $FOLDERS ] ; do
        DIRNAME=`cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1`
        DIRPATH=$DESTDIR/$DIRNAME
        mkdir -p $DIRPATH
        while [ $FILECOUNT -lt $FILES ] ; do
                FILESIZE=$RANDOM
                FILESIZE=$(($FILESIZE+1024))
                FILENAME=$DIRPATH/file${FILECOUNT}.data
                echo "Creating $FILENAME"
                dd if=/dev/urandom of=${FILENAME} bs=$BLOCKSIZE count=$FILESIZE > /dev/null 2>&1
                ((FILECOUNT++))
        done
        FILECOUNT=0
        ((DIRCOUNT++))
done

Update DESTDIR with the right directory and adjust FILES and FOLDERS as desired. The above settings will create 10 folders, with each folder having 10 files. with each file size between 6KB and 125KB.

To increase (or decrease) the random file size, change BLOCKSIZE, generally use ^2 values: 1,2,4,8,16,32,64,128,256 and so on.

At 128 you will get files from 100KB -> ~3.8MB, at 256 most files will be between 500KB --> 7.5MB

bpozdena · 2023-12-22T22:25:27Z

bpozdena
Dec 22, 2023

Great job as always, @abraunegg !

There are some cosmetic issues during file upload. This just make stdout parsing a little harder :-).

1)The initial ETA shows -596523:-14:-8 when upload progress is at 0%. Expectation: --:--:--
2)Percentage is shown with two decimal points 80.00%. Expectation: 80%
3)There is a missing whitespace between the dots and "100%". Expectation: ..... 100%

Uploading: 50MBa.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBa.zip ....................................................... 20.00% |  ETA   00:00:16
[...]
Uploading: 50MBa.zip ....................................................... 80.00% |  ETA   00:00:03
Uploading: 50MBa.zip .......................................................100%    | DONE in 00:00:16

For comparison, the download details already look great:

Downloading: 50MBa.zip ...................................................... 0%    |  ETA   --:--:--
Downloading: 50MBa.zip ...................................................... 5%    |  ETA   00:00:19
[...]
Downloading: 50MBa.zip ...................................................... 95%   |  ETA   00:00:00
Downloading: 50MBa.zip ...................................................... 100%  | DONE in 00:00:03

2 replies

abraunegg Dec 22, 2023
Maintainer Author

The initial ETA shows -596523:-14:-8 when upload progress is at 0%. Expectation: --:--:--
here is a missing whitespace between the dots and "100%". Expectation: ..... 100%

Can you make sure you are using: v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1 or greater ?

bpozdena Dec 23, 2023

I was using Application Version: v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1 as per instructions. Sorry I did not mention it.

bpozdena · 2023-12-22T22:33:22Z

bpozdena
Dec 22, 2023

Multi-threading does not seem to work with file uploads. When I drop 3 larger files into a sync_dir while onedrive is running in monitor mode, I can see they are being uploaded one at a time.

Example:

Uploading: 50MBa.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBa.zip ....................................................... 20.00% |  ETA   00:00:16
Uploading: 50MBa.zip ....................................................... 40.00% |  ETA   00:00:09
Uploading: 50MBa.zip ....................................................... 60.00% |  ETA   00:00:06
Uploading: 50MBa.zip ....................................................... 80.00% |  ETA   00:00:03
Uploading: 50MBa.zip .......................................................100%    | DONE in 00:00:16
Uploading new file ./50MBa.zip ... done.
Uploading: 50MBb.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBb.zip ....................................................... 20.00% |  ETA   00:00:52
Uploading: 50MBb.zip ....................................................... 40.00% |  ETA   00:00:24
Uploading: 50MBb.zip ....................................................... 60.00% |  ETA   00:00:12
Uploading: 50MBb.zip ....................................................... 80.00% |  ETA   00:00:05
Uploading: 50MBb.zip .......................................................100%    | DONE in 00:00:26
Uploading new file ./50MBb.zip ... done.
Uploading: 50MBc.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBc.zip ....................................................... 20.00% |  ETA   00:00:12
Uploading: 50MBc.zip ....................................................... 40.00% |  ETA   00:00:09
Uploading: 50MBc.zip ....................................................... 60.00% |  ETA   00:00:06
Uploading: 50MBc.zip ....................................................... 80.00% |  ETA   00:00:03
Uploading: 50MBc.zip .......................................................100%    | DONE in 00:00:15
Uploading new file ./50MBc.zip ... done.

EDIT:
Multi-threaded upload does work, but only after I manually restart onedrive monitoring client.

Example after stopping and starting the client in monitoring mode:

Scanning the local file system '~/OneDrive_boris@personal.com2' for new data to upload ...
Uploading: 50MBc.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBa.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBb.zip ....................................................... 00.00% |  ETA   -596523:-14:-8
Uploading: 50MBc.zip ....................................................... 20.00% |  ETA   00:00:28
Uploading: 50MBb.zip ....................................................... 20.00% |  ETA   00:00:40
Uploading: 50MBa.zip ....................................................... 20.00% |  ETA   00:00:40

2 replies

abraunegg Dec 22, 2023
Maintainer Author

@bpozdena

Multi-threaded upload does work, but only after I manually restart onedrive monitoring client.

How are you adding the files in for upload in the initial instance? Are you using --monitor for this initially?

bpozdena Dec 23, 2023

Yes, I'm using only the --monitor mode.

Steps to reproduce:

Start onedrive --monitor and keep it running
Paste multiple large files into sync_dir

bpozdena · 2023-12-22T22:38:44Z

bpozdena
Dec 22, 2023

Question: Is there a configuration option to influence how many threads should be used for downloads/uploads?

3 replies

abraunegg Dec 22, 2023
Maintainer Author

Question: Is there a configuration option to influence how many threads should be used for downloads/uploads?

No: hard locked for optimum at 16 threads. Lots of testing went into that. Down the track, this might be exposed as a configurable item, but for now - no.

bpozdena Dec 23, 2023

OK, great to know. So, when I configure rate_limit to 10Mb/s , onedrive client can theoretically use up to 160Mb/s ?

abraunegg Dec 23, 2023
Maintainer Author

Potentially yes

bpozdena · 2023-12-22T23:33:23Z

bpozdena
Dec 22, 2023

I'm not sure if resumption of partial downloads works as expected. When I stop monitoring mode during large file download, the partial file is not completed after I start the monitoring mode again.

I have to wait until monitor_fullscan_frequency is reached, at which point, the file is completely re-downloaded.

./onedrive --version                                                                                                                                          
v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1

./onedrive --confdir='/home/bob/.config/onedrive/accounts/boris@personal.com' --monitor                                                                    
Reading configuration file: /home/bob/.config/onedrive/accounts/boris@personal.com/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: dc6e304xxxxxxx ..
Processing 1 applicable changes and items received from Microsoft OneDrive .
Downloading: 50MBa.zip ...................................................... 0%    |  ETA   --:--:--
Downloading: 50MBa.zip ...................................................... 5%    |  ETA   00:00:38
Downloading: 50MBa.zip ...................................................... 10%   |  ETA   00:00:18
Downloading: 50MBa.zip ...................................................... 15%   |  ETA   00:00:00
Downloading: 50MBa.zip ...................................................... 20%   |  ETA   00:00:00
^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data
 
./onedrive --confdir='/home/bob/.config/onedrive/accounts/boris@personal.com' --monitor                                                                 
Reading configuration file: /home/bob/.config/onedrive/accounts/boris@personal.com/config
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
OneDrive synchronisation interval (seconds): 300
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: dc6e304xxxxxxx ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data ... 
Scanning the local file system '~/OneDrive_boris@personal.com2' for new data to upload ...
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: dc6e304ae2d1d3ad ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete

2 replies

abraunegg Dec 22, 2023
Maintainer Author

I'm not sure if resumption of partial downloads works as expected. When I stop monitoring mode during large file download, the partial file is not completed after I start the monitoring mode again.

Resumption of downloads should be working .. however the logging output you are using is not --verbose .. increase your logging output ... this is probably one of the logging output tweaks that need fine tuning.

bpozdena Dec 23, 2023

Unfortunately, it does not work. The .partial file just sits there until 'monitor_fullscan_frequency' is reached. When that happens the '.partial' file is just deleted and the entire file is downloaded again from 0%.

./onedrive --confdir='/home/bob/.config/onedrive/accounts/boris@personal.com' --monitor -v --monitor-interval 15 --monitor-fullscan-frequency 1   
Reading configuration file: /home/bob/.config/onedrive/accounts/boris@personal.com/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/bob/.config/onedrive/accounts/boris@personal.com
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1
Account Type:         personal
Default Drive ID:     dc6e304ae2d1d3ad
Default Root ID:      DC6E304AE2D1D3AD!101
Remaining Free Space: 4.95 GB (5316280320 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/bob/OneDrive_boris@personal.com2
OneDrive synchronisation interval (seconds): 15
Maximum allowed open files:                  9223372036854775807
Maximum allowed inotify user watches:        524288
Initialising filesystem inotify monitoring ...
Monitoring directory: .
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching /delta response from the OneDrive API for Drive ID: dc6e304ae2d1d3ad
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3
Processing API Response Bundle: 2 - Quantity of 'changes|items' in this bundle to process: 1
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive 
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Number of items to download from OneDrive: 1
Downloading: 100MBa.zip ..................................................... 0%    |  ETA   --:--:--
Downloading: 100MBa.zip ..................................................... 5%    |  ETA   00:00:19
Downloading: 100MBa.zip ..................................................... 10%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 15%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 20%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 25%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 30%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 35%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 40%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 45%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 50%   |  ETA   00:00:00
^CDownloading: 100MBa.zip ..................................................... 55%   |  ETA   00:00:00
Got termination signal, performing clean up
Shutting down DB connection and merging temporary data


 ls -la /home/bob/OneDrive_boris@personal.com2                                                                                                    
total 59584
drwx------  2 bob bob     4096 23. pro 01.08 .
drwx------ 23 bob bob     4096 23. pro 01.09 ..
-rw-r--r--  1 bob bob 61001728 23. pro 01.08 100MBa.zip.partial


./onedrive --confdir='/home/bob/.config/onedrive/accounts/boris@personal.com' --monitor -v --monitor-interval 15 --monitor-fullscan-frequency 1                
Reading configuration file: /home/bob/.config/onedrive/accounts/boris@personal.com/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/bob/.config/onedrive/accounts/boris@personal.com
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1
Account Type:         personal
Default Drive ID:     dc6e304ae2d1d3ad
Default Root ID:      DC6E304AE2D1D3AD!101
Remaining Free Space: 4.95 GB (5316280320 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/bob/OneDrive_boris@personal.com2
OneDrive synchronisation interval (seconds): 15
Maximum allowed open files:                  9223372036854775807
Maximum allowed inotify user watches:        524288
Initialising filesystem inotify monitoring ...
Monitoring directory: .
Performing initial syncronisation to ensure consistent local state ...
Starting a sync with Microsoft OneDrive
Fetching /delta response from the OneDrive API for Drive ID: dc6e304ae2d1d3ad
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data ... 
Processing DB entries for this Drive ID: dc6e304ae2d1d3ad
Processing ~/OneDrive_boris@personal.com2
The directory has not changed
Scanning the local file system '~/OneDrive_boris@personal.com2' for new data to upload ...
Skipping item - excluded by skip_file config: ./100MBa.zip.partial
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: dc6e304ae2d1d3ad
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
Starting a sync with Microsoft OneDrive
Performing a full scan of online data to ensure consistent local state
Fetching /delta response from the OneDrive API for Drive ID: dc6e304ae2d1d3ad
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3
Processing API Response Bundle: 2 - Quantity of 'changes|items' in this bundle to process: 1
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive 
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Number of items to download from OneDrive: 1
Downloading: 100MBa.zip ..................................................... 0%    |  ETA   --:--:--
Downloading: 100MBa.zip ..................................................... 5%    |  ETA   00:00:19
Downloading: 100MBa.zip ..................................................... 10%   |  ETA   00:00:00
[...]
Downloading: 100MBa.zip ..................................................... 85%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 90%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 95%   |  ETA   00:00:00
Downloading: 100MBa.zip ..................................................... 100%  | DONE in 00:00:04
Downloading file ./100MBa.zip ... done
Performing a database consistency and integrity check on locally stored data ... 
Processing DB entries for this Drive ID: dc6e304ae2d1d3ad
Processing ~/OneDrive_boris@personal.com2
The directory has not changed
Processing 100MBa.zip
The file has not changed
[...]

abraunegg · 2023-12-23T08:29:56Z

abraunegg
Dec 23, 2023
Maintainer Author

@bpozdena
Please add any other issues / feedback here: #2570

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OneDrive Client Version v2.5.0-alpha-4 is now available for testing #2568

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

OneDrive Client Version v2.5.0-alpha-4 is now available for testing #2568

abraunegg Dec 22, 2023 Maintainer

How can you get v2.5.0-alpha-4 ?

I have no test files or test data - what can I use?

Replies: 5 comments · 9 replies

abraunegg Dec 22, 2023 Maintainer Author

abraunegg Dec 22, 2023 Maintainer Author

abraunegg Dec 22, 2023 Maintainer Author

abraunegg Dec 23, 2023 Maintainer Author

abraunegg Dec 22, 2023 Maintainer Author

abraunegg Dec 23, 2023 Maintainer Author

abraunegg
Dec 22, 2023
Maintainer

Replies: 5 comments 9 replies

abraunegg Dec 22, 2023
Maintainer Author

abraunegg Dec 22, 2023
Maintainer Author

abraunegg Dec 22, 2023
Maintainer Author

abraunegg Dec 23, 2023
Maintainer Author

abraunegg Dec 22, 2023
Maintainer Author

abraunegg
Dec 23, 2023
Maintainer Author