Enhancement: progress of large uploads on normal log

[Abasz]

It would be good to be able to follow the upload progress in normal log not just in verbose mode (e.g. logging the percentage of the fragments)

[Abasz]

Hi, since I was testing, I have given some thought to this as well. So the simplest way to do is put a log.log in upload.d in the JSONValue upload() function just below the log.vlog("Uploading fragment: ", offset, "-", offset + fragSize, "/", fileSize); ) call

The following code could do the trick: log.log("Progress: ", double(offset)/fileSize*100,"%");

If you feel you can cut the decimals to one decimal place to have a more elegant output

[abraunegg]

Thanks for the feedback & trying something out on this - will look at this later today

[abraunegg]

So with adding in the data this way it is totally dependant on the file size / fragment size. If still a large file but <10Mb progress stays at 0% then gets done, if 20Mb and so on it gets Progress: 50% then done. I was reading a number of other articles a while back on some other items - will dig this out as I think the right way to do this is elsewhere.

[abraunegg]

The idea I was chasing down was to attach output via the http.onProgress - as this would give (and does) finer grain output as to what is being sent / received & would do so, not just for large uploads but smaller ones as well.

In the end tho this was being too fickle in working right at the moment - so going with your suggestion with a couple of tweaks.

Can you check PR #99 for reference?

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/99/head:pr99
git checkout pr99
make
make install

[Abasz]

You are absolutely correct, this outputs a lot of 0% as the main source of truth here are the fragments and onedirve large file logic. I also mentioned that this is the simplest way just with 1 line of code to add :)

The problem here is that the need for information depends on two things: file size (that is an easy one as we have that info) and the upload speed (this is problematic as it likely changes over time). The reason being is that if you have a 10mb file but your upload speed is only 10kb than you surely want some output of the progress, but if your upload speed is 10mb you would not want upload progress for a file less that a few hundred mb.

[abraunegg]

I think the right way to do this is to utilise http.onProgress to display this information - will work on that aspect & why it's being fickle at the moment.

[Abasz]

Confirmed, log works

[abraunegg]

Can you check the latest pull of PR #99? It should generate a progress bar that provides an ETA & % uploaded for large file uploads

Loading config ...
Using Config Dir: /root/.config/onedrive
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /root/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: <redacted>
Default Root ID: <redacted>
Remaining Free Space: 5250743956
Fetching details for OneDrive Root
OneDrive Root exists in the database
Syncing changes from OneDrive ...
Applying changes of Path ID: <redacted>
Uploading differences of .
Processing root
The directory has not changed
Processing file5.txt
The file last modified time has changed
The file content has changed
Uploading file file5.txt ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
done.
Processing file1.txt
The file last modified time has changed
The file content has changed
Uploading file file1.txt ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
done.
Processing file2.txt
The file last modified time has changed
The file content has changed
Uploading file file2.txt ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
done.
Processing file3.txt
The file last modified time has changed
The file content has changed
Uploading file file3.txt ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
done.
Processing file4.txt
The file last modified time has changed
The file content has changed
Uploading file file4.txt ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
done.
Processing makefiles.sh
The file has not changed
Uploading new items of .
Uploading file ./file6.txt ...
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
 done.
Remaining free space: 5209849492
Uploading file ./file7.txt ...
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| ETA 00:00:00                                                                                    
 done.
Remaining free space: 5167382164
Applying changes of Path ID: <redacted>

When creating test files I am using the following:

dd if=/dev/urandom of=file1.txt count=21 bs=1572864
dd if=/dev/urandom of=file2.txt count=22 bs=1572864
dd if=/dev/urandom of=file3.txt count=23 bs=1572864
dd if=/dev/urandom of=file4.txt count=24 bs=1572864
dd if=/dev/urandom of=file5.txt count=25 bs=1572864
dd if=/dev/urandom of=file6.txt count=26 bs=1572864
dd if=/dev/urandom of=file7.txt count=27 bs=1572864

[Abasz]

This works just great. Nice work!

It was a good idea to remove it from the log and put it to screen only (if one just thinks about it, from a log perspective the progress is not relevant - e.g. when you run it as a daemon, but when you actually have a terminal open to track the progress than you do want some feedback)

one thing, but its more like a nice to have: we upload/download is finished, the ETA that is currently changes to 00:00:00 could show the time required for upload.

[abraunegg]

thanks for the suggestion - agree that the ETA should be changed from ETA 00:00:00 to list how long it actually took.

Will also look at how best to integrate with a download operation - as that will be easier than handling the chunks for upload using http.onProgress

[abraunegg]

@Abasz
Can you check the latest pull of PR #99? You should get something similar to this for uploads now:

In-progress upload:

Uploading  66% |ooooooooooooooooooooooooooo             |   ETA   00:00:03     

Completed upload:

Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:17

[Abasz]

Works as expected, thanks!

[abraunegg]

@Abasz
Can you help validate the latest code from PR #99?

Along with the upload progress bar, there should now be a working download progress bar for large files - where the file threshold is the same for uploads of 4MiB:

Loading config ...
Using Config Dir: /root/.config/onedrive
No config file found, using defaults
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /root/OneDrive
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: <redacted>
Default Root ID: <redacted>
Remaining Free Space: 5344558311
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Syncing changes from OneDrive ...
Applying changes of Path ID: <redacted>
Downloading file large_file1.txt ... 
Downloading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:08                                                                                                         
done.
Downloading file white_paper_c11-494994.pdf ... done.
Uploading differences of .
Processing root
The directory has not changed
Processing large_file1.txt
The file has not changed
Processing white_paper_c11-494994.pdf
The file has not changed
Uploading new items of .
Applying changes of Path ID: <redacted>

[abraunegg]

@Abasz
Any update testing the latest PR #99?

[abraunegg]

Closing due to merge into master of PR #99

[Abasz]

Apologies, I am on holiday with limited access :) I was able to do a simple test and worked nicely. I'll be able to have a more in depth look at this on Sunday the earliest if still needed.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.