improve test coverage

[AndreyNikiforov]

No description provided.

[coveralls]

Pull Request Test Coverage Report for Build 170

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.6%) to 100.0%

---

Totals
Change from base Build 163:	0.6%
Covered Lines:	471
Relevant Lines:	471

---

💛 - Coveralls

[AndreyNikiforov]

Travis says that icloudpd/base.py 251 3 99% 281-284 missing coverage. Locally (in docker) coverage is 100%. Checking...

[menkej]

Travis says that icloudpd/base.py 251 3 99% 281-284 missing coverage. Locally (in docker) coverage is 100%. Checking...

This is my last change for the throttling, it seems... How could we force this exception in a test case? Just ignoring these lines (pragma: no cover) is not really a good solution...

[AndreyNikiforov]

Travis says that icloudpd/base.py 251 3 99% 281-284 missing coverage. Locally (in docker) coverage is 100%. Checking...

This is my last change for the throttling, it seems... How could we force this exception in a test case? Just ignoring these lines (pragma: no cover) is not really a good solution...

I added a test and now coverage is 100% in Travis.

[menkej]

Excellent work @AndreyNikiforov ! Makes sense to go for single threading here, even though I'm still a little unsure about the multithreaded downloads. Multithreading can be a beast and I remember we had issues with it last year. I did not find the calm yet to dive into the current implementation...

[AndreyNikiforov]

@menkej I am skeptical about multithreaded downloading from a single source with a goal to improve speed because 1) network speed will be a limiting factor if no throttling at the source 2) if throttled at the source, it may be too naïve to expect abuse prevention to be implemented at connection level only. I would be curious to see some test results (single threaded vs multi threaded downloading)

[menkej]

@menkej I am skeptical about multithreaded downloading from a single source [...] I would be curious to see some test results (single threaded vs multi threaded downloading)

I run some test downloads. I am on a 100 MBit connection. No clear lab conditions - Netflix and Skype session running. Downloading 1000 recent files.

default setting (multithreaded)
87,59s user 30,36s system 11% cpu 16:33,71 total
77,79s user 27,50s system 12% cpu 13:58,24 total
77,67s user 27,30s system 13% cpu 12:34,12 total

--threads-num 1
79,00s user 27,64s system 14% cpu 12:34,61 total
79,17s user 27,51s system 11% cpu 15:04,41 total
78,30s user 27,80s system 14% cpu 12:36,18 total

Just three runs. Just 1000 pics. Seeing not much of a difference really.

Multithreading might make sense for parallel image/video conversions and stuff. For download... not so sure...

Maybe it makes a difference on a clear line with a big pipe...

[AndreyNikiforov]

@menkej I am skeptical about multithreaded downloading from a single source [...] I would be curious to see some test results (single threaded vs multi threaded downloading)

I run some test downloads. I am on a 100 MBit connection. No clear lab conditions - Netflix and Skype session running. Downloading 1000 recent files.

default setting (multithreaded)
87,59s user 30,36s system 11% cpu 16:33,71 total
77,79s user 27,50s system 12% cpu 13:58,24 total
77,67s user 27,30s system 13% cpu 12:34,12 total

--threads-num 1
79,00s user 27,64s system 14% cpu 12:34,61 total
79,17s user 27,51s system 11% cpu 15:04,41 total
78,30s user 27,80s system 14% cpu 12:36,18 total

Just three runs. Just 1000 pics. Seeing not much of a difference really.

Multithreading might make sense for parallel image/video conversions and stuff. For download... not so sure...

Maybe it makes a difference on a clear line with a big pipe...

Thanks for running tests, @menkej ! We can look at deprecating multithreading support eventually to reduce complexity

[boredazfcuk]

I ran some tests on this tonight. My home internet connection is only 36Mbps.

Here's two log files generated:

Multi-threaded - https://pastebin.com/6RKYfgsy
Single-threaded - https://pastebin.com/7zhgtjc7

Results:

threads   start      finish      total
1         22:55:00   23:16:32    21m 32s
default   22:35:00   22:50:42    15m 42s

To sum up, I see a significant performance benefit to multi-threaded downloads. However, multi-threaded results in occasional errors e.g.:

ERROR    IOError while writing file to /home/user/iCloud/2020/camphoto_1804928587.jpg! You might have run out of disk space, or the file might be too large for your OS. Skipping this file...

[AndreyNikiforov]

@boredazfcuk thanks for measuring time! do you mind repeat same test multiple times to eliminate of any possible network speed fluctuations etc?

I think it would be helpful to add a summary at the end of the the icloudpd run with total bytes downloaded, overall time, avg speed, and may be something else. Will be valuable beyond just perf tests.

[boredazfcuk]

Sorry for not getting back sooner guys... I was busy last night and most of tonight too.

I've run the tests again, but I've only had time to do one of each. The Mrs had a late night and she kills my connection with Netflix until she falls asleep lol

I should have more time tomorrow to repeat again, hopefully two or three times.

The results were pretty much the same as yesterday, multi-threaded was significantly faster, with single threaded taking nearly twice as long:

Threads     Start       Finish      Total
Default     23:19:59    23:33:40    13m 41s
1           00:09:59    00:34:51    24m 52s

Log files:
Multi-thread - https://pastebin.com/7Dku3gUg
Single-thread - https://pastebin.com/Ya7Wi4Cs

[boredazfcuk]

I managed to do some more testing overnight, as I put a speed test routine into my container and kicked off 8 runs (4-multi and 4-single).

I am no longer seeing the large delays on single threaded downloads. Here are the details and times calculated:

Multi:
https://pastebin.com/94E77DJx - Start: 00:25:32 End: 00:37:42 Total: 0:12:10
https://pastebin.com/PzELtKis - Start: 23:08:16 End: 23:20:09 Total: 0:11:53
https://pastebin.com/7SU4K52e - Start: 23:33:16 End: 23:45:36 Total: 0:12:20
https://pastebin.com/JER2eKJS - Start: 23:59:21 End: 00:12:33 Total: 0:13:12

Single
https://pastebin.com/EzXds2Vz - Start: 23:45:37 End: 23:59:20 Total: 0:13:43
https://pastebin.com/XNVpPd8q - Start: 00:12:34 End: 00:25:31 Total: 0:12:57
https://pastebin.com/DT2bKR6T - Start: 23:20:10 End: 23:33:15 Total: 0:13:05
https://pastebin.com/5TQRTLkw - Start: 00:37:43 End: 00:49:22 Total: 0:11:39

There seems to be a bug in my container that calculates the total time taken incorrectly, which is why they all show as taking 20:32 in the log files. When calculating manually, the times shown above are accurate.

I'm not sure why this is different to before. As I could reliably reproduce it before under the same network conditions. These times have been taken using the latest version which defaults to single threaded.

[AndreyNikiforov]

@boredazfcuk thanks a lot for the test! Not we can proceed with #180 with confidence.

[boredazfcuk]

@boredazfcuk thanks a lot for the test! Not we can proceed with #180 with confidence.

Time calculation bug should have been fixed and I've pushed latest version to DockerHub. People should be able to replicate my tests using this script I created:

#!/bin/bash
apple_id="notmy@appleid.com"
apple_password="notmypasswordeither"
mkdir --parents ~/single ~/multi

CreateCookies(){
   docker run -it --rm \
      --name iCloudPD_Test_single \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/single:/config \
      boredazfcuk/icloudpd
   docker run -it --rm \
      --name iCloudPD_Test_multi \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env multi_thread=Enabled \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/multi:/config \
      boredazfcuk/icloudpd
}

RunSingleThread(){
   docker run --rm \
      --name iCloudPD_Test_single \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/single:/config \
      boredazfcuk/icloudpd
}

RunMultiThread(){
   docker run --rm \
      --name iCloudPD_Test_multi \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env multi_thread=Enabled \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/multi:/config \
      boredazfcuk/icloudpd
}

CreateCookies
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"