Download hangs unexpectedly

[facadev]

Usually after a massive amount of download, the process hangs with no errors prompted, sitting there doing nothing. Is there anyone who's encountered the same problem?

[metahnetoutage]

Can you please elaborate? What was the command that you entered via command line?

It is necessary to be able to reproduce it.
type history and man history so you can check out the command necessary to track what you entered so you can paste it here. I should be able to reproduce the same error with my own personal data.

[facadev]

Can you please elaborate? What was the command that you entered via command line?

It is necessary to be able to reproduce it.
type history and man history so you can check out the command necessary to track what you entered so you can paste it here. I should be able to reproduce the same error with my own personal data.

The command I run is as simple as python3 icloudpd.py ./ --username xxx@icloud.com. I'm not sure whether it can be reproduced easily, as I have hundreds of thousands of items to download and it hangs usually after downloading quite a large amount. I suppose there is something blocked in the download logic?

[metahnetoutage]

Ok, you entered this: icloudpd.py ./ --username xxx@icloud.com I Know where you made the mistake. The first problem with your instruction is that you specify current directory as . Not as ./ unless you say, ./somedirectory You also forgot the flag -d So here is what you should do, Make a directory where you want to save your data. Example, P. Then you would type, icloudpd -d ./P —username xxx@icloud.com You could type: icloudpd.py -d ./P —username xxx@icloud.com Yet I am not sure if having .py or omitting it makes a difference. Then enter the password (if you know, otherwise you first need to reset your password and do the instruction I explained above) When you entered your password correctly you can have a choice if you want the passcode be sent to you via sms(easiest), or via email. You will press a number 0, 1, 2 from the specified options. After this part is done, the download will start and it will take a WHILE. Make sure you set you computer to never turn off, no dim screen, give it 48 hours. Rerun until you have your data. Let me know if you have any questions or run into any issues. I will do my best to help. Dima

…

On Mon, Oct 7, 2019 at 11:50 PM illuminatio ***@***.***> wrote: Can you please elaborate? What was the command that you entered via command line? It is necessary to be able to reproduce it. type history and man history so you can check out the command necessary to track what you entered so you can paste it here. I should be able to reproduce the same error with my own personal data. The command I run is as simple as python3 icloudpd.py ./ --username ***@***.*** I'm not sure whether it can be reproduced easily, as I have hundreds of thousands of items to download and it hangs usually after downloading quite a large amount. I suppose there is something blocked in the download logic? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#145>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALE26ZSYTUD2VQSVTGQA5W3QNQURPANCNFSM4I3OZH5A> .

[greatestrabit]

Usually after a massive amount of download, the process hangs with no errors prompted, sitting there doing nothing. Is there anyone who's encountered the same problem?

I also encountered the same problem
./icloudpd -d ~/icloud/ --username email@email.com --password password

[metahnetoutage]

Well, I do agree that it looks like it’s doing nothing, however that is in the eyes of the impatient(used to be one). Unless an error pops, assume it’s running. If you have a solid connection and 48 hours passed...Huston There May be a problem and we must dig in. Be careful with —password, Why? Because —password can fail you if you have some characters which can be expanded by the command line. Best to Not supply the password for now. Let me know if it make a difference. Dima

…

On Tue, Oct 8, 2019 at 1:50 AM 晓渡 ***@***.***> wrote: Usually after a massive amount of download, the process hangs with no errors prompted, sitting there doing nothing. Is there anyone who's encountered the same problem? I also encountered the same problem ./icloudpd -d ~/icloud/ --username ***@***.*** --password password — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#145>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALE26ZWVH4J6R6RG2ZRAEL3QNRCUHANCNFSM4I3OZH5A> .

[brbeaird]

My download also seems to be hanging.

• When I run icloudpd, I get the "Downloading all 11866 original photos..." message.
• It makes it to around the point where it left off last time saying "XXX already exists."
• It goes on for awhile giving me feedback on which file it's downloading, and the file counter and percentages update in real-time
• After 5 minutes or so, the whole thing freezes. I haven't waited 48 hours, but it's odd to me how it would go from constant feedback to nothing at all.

[metahnetoutage]

Can you please state the shell command that you enter so I can test it using my own cloud thereby reproducing the result. Just saying you also experience is only partially helpful(example, in order for our healthcare to actually improve someone observed the doctors work and said they were the problem for not washing their hands before accepting birth from patients) Same scenario. Please

[brbeaird]

This was the variant I ended up with:

/usr/bin/icloudpd --directory /home/<myuser>/iCloud --cookie-directory '/config' --username <icloudUserName> --password <icloudPassword>--folder-structure={:%Y}'

Over the last couple days, I just kept killing and restarting the process after it stalled out, and eventually it did catch up. It works totally fine now that it's current - just seemed to be an issue when I was starting from a blank slate and needed 10k+ files to download.

[metahnetoutage]

Can you try what I am asking and tell me the results? /usr/bin/icloudpd --directory /home/<myuser>/newDirectory --username <icloudUserName> (If you can get your data without the —cookie Please do so, and if you must have it, include it. Nevertheless do not provide your password using —password flag. Provide the password at the prompt. Let me know the results please. So run either: /usr/bin/icloudpd --directory /home/<myuser>/iCloud --cookie-directory '/config' --username <icloudUserName> Or This: /usr/bin/icloudpd --directory /home/<myuser>/iCloud --username <icloudUserName> Please tell me which command you ran and the results.

…

On Tue, Oct 15, 2019 at 8:30 PM Brian Beaird ***@***.***> wrote: This was the variant I ended up with: /usr/bin/icloudpd --directory /home/<myuser>/iCloud --cookie-directory '/config' --username <icloudUserName> --password <icloudPassword>--folder-structure={:%Y}' Over the last couple days, I just kept killing and restarting the process after it stalled out, and eventually it did catch up. It works totally fine now that it's current - just seemed to be an issue when I was starting from a blank slate and needed 10k+ files to download. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#145>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALE26ZUTLJLJOTM4KWPDLRLQO2DG3ANCNFSM4I3OZH5A> .

[Wayne2Wen]

Usually after a massive amount of download, the process hangs with no errors prompted, sitting there doing nothing. Is there anyone who's encountered the same problem?

I also encountered the same problem.

[dvgeek]

I have the same issue
Python3.8.3 on Windows 10ProN
icloudpd --username xxxxx@gmail.com --directory .\Photos

As long as the number of items are less than a few hundred (less than 500), works okay. But downloading entire library of 19449 items - it downloads 19448 and hangs - left it like that for more than 12 hours but still hung. No response in the DOS window - had to close the window to kill the process.

Also noticed that about 10% of the video files did not fully down load - totally inconsistent - multiple retries after deleting the downloaded smaller file succeeds in downloading the full file - mainly with *.MOV and *.HEVC files. Temporary fix given below for only videos --

1. Download ffmpeg
2. There is a tool called ffprobe.exe that comes along with ffmpeg
3. I noticed this when I was downloading all videos
4. My ffpgeg was in the D:\ directory and my videos were in Z:\ drive in the Videos subdirectory
5. From the Z:\Videos directory issued the following command

for /F %i in ('dir /s /b /A-D') do d:ffprobe -v error -i %i

What this does is for every video file in the Videos directory (any format - and any sub directory) it tests if it's valid. In the list of video file names printed out on the screen my video files with errors were flagged in a different color and the error was printed.

I deleted all the corrupted/incomplete video file and issued the icloudpd command to download the Videos album.

Went through the above process multiple times till I got a clean list of video files without any errors.

[metahnetoutage]

I decided to not work on iPhoto downloading and uploading.

…

On Wed, Jul 1, 2020 at 12:40 PM dvgeek ***@***.***> wrote: I have the same issue Python3.8.3 on Windows 10ProN icloudpd --username ***@***.*** --directory .\Photos As long as the number of items are less than a few hundred (less than 500), works okay. But downloading entire library of 19449 items - it downloads 19448 and hangs - left it like that for more than 12 hours but still hung. No response in the DOS window - had to close the window to kill the process. Also noticed that about 10% of the video files did not fully down load - totally inconsistent - multiple retries after deleting the downloaded smaller file succeeds in downloading the full file - mainly with *.MOV and *.HEVC files. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALE26ZWFIVQBHCNSBNIUWYTRZOGKFANCNFSM4I3OZH5A> .

[gklka]

It also happens with me. I have a massive iPhoto Library, containing 149883 items. The script runs for a few hours then gets slower and slower and eventually stops and never moves forward. Everything seems normal, no error message or anything.

When I restart it, quickly catches up and resumes the download.

My command is also a very basic

icloudpd --directory /mnt/disk6/iCloud \
  --username XXX \
  --password XXX

OS: Ubuntu (latest)

[menkej]

Could you reproduce this behaviour? Does ist also happen with the latest version, and/or using parameter --threads-num 1 ?

[gklka]

Could you reproduce this behaviour? Does ist also happen with the latest version, and/or using parameter --threads-num 1 ?

It's hard to say. It is icloudpd, version 1.6.1, but with only 1 thread running would take 10-20 hours to just do a test with my size of photo library.

[AndreyNikiforov]

Could you reproduce this behaviour? Does ist also happen with the latest version, and/or using parameter --threads-num 1 ?

It's hard to say. It is icloudpd, version 1.6.1, but with only 1 thread running would take 10-20 hours to just do a test with my size of photo library.

@gklka with multiple tests we conducted so far, multithreaded downloading has not been faster than single threaded on fast and slow connections. I encourage you to test speed yourself on a reasonable subset of your files and a number of repetitions. We are planning to add summary #183 , which should help with measurements, but meanwhile you'll have to exclude authentication time yourself.

[gklka]

It seems it works on one thread. :)

[AndreyNikiforov]

Closing as resolved

[gklka]

Hey, why is it resolved, when it does not work, only on one thread?

[menkej]

Hey, why is it resolved, when it does not work, only on one thread?

Good point. Could you reproduce that behaviour with multiple threads? If we have a way to reproduce it, then we should reopen.

[gklka]

Yes, I can. How can I help you to fix this? Provide logs?

[AndreyNikiforov]

Hey, why is it resolved, when it does not work, only on one thread?

@gklka we faced with a number of issues and instabilities specific to multi-threaded downloading lately. Speed tests confirmed that multi-threaded downloads are not faster than single threaded because network connection is a limiting factor. As a result, we changed default behavior to use single-threaded downloads in 1.6.2 and plan on removing multi-threading in next major release. See #180 for more information.

That means that putting an effort into fixing multi-threading downloads makes little sense now, so the issue was closed. Sorry for not providing this context with the closure.

[gklka]

Understood. 👍