Only small number of photos is downloaded from library with 44k photos

[skumka]

Overview

Only 714 photos from 44k is downloaded in very first run of the command

Steps to Reproduce

1. I try to download all photos:
   root@docker:/tmp/temp# ./icloudp --username sxxx@xx.xxx --password :xxx --watch-with-interval 3600 --directory ./data
2. I try to download selected album from the past :
   root@docker:/tmp/temp# ./icloudp --username sxxx@xx.xxx --password :xxx --album "Rok 2003-01 - Bergamo" --directory ./data

I have following results:

1. 714 photos downloaded in case of command for All photos
2. 0 photos downloaded in case command run for selected album

Here is screen from Album option:

root@docker:/tmp/temp# ./icloudp --username sxx@xx.xxx  --password :xxxx --album "Rok 2003-01 - Bergamo" --directory ./data
2024-02-04 17:07:38 DEBUG    Authenticating...
2024-02-04 17:07:42 DEBUG    Looking up all photos and videos from album Rok 2003-01 - Bergamo...
2024-02-04 17:07:43 INFO     Downloading 31 original photos and videos to data ...                                                                                                                            
2024-02-04 17:07:43 INFO     All photos have been downloaded                                                                                                                                                  
root@docker:/tmp/temp#

folder data is empty ?!!

root@docker:/tmp/temp# cd data/
root@docker:/tmp/temp/data# ls
root@docker:/tmp/temp/data# 

It looks that command sees 31 files to be downloaded but downloads 0 since there is sometning which make these photos "downloaded" for the command.

What is could be????

Expected Behavior

44K photos downlaoded in case of command for All photos
31 photos downlaoded in case command run for selected album

Actual Behavior

714 photos downlaoded in case of command for All photos
0 photos downlaoded in case command run for selected album

Context

Very first run of command, nothing was downloaded before , no command run up to now

[AndreyNikiforov]

cmd root@docker:/tmp/temp# ./icloudp --username sxxx@xx.xxx --password :xxx --watch-with-interval 3600 --directory /data is downloading into data folder in the root of your drive, while ls cmd later checks /tmp/temp/data.

[skumka]

Bad copy - it is ./data I removed . by mistake it when anonymizing pass and user
root@docker:/tmp/temp# ./icloudp --username sxxx@xx.xxx --password :xxx --watch-with-interval 3600 --directory ./data
to be completely sure foleder /data does not exist.

[AndreyNikiforov]

Where did you get ./icloudp binary?

Do you mind trying docker run --rm -it icloudpd/icloudpd:1.17.3 icloudpd -u your@email.address --recent 2 --dry-run -d ./ pls? Expecting log with two recent assets picked for downloading (but will not be downloaded as there is --dry-run param). If that cmd succeeds than verified copy of icloudpd ran for your account successfully and you can bisect to the problem.

[skumka]

root@docker:/tmp/temp/data# docker run --rm -it icloudpd/icloudpd:1.17.3 icloudpd -u xxx@xx.xxx --recent 2 --dry-run -d ./
Unable to find image 'icloudpd/icloudpd:1.17.3' locally
1.17.3: Pulling from icloudpd/icloudpd
c926b61bad3b: Pull complete 
f821361375b1: Pull complete 
e619753a56d7: Pull complete 
eb0652d33051: Pull complete 
4f4fb700ef54: Pull complete 
2647709a09f9: Pull complete 
Digest: sha256:c28e9ac5bae49122bd8007be7d89582e793434341c539b7d1244f2e3bcb52898
Status: Downloaded newer image for icloudpd/icloudpd:1.17.3
2024-02-04 18:25:26 DEBUG    Authenticating...
iCloud Password: 
2024-02-04 18:25:46 INFO     Two-step/two-factor authentication is required (2fa)
Please enter two-factor authentication code: 061632
2024-02-04 18:26:05 WARNING  Failed to parse response with JSON mimetype
2024-02-04 18:26:07 INFO     Great, you're all set up. The script can now be run without user interaction until 2SA expires.
You can set up email notifications for when the two-step authentication expires.
(Use --help to view information about SMTP options.)
2024-02-04 18:26:13 DEBUG    Looking up all photos and videos from album All Photos...
2024-02-04 18:26:13 INFO     Downloading 2 original photos and videos to . ...                                                                                                                                
2024-02-04 18:26:16 DEBUG    Downloading 2024/01/30/IMG_7037.JPG...                                                                                                                                           
2024-02-04 18:26:16 DEBUG    [DRY RUN] Would create folder hierarchy 2024/01/30                                                                                                                               
2024-02-04 18:26:17 INFO     [DRY RUN] Would download 2024/01/30/IMG_7037.JPG                                                                                                                                 
2024-02-04 18:26:17 INFO     Downloaded 2024/01/30/IMG_7037.JPG                                                                                                                                               
2024-02-04 18:26:17 DEBUG    Downloading 2024/01/26/IMG_7036.JPG...                                                                                                                                           
2024-02-04 18:26:17 DEBUG    [DRY RUN] Would create folder hierarchy 2024/01/26                                                                                                                               
2024-02-04 18:26:18 INFO     [DRY RUN] Would download 2024/01/26/IMG_7036.JPG                                                                                                                                 
2024-02-04 18:26:18 INFO     Downloaded 2024/01/26/IMG_7036.JPG                                                                                                                                               
2024-02-04 18:26:18 INFO     All photos have been downloaded                                                                                                                                                  
root@docker:/tmp/temp/data# 

[skumka]

If I try to run it for all photos the command will be downloading the first 714 out of 44045 Photos and 935 Videos.

with the statement 'first 714' there is an issue too. Issue is that those 714 are the recents but the recent in the reality - there is number of photos skipped even they are chronologically in between those downloaded.

[AndreyNikiforov]

If I try to run all photos this command will run downloading the first 714 out of 44045 Photos and 935 Videos. Issue is that thos 714 are the recents but not in sequence - there is number of photos skipped even they are chronologically in between those downloaded.

Do you mind trying to download to regular storage, not /tmp, pls? If your /tmp is mounted to ram, then it will have limited size and that affect the number of assets you can download there. The log should have shown that some downloads failed and if it did not, it is a bug in logging.

[skumka]

Well, I can and will do shortly. However I have shawn it does not download any files from those 31 in album "Rok 2003-01 - Bergamo". Literaly zero. It also downloads 714 files for All files option - and downloads it physically on that location, then I do not think you it is a problem with space. But let me run it for you in $home directory.

[skumka]

Here is result for 1000 recent files.

root@docker:/tmp/temp/data# cd
root@docker:~# pwd
/root
root@docker:~# ls
interfaces
root@docker:~# docker run --rm -it icloudpd/icloudpd:1.17.3 icloudpd -u xxx@xx.xxx --recent 1000 --dry-run -d ./
2024-02-04 18:45:24 DEBUG    Authenticating...
iCloud Password: 
2024-02-04 18:45:42 INFO     Two-step/two-factor authentication is required (2fa)
Please enter two-factor authentication code: 567549
2024-02-04 18:45:54 WARNING  Failed to parse response with JSON mimetype
2024-02-04 18:45:57 INFO     Great, you're all set up. The script can now be run without user interaction until 2SA expires.
You can set up email notifications for when the two-step authentication expires.
(Use --help to view information about SMTP options.)
2024-02-04 18:46:02 DEBUG    Looking up all photos and videos from album All Photos...
2024-02-04 18:46:02 INFO     Downloading 1000 original photos and videos to . ...                                                                                        
2024-02-04 18:46:05 DEBUG    Downloading 2024/01/30/IMG_7037.JPG...                                                                                                      
2024-02-04 18:46:05 DEBUG    [DRY RUN] Would create folder hierarchy 2024/01/30                                                                                          
2024-02-04 18:46:06 INFO     [DRY RUN] Would download 2024/01/30/IMG_7037.JPG                                                                                            
2024-02-04 18:46:06 INFO     Downloaded 2024/01/30/IMG_7037.JPG                                                                                                          
. . . .
2024-02-04 18:57:58 DEBUG    Downloading 2019/03/14/IMG_0956.PNG...                                                                                                      
2024-02-04 18:57:58 DEBUG    [DRY RUN] Would create folder hierarchy 2019/03/14                                                                                          
2024-02-04 18:57:58 INFO     [DRY RUN] Would download 2019/03/14/IMG_0956.PNG                                                                                            
2024-02-04 18:57:58 INFO     Downloaded 2019/03/14/IMG_0956.PNG                                                                                                          
2024-02-04 18:57:58 DEBUG    Downloading 2018/04/20/IMG_0010.PNG...                                                                                                      
2024-02-04 18:57:58 DEBUG    [DRY RUN] Would create folder hierarchy 2018/04/20                                                                                          
2024-02-04 18:57:59 INFO     [DRY RUN] Would download 2018/04/20/IMG_0010.PNG                                                                                            
2024-02-04 18:57:59 INFO     Downloaded 2018/04/20/IMG_0010.PNG                                                                                                          
2024-02-04 18:58:01 INFO     All photos have been downloaded                                                                                                             
root@docker:~# 

[skumka]

Atteching screenshots too.
The last one is the summary from Photos to confirm I have 44k files.

[skumka]

It evidently shows that from one moment the tools stops seeing files , however it recognises there are files to download (ie. 'album' case where it sees # of files but download 0 files).

Either there is something either in iCloud Photos library making photos possible to see but invisible to download or download algorythm does not contain some specific cases and pretends files are downloaded.

[skumka]

BTW - it is not only my problem - uff.
There are issues reported by other users too. For example boredazfcuk/docker-icloudpd#446

[AndreyNikiforov]

BTW - it is not only my problem - uff. There are issues reported by other users too. For example boredazfcuk/docker-icloudpd#446

Are you using Shared Libs? Linked issues seems to be related to Shared Libs.

I don't use and I am not familiar with Shared Libs. If you are using Shared Libs, does 714 count somehow related to the volume in one of the Shared Libs?

[skumka]

I have all photos and movies in Primary library.
But I have also Shared Library which is now empty.
In the past I had photos in both, but couple months back I moved all to primary only.
It could be a reason, however there must be something specific since I explicitly tried to download all photos from Primary Library using option --library PrimarySync

I used PrimarySync as it is one of the libraries listed with option --list-libraries

it gives:

2024-02-04 19:46:50 WARNING  Failed to parse response with JSON mimetype
2024-02-04 19:46:52 INFO     Great, you're all set up. The script can now be run without user interaction until 2SA expires.
You can set up email notifications for when the two-step authentication expires.
(Use --help to view information about SMTP options.)
PrimarySync
SharedSync-6FBDC24B-4BA8-4F73-9A10-92339C2226B2
root@docker:~# 

[kmplngj]

I have the same problem. 25660 items in iCloud library but icloudpb only tries to download 1266.

I use the docker image (version 1.18.0) and I have a Shared iCloud library.