pyicloud_ipd.exceptions.PyiCloudAPIResponseError: private db access disabled for this account.

[biGdada]

hi.
i'm getting alot of those lately:
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: private db access disabled for this account. Please wait a few minutes then try again. The remote servers might be trying to throttle requests. (ACCESS_DENIED)
with my icloud account. my sons account never gets this error.

i suppose thats because my photo library is much bigger. what is the actual limit of requests? is it per hour? is there any info of how this throttling mechanism works?

[boredazfcuk]

My photostream has around 600 photos in it. I’ll hit this error if I try syncing too many times in a short space of time... I’d guess at about 6 syncs in an hour.

It took days to clear the first time I ran into it. I had sync scheduled every hour an hit the error a lot of times before I noticed it. I think that once you hit the error, every failed attempt adds to the lockout time.

I switched my container off for two days, set it to sync every 6hrs and it’s gone away after that.

[biGdada]

yea, i figured out that the lockout time is accumulating. i just wanted to know is there a way to calculate/check the lockout time and maybe modify the script accordingly. for example, before running it - check wherever the account is locked out and, ideally, for how long....
thanks for a fast response 👍

[menkej]

Maybe you're just running the script too often, @biGdada ? I've been running it on a daily cron for some time and did not run into this recently. How often are you running it? Are you still receiving these messages?

[metahnetoutage]

If someone is running a script too often, and an error occurs then it’s honestly not good enough coding! You guys got it! Check what cause the error and instead throw a good exception such as, no pictures and exit cleanly. Is that a possibility?

…

On Thu, Oct 1, 2020 at 4:41 AM menkej ***@***.***> wrote: Maybe you're just running the script too often, @biGdada <https://github.com/biGdada> ? I've been running it on a daily cron for some time and did not run into this recently. How often are you running it? Are you still receiving these messages? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#163 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALE26ZSKPHPHZOOAKF4UXULSIRTGHANCNFSM4QBZIJCQ> .

[menkej]

Hi! I have not seen this error myself and I'm just trying to get a handle on this. So please let's move this into a constructive way of communication, so that we can find a solution for this together, ok?

Can you provide more of the output you're getting, so it might be possible to find where this message occurs?

[boredazfcuk]

Just set your cron to run every ten minutes and you’ll soon run into it. I’m fairly certain this is due to Apple rate limiting.

[metahnetoutage]

I hope my comment was not being none constructive. That is why I said that the coding is not that good and you guys can nail it! Once you know the issue maybe limit number of runs.

[boredazfcuk]

Somebody running the script too often is a user problem, not a coding problem.

[menkej]

I was able to reproduce it now and catch the exception. I'll think about a "proper user message" and then check it in.

I re-ran a download 6/7 times right after another and came into this situation. Apple is obviously having some limits implemented here, which makes perfect sense for my. After just a couple of minutes I was able to run the download again for some retries. I guess this is quite a rare situation where "a normal user" would run this script in such short intervals anyways...

[biGdada]

@menkej i think you are missing a point here. the throttling depends,among other things, on the amount of photos in your camera roll. if your camera roll has about 5000 pics, you will run into throttling if you run the script once an hour. i was merely asking if anyone knows how this mechanism works in order to calculate the safe frequency of running this script. i dont think a fix is needed.

[metahnetoutage]

Well calculation is the fix so you run a command and it just works! Am I missing something?

[menkej]

So we have two different issues here.

1. The script is started too often in a short period of time hand closes with exception from the base-lib before downloading anything.
2. You are downloading many photos in one call and run into throttling and receive this exception as well.

For the first issue I have a patch witch I'll put in a pull request later, so that the program exits clearly and informs the user not to run it in short intervals.

For the second issue, I guess from how I read the code that we already have some waiting/retry in the code but maybe the exception should not come out. But I could not reproduce it up till now, as I'm running on a very slow internet connection here (I'm currently in a holiday house). I downloaded some 3k pictures right now without errors. If you could provide the full output of the error you're getting I might be able to figure out where to look. Which command line parameters are you using?

[menkej]

HI @biGdada,
I'm still not able to reproduce the behaviour when downloading many files (6k). Are you able to reproduce the error message during downloads? If you can, could you try my fix (see the released pull request). Does it solve the issue for you?

[biGdada]

@menkej i couldn't reproduce it as well, the script has been running every hour for about 2 days

[menkej]

i couldn't reproduce it as well, the script has been running every hour for about 2 days

Great, so this should be good now. Thanks for the feedback!

[biscuitehh]

Hey there - I just tried out this yesterday and I've been receiving the dreaded "private db access disabled for this account" error. I've also tried waiting a few hours between attempts and still receive the same error. I have around 24,000 - 25,000 photos and have been trying to use this command:
./icloudpd.py --username <redacted> -p hunter12 --directory ./data --cookie-directory ./cookies --size original --auto-delete --recent 500 --list-albums

[menkej]

@biscuitehh you do --list-albums, meaning no download at all... Did you try without --list-albums?

[biscuitehh]

@menkej yup - I've tried it without --list-albums, --auto-delete, --recent x, and inside the Docker container. No luck no matter what I try/how long I space my requests apart

[biscuitehh]

@menkej just a heads up - it looks like the latest commit that set the default thread count to one fixed my issue. I'll let ya'll know if comes up again :)

[boredazfcuk]

I just hit this error again last night... I wasn't doing anything all night so this shouldn't be a rate limiting issue.

The check I do prior to download was successful, but the actual download failed. It fixed itself without any intervention.

It may be possible that this error occurs for other reasons beyond throttling.

2020-10-28 11:01:05 INFO     Check 2FA Cookie
2020-10-28 11:01:05 INFO     Cookie exists, check expiry date
2020-10-28 11:01:05 INFO     Valid two factor authentication cookie found. Days until expiration: 89
2020-10-28 11:01:05 INFO     Check download directory mounted correctly
2020-10-28 11:01:19 INFO     Check for new files using password stored in keyring...
2020-10-28 11:01:53 INFO     Check successful
2020-10-28 11:01:53 INFO     No new files detected. Nothing to download
2020-10-28 11:01:53 INFO     Web cookie expires: 2020-12-26 @ 22:59:37
2020-10-28 11:01:53 INFO     Two factor authentication cookie expires: 2021-01-25 @ 22:59:53
2020-10-28 11:01:53 INFO     Next syncronisation at 23:01
2020-10-28 23:01:53 INFO     Check 2FA Cookie
2020-10-28 23:01:53 INFO     Cookie exists, check expiry date
2020-10-28 23:01:53 INFO     Valid two factor authentication cookie found. Days until expiration: 88
2020-10-28 23:01:53 INFO     Check download directory mounted correctly
2020-10-28 23:01:53 INFO     Check for new files using password stored in keyring...
/home/user/iCloud/2020/IMG_2141.HEIC
2020-10-28 23:02:31 INFO     Check successful
2020-10-28 23:02:31 INFO     Check detected 1 files requiring download. Verifying list accuracy
2020-10-28 23:02:31 INFO     Ignoring 0 files which have already been downloaded
2020-10-28 23:02:31 INFO     New files detected: 1
2020-10-28 23:02:31 INFO     Starting download of new files for user: user
2020-10-28 23:02:31 INFO     Downloading new files using password stored in keyring...
2020-10-28 23:02:32 DEBUG    Authenticating...
private db access disabled for this account.  Please wait a few minutes then try again.  The remote servers might be trying to throttle requests. (ACCESS_DENIED)
2020-10-28 23:02:39 ERROR    Error during download - Exit code: 1
2020-10-28 23:02:39 INFO     Sending Telegram failure notification
2020-10-28 23:02:39 INFO     Telegram failure notification sent successfully
2020-10-28 23:02:39 INFO     Web cookie expires: 2020-12-26 @ 22:59:37
2020-10-28 23:02:39 INFO     Two factor authentication cookie expires: 2021-01-25 @ 22:59:53
2020-10-28 23:02:39 INFO     Next syncronisation at 11:02
2020-10-29 11:02:39 INFO     Check 2FA Cookie
2020-10-29 11:02:39 INFO     Cookie exists, check expiry date
2020-10-29 11:02:39 INFO     Valid two factor authentication cookie found. Days until expiration: 88
2020-10-29 11:02:39 INFO     Check download directory mounted correctly
2020-10-29 11:02:39 INFO     Check for new files using password stored in keyring...
/home/user/iCloud/2020/IMG_2141.HEIC
2020-10-29 11:03:13 INFO     Check successful
2020-10-29 11:03:13 INFO     Check detected 1 files requiring download. Verifying list accuracy
2020-10-29 11:03:13 INFO     Ignoring 0 files which have already been downloaded
2020-10-29 11:03:13 INFO     New files detected: 1
2020-10-29 11:03:13 INFO     Starting download of new files for user: user
2020-10-29 11:03:13 INFO     Downloading new files using password stored in keyring...
2020-10-29 11:03:13 DEBUG    Authenticating...
2020-10-29 11:03:17 DEBUG    Looking up all photos and videos from album All Photos...
2020-10-29 11:03:17 INFO     Downloading 854 original photos and videos to /home/user/iCloud ...
2020-10-29 11:03:20 INFO     Downloading /home/user/iCloud/2020/IMG_2141.HEIC

[menkej]

No concrete explanation, yet. This is a known behaviour that happens from time to time. See also picklepete/pyicloud#194

Maybe some indexing job... Is this reproducible?

[benkrebs]

Hey all,
my lib is roughly 140.000 photos.
I never saw one file being downloaded. I always get the error immediately.

"private db access disabled for this account. Please wait a few minutes then try again. The remote servers might be trying to throttle requests. (ACCESS_DENIED)"

I waited hours, days and now 2 weeks.
I do use docker and as only env synchronisation_interval=21600 (and tz, user, folder, ...)
Is there a way how I can validate the cookie? I once did enter a wrong 2FA code and still got a cookie file?!

Any advice?

Thanks,
Ben

[menkej]

I never saw one file being downloaded. I always get the error immediately.
@benkrebs can you provide additional information on how you run this program and your python environment, please? Can you see your photos, when you log in into icloud.com with your browser?

[benkrebs]

Hey @menkej
thanks for your fast response.
1.) yes, I can see and browse the photos on iCloud.com:

2.) I am using boredazfcuk/icloudpd dockerized version (latest); running on a local ubuntu server

My config:
services:
icloudsync:
container_name: icloudsync
image: boredazfcuk/icloudpd
restart: unless-stopped
environment:
- TZ=Europe/Berlin
- user=xyz
- user_id=1000
- group=synusers
- group_id=1007
- apple_id=mail@xyz.com
- authentication_type=2FA
- folder_structure={:%Y}
- auto_delete=False
- recent_only=10
- synchronisation_interval=21600
hostname: icloudsync
volumes:
- xyz:/config
- abc:/home/benkrebs/iCloud

3.) then I init: docker exec -it icloudsync sync-icloud.sh --Initialise
-> what I saw here: even if I provide a wrong 2FA code I get a key and it proceeds to the error message

4.) Python inside the docker:

Hope that helps and thanks!