Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can no longer log in: Bad username or password #729

Closed
boredazfcuk opened this issue Dec 6, 2023 · 87 comments
Closed

Can no longer log in: Bad username or password #729

boredazfcuk opened this issue Dec 6, 2023 · 87 comments
Labels

Comments

@boredazfcuk
Copy link
Contributor

Overview

As of about 45mins ago, all four of my containers attempted a download, and all four failed to login.

After removing the keyring file and attempting to recreate it with icloud --username my@email.com the password is rejected. I've confirmed the e-mail and password combination is correct.

Steps to Reproduce

  1. Start container

Expected Behavior

Logs into icloud

Actual Behavior

Fails to login. If password is saved to the keyring, this error is generated:

  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 220, in authenticate
    req = self.session.post(
          ^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 105, in request
    self._raise_error(code, reason)
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 127, in _raise_error
    raise api_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/icloudpd_latest/bin/icloudpd", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/icloudpd/base.py", line 317, in main
    core(
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/icloudpd/base.py", line 744, in core
    icloud = authenticator(logger, domain)(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/icloudpd/authentication.py", line 31, in authenticate_
    icloud = pyicloud_ipd.PyiCloudService(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 204, in __init__
    self.authenticate()
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 228, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))

Context

Seems that Apple may have changed something. Can replicate on Alpine Linux 3.18.3 icloudpd 1.16.2 and Alpine Linux 3.18.5 and icloudpd 1.16.3

@boredazfcuk boredazfcuk added the bug label Dec 6, 2023
@sonicflame17
Copy link

I'm having the same issue. Hoping it's a temporary problem with iCloud itself. I was able to log in just yesterday with iCloudPD, and my configuration has not changed since then.

@gunner007cd
Copy link

gunner007cd commented Dec 7, 2023

Same issue as well, failure to login, even after changing password to something simple.

@AndreyNikiforov
Copy link
Collaborator

I can replicate the issue on docker with 1.16.3 without keyring, so it is general issue and most likely related to some changes on Apple side as my 1.16.3 was running for a number of days before the issue started.

@OZidkani
Copy link

OZidkani commented Dec 7, 2023

Same here

1 similar comment
@dohgren78
Copy link

Same here

@seankerrigan
Copy link

also experiencing the same issue with 1.16.2 as well as after upgrading to 1.16.3

@jckesser
Copy link

jckesser commented Dec 7, 2023

Same here

@benstn
Copy link

benstn commented Dec 7, 2023

Same issue - container has stopped downloading today

@oliplot
Copy link

oliplot commented Dec 7, 2023

Same issue here, stopped working yesterday Dec 6

@jrodmonaco
Copy link

Same. Following this thread for updates.

@gxander85
Copy link

Following

@petebocken
Copy link

Same here, following

@eastexxiao
Copy link

Same, following;

@staffankvisth
Copy link

I also get the same error on two different iCloud accounts. Suddenly stopped working 2 days ago. Both of my boredazfcuk/docker-icloudpd containers stopped and also manual execution of the script
~/.local/bin/icloudpd -u my@email.address -p ****** -d . --folder-structure={:%Y/%m/%d} --delete-after-download
fails with:
_2023-12-07 23:24:36 DEBUG Authenticating... 2023-12-07 23:24:36 ERROR Unknown reason Traceback (most recent call last): File "pyicloud_ipd/base.py", line 220, in authenticate File "requests/sessions.py", line 637, in post File "pyicloud_ipd/base.py", line 105, in request File "pyicloud_ipd/base.py", line 127, in _raise_error pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "starters/icloudpd.py", line 5, in File "click/core.py", line 1157, in call File "click/core.py", line 1078, in main File "click/core.py", line 1434, in invoke File "click/core.py", line 783, in invoke File "icloudpd/base.py", line 317, in main File "icloudpd/base.py", line 744, in core File "icloudpd/authentication.py", line 31, in authenticate_ File "pyicloud_ipd/base.py", line 204, in init File "pyicloud_ipd/base.py", line 228, in authenticate pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason')) [1357605] Failed to execute script 'icloudpd' due to unhandled exception!_

Logging in on the icloud.com website works as it should.
Does this mean there is no remedy for this problem yet? Has Apple changed something?

@syunlee66
Copy link

Same, following;

@Designator-Ol
Copy link

just following.

@meilon
Copy link

meilon commented Dec 8, 2023

just following.

Then please just please subscribe, nobody wants to get spammed with "just following" updates!

@MaPa80711
Copy link

Hi, maybe it will help to solve the problem. I have three icloud Accounts running and the issue is the same in every account.

icloudpd --directory /xxx/xxx/xxx/icloudpd_photo_backup --username xxxxxxxxxxxxx@icloud.com --password xxxxxxxx --log-level debug
2023-12-08 16:16:11 DEBUG Authenticating...
2023-12-08 16:16:12 ERROR Unknown reason
Traceback (most recent call last):
File "pyicloud_ipd/base.py", line 220, in authenticate
File "requests/sessions.py", line 637, in post
File "pyicloud_ipd/base.py", line 105, in request
File "pyicloud_ipd/base.py", line 127, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "starters/icloudpd.py", line 5, in
File "click/core.py", line 1157, in call
File "click/core.py", line 1078, in main
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "icloudpd/base.py", line 317, in main
File "icloudpd/base.py", line 744, in core
File "icloudpd/authentication.py", line 31, in authenticate_
File "pyicloud_ipd/base.py", line 204, in init
File "pyicloud_ipd/base.py", line 228, in authenticate
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))
[201833] Failed to execute script 'icloudpd' due to unhandled exception!

@espro1
Copy link

espro1 commented Dec 8, 2023

I'm experiencing this too. Started on the 6th at around 5pm EST.

@AndreyNikiforov
Copy link
Collaborator

AndreyNikiforov commented Dec 8, 2023

I see the following paths:

  1. dig into current code/protocol to see if minor tweak will make it working
    • rationale: fastest and least invasive
    • concerns: low chances
    • chances: low; suspect Apple turned off old protocol that icloudpd uses (they support OAUTH on web for some time)
    • investments: low
  2. use original pyicloud lib
    • rationale: IIUC pyicloud was updated to support OAUTH some time ago
    • concerns: may break some of our code
    • chances: mid-high; need to confirm that pyicloud works with current Apple services first
    • investments: mid; need to update tests and that has been a challenge in the prev attempts to bring latest pyicloud
  3. use own code to support latest Apple auth
    • rationale: guaranteed success, opens door for other long-requested improvements
    • concerns: may force larger updates, leading to breaking changes, long alpha-beta cycle
    • chances: high
    • investments: high

Is anybody interested in 1) and/or 2)? I plan to allocate some time in the next number of days to look into this issue and will invest in 1 & 3. IMO it is okay for multiple ppl to look at the same path as there are always edge cases and different approaches that may give different or complementing outcomes.

@cfurrow
Copy link

cfurrow commented Dec 8, 2023

@AndreyNikiforov Thanks for the summary / plans of attack.

To help guide others that may want to debug, a possible good test case to try is within the tests/test_authentication.py tests, test_2sa_required. Per the instructions in the code, one can delete the pre-recorded VCR response file tests/vcr_cassettes/auth_requires_2sa.yml, and fill in their actual username/password in the test, and then run scripts/test (SEE https://github.com/icloud-photos-downloader/icloud_photos_downloader/blob/master/CONTRIBUTING.md#setting-up-the-development-environment on how to setup your local environment to test this code)

The tests will fail, and the VCR recording (aka "cassette") will be updated with the actual response received from icloud.

BE SURE TO NOT SHARE YOUR USERNAME/PASSWORD WHICH ARE PRESENT IN THE VCR RECORDING!

For instance, after doing the above, this is the diff of the VCR cassette. I can't glean much useful information from the response yet:

  1. My new yml file has lots of whitespace/formatting differences vs the original
  2. The new response is less verbose than the expected "good" response
  3. I'm not as familiar with Python or these icloud related Python libraries.
"git diff" of the VCR yaml file
diff --git a/tests/vcr_cassettes/auth_requires_2sa.yml b/tests/vcr_cassettes/auth_requires_2sa.yml
index d0bbf78..ad75ec9 100644
--- a/tests/vcr_cassettes/auth_requires_2sa.yml
+++ b/tests/vcr_cassettes/auth_requires_2sa.yml
@@ -1,37 +1,59 @@
 interactions:
 - request:
-    body: !!python/unicode '{"apple_id": "jdoe@gmail.com", "password": "password1",
-      "extended_login": false}'
+    body: '{"apple_id": "jdoe@gmail.com", "password": "password1", "extended_login":
+      false}'
     headers:
-      Accept: ['*/*']
-      Accept-Encoding: ['gzip, deflate']
-      Connection: [keep-alive]
-      Content-Length: ['88']
-      Origin: ['https://www.icloud.com']
-      Referer: ['https://www.icloud.com/']
-      User-Agent: [Opera/9.52 (X11; Linux i686; U; en)]
+      Accept:
+      - '*/*'
+      Accept-Encoding:
+      - gzip, deflate
+      Connection:
+      - keep-alive
+      Content-Length:
+      - '90'
+      Origin:
+      - https://www.icloud.com
+      Referer:
+      - https://www.icloud.com/
+      User-Agent:
+      - Opera/9.52 (X11; Linux i686; U; en)
     method: POST
-    uri: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=17DHotfix5&clientMasteringNumber=17DHotfix5&clientId=EC5646DE-9423-11E8-BF21-14109FE0B321&ckjsVersion=2.0.5&ckjsBuildVersion=17DProjectDev77
+    uri: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=17DHotfix5&clientMasteringNumber=17DHotfix5&ckjsBuildVersion=17DProjectDev77&ckjsVersion=2.0.5&clientId=EC5646DE-9423-11E8-BF21-14109FE0B321
   response:
-    body: {string: !!python/unicode '{"dsInfo":{"lastName":"Doe","iCDPEnabled":false,"dsid":"123456789","hsaEnabled":true,"ironcadeMigrated":true,"locale":"en-us_US","brZoneConsolidated":false,"isManagedAppleID":false,"gilligan-invited":"true","appleIdAliases":["jdoe@icloud.com"],"hsaVersion":2,"isPaidDeveloper":true,"countryCode":"USA","notificationId":"12341234-1234-1234-1234-143241234123","primaryEmailVerified":true,"aDsID":"12341234123412341234","locked":false,"hasICloudQualifyingDevice":true,"primaryEmail":"jdoe@gmail.com","appleIdEntries":[{"isPrimary":true,"type":"EMAIL","value":"jdoe@gmail.com"}],"gilligan-enabled":"true","fullName":"John
-        Doe","languageCode":"en-us","appleId":"jdoe@gmail.com","firstName":"John","iCloudAppleIdAlias":"jdoe@icloud.com","notesMigrated":true,"hasPaymentInfo":true,"pcsDeleted":false,"appleIdAlias":"","brMigrated":true,"statusCode":2},"hasMinimumDeviceForPhotosWeb":true,"iCDPEnabled":false,"webservices":{"reminders":{"url":"https://p10-remindersws.icloud.com:443","status":"active"},"notes":{"url":"https://p10-notesws.icloud.com:443","status":"active"},"mail":{"url":"https://p10-mailws.icloud.com:443","status":"active"},"ckdatabasews":{"pcsRequired":true,"url":"https://p10-ckdatabasews.icloud.com:443","status":"active"},"photosupload":{"pcsRequired":true,"url":"https://p10-uploadphotosws.icloud.com:443","status":"active"},"photos":{"pcsRequired":true,"uploadUrl":"https://p10-uploadphotosws.icloud.com:443","url":"https://p10-photosws.icloud.com:443","status":"active"},"drivews":{"pcsRequired":true,"url":"https://p10-drivews.icloud.com:443","status":"active"},"uploadimagews":{"url":"https://p10-uploadimagews.icloud.com:443","status":"active"},"schoolwork":{},"cksharews":{"url":"https://p10-ckshare.icloud.com:443","status":"active"},"findme":{"url":"https://p10-fmipweb.icloud.com:443","status":"active"},"ckdeviceservice":{"url":"https://p10-ckdevice.icloud.com:443"},"iworkthumbnailws":{"url":"https://p10-iworkthumbnailws.icloud.com:443","status":"active"},"calendar":{"url":"https://p10-calendarws.icloud.com:443","status":"active"},"docws":{"pcsRequired":true,"url":"https://p10-docws.icloud.com:443","status":"active"},"settings":{"url":"https://p10-settingsws.icloud.com:443","status":"active"},"ubiquity":{"url":"https://p10-ubiquityws.icloud.com:443","status":"active"},"streams":{"url":"https://p10-streams.icloud.com:443","status":"active"},"keyvalue":{"url":"https://p10-keyvalueservice.icloud.com:443","status":"active"},"archivews":{"url":"https://p10-archivews.icloud.com:443","status":"active"},"push":{"url":"https://p10-pushws.icloud.com:443","status":"active"},"iwmb":{"url":"https://p10-iwmb.icloud.com:443","status":"active"},"iworkexportws":{"url":"https://p10-iworkexportws.icloud.com:443","status":"active"},"geows":{"url":"https://p10-geows.icloud.com:443","status":"active"},"account":{"iCloudEnv":{"shortId":"p","vipSuffix":"p"},"url":"https://p10-setup.icloud.com:443","status":"active"},"fmf":{"url":"https://p10-fmfweb.icloud.com:443","status":"active"},"contacts":{"url":"https://p10-contactsws.icloud.com:443","status":"active"}},"pcsEnabled":true,"configBag":{"urls":{"accountCreateUI":"https://appleid.apple.com/widget/account/?widgetKey=12312412412341234123412341234123412341234#!create","accountLoginUI":"https://idmsa.apple.com/appleauth/auth/signin?widgetKey=83545bf919730e51dbfba24e7e8a78d2","accountLogin":"https://setup.icloud.com/setup/ws/1/accountLogin","accountRepairUI":"https://appleid.apple.com/widget/account/?widgetKey=12312412412341234123412341234123412341234#!repair","downloadICloudTerms":"https://setup.icloud.com/setup/ws/1/downloadLiteTerms","repairDone":"https://setup.icloud.com/setup/ws/1/repairDone","vettingUrlForEmail":"https://id.apple.com/IDMSEmailVetting/vetShareEmail","accountCreate":"https://setup.icloud.com/setup/ws/1/createLiteAccount","getICloudTerms":"https://setup.icloud.com/setup/ws/1/getTerms","vettingUrlForPhone":"https://id.apple.com/IDMSEmailVetting/vetSharePhone"},"accountCreateEnabled":"true"},"hsaTrustedBrowser":false,"appsOrder":["mail","contacts","calendar","photos","iclouddrive","notes2","reminders","pages","numbers","keynote","newspublisher","fmf","find","settings"],"version":2,"isExtendedLogin":false,"pcsServiceIdentitiesIncluded":false,"hsaChallengeRequired":true,"requestInfo":{"country":"TH","timeZone":"GMT+7","isAppleInternal":true},"pcsDeleted":false,"iCloudInfo":{"SafariBookmarksHasMigratedToCloudKit":false},"apps":{"calendar":{},"reminders":{},"keynote":{"isQualifiedForBeta":true},"settings":{"canLaunchWithOneFactor":true},"mail":{},"numbers":{"isQualifiedForBeta":true},"photos":{},"pages":{"isQualifiedForBeta":true},"find":{"canLaunchWithOneFactor":true},"notes2":{},"iclouddrive":{},"newspublisher":{"isHidden":true},"fmf":{},"contacts":{}}}'}
+    body:
+      string: '{"success":false,"error":1}'
     headers:
-      access-control-allow-credentials: ['true']
-      access-control-allow-origin: ['https://www.icloud.com']
-      access-control-expose-headers: [X-Apple-Request-UUID, Via]
-      apple-originating-system: [UnknownOriginatingSystem]
-      apple-seq: ['0']
-      apple-tk: ['false']
-      cache-control: ['no-cache, no-store, private']
-      connection: [keep-alive]
-      content-length: ['4895']
-      content-type: [application/json; charset=UTF-8]
-      date: ['Mon, 30 Jul 2018 19:00:39 GMT']
-      server: [AppleHttpServer/2f080fc0]
-      strict-transport-security: [max-age=31536000; includeSubDomains]
-      via: ['icloudedge:si03p00ic-ztde010417:7401:18RC341:Singapore']
-      x-apple-jingle-correlation-key: [SJHIUN7879234KJHH8JBH]
-      x-apple-request-uuid: [NISUHFIOSUHFOSIDUHFOSIDF]
-      x-responding-instance: ['setupservice:328457238759328579234875']
-    status: {code: 200, message: OK}
+      Access-Control-Allow-Credentials:
+      - 'true'
+      Access-Control-Allow-Origin:
+      - https://www.icloud.com
+      Cache-Control:
+      - no-cache, no-store, private
+      Connection:
+      - keep-alive
+      Content-Type:
+      - application/json; charset=UTF-8
+      Date:
+      - Fri, 08 Dec 2023 16:37:26 GMT
+      Server:
+      - AppleHttpServer/78689afb4479
+      Strict-Transport-Security:
+      - max-age=31536000; includeSubDomains
+      X-Apple-Edge-Response-Time:
+      - '45'
+      X-Apple-Request-UUID:
+      - e7626cfc-bb0c-4ccd-986d-b9327f8bcef3
+      X-Responding-Instance:
+      - setupservice:44700503:pv51p47ic-qukt21032501:8003:2404B363:5934c9004de5
+      access-control-expose-headers:
+      - X-Apple-Request-UUID,Via
+      content-length:
+      - '27'
+      via:
+      - 631194250daa17e24277dea86cf30319:7ecc2ff259156058e890ba5cf00be461:uschi7
+      x-apple-user-partition:
+      - '47'
+    status:
+      code: 421
+      message: Misdirected Request
 version: 1

However, the HTTP response code of "421: Misdirected Request" may be the key, and indicate that icloud has updated its API and therefore pyicloud needs updating to reflect that.

Back in Oct 2023, someone submitted an issue to pyicloud regarding the 421 response: picklepete/pyicloud#441, and 4 days ago there is a possible fix: picklepete/pyicloud#406 (it's not merged yet)

My system:

  • macOS Sonoma 14.1.2 (Apple M3 Macbook Pro)
  • Python 3.12.0

@robflate
Copy link

robflate commented Dec 9, 2023

Total speculation but I wonder if this has anything to do with Apple's attempts to block iMessage on Android;

https://www.theverge.com/2023/12/8/23994089/apple-beeper-mini-android-blocked-imessage-app

@nscheer
Copy link

nscheer commented Dec 9, 2023

My guess is that it has something to do with the recent support for passkey logins.
At least for the iCloud website it asks for passkey and logging in via password takes one more click.

Such a thing would be unusual for an api, nevertheless a change that requires specification of the desired login method sounds very likely.

@cutzenfriend
Copy link

My guess is that it has something to do with the recent support for passkey logins. At least for the iCloud website it asks for passkey and logging in via password takes one more click.

Such a thing would be unusual for an api, nevertheless a change that requires specification of the desired login method sounds very likely.

This sounds like the issue for me...

@charlesb87
Copy link

Same error here running on Windows x64 with 1.16.3 version

@OrZidkani
Copy link

OrZidkani commented Dec 15, 2023

@NGC3982
Here you go: https://github.com/OrZidkani/icloudpd_auth_fix

Make it easier to run.

Big thanks to @scaraebeus

@lost-while-translating
Copy link

Is there a update for the PIP (windows) Version? I don't use docker.
Or do I have to wait for next friday?

Thanks for all the efforts to fix it.

Frank

@Eagle248
Copy link

Does anyone know how to apply this fix for docker running in synology nas?

@scaraebeus
Copy link
Contributor

Is there a update for the PIP (windows) Version? I don't use docker.
Or do I have to wait for next friday?

Not sure on timing for the official update. Unfortunately for my fix to be pulled in, there are a few things that need to be in place on my branch for it to be accepted. I've been working through updating the various tests and code coverage and in doing so I am finding some other parts of the pyicloud_ipd/base.py auth flow that are no longer working correctly due to (possibly) recent changes in how the responses are coming through.

I'm working through it but it will take some time - especially with the holidays coming up.

The only option I'm aware of at this time is to consume the auth_fix branch directly if your environment and setup allows you - with the risk that it has yet to be fully tested and vetted.

@AndreyNikiforov
Copy link
Collaborator

@scaraebeus thanks a lot for your hard work and dedication. It is well appreciated by many.

@revpdev
Copy link

revpdev commented Dec 16, 2023

@scaraebeus I join previous comment to thank you for your great contribution here, I patched the main branch with #734, built and installed it with pip and got it working on both raspbian/debian bullseye (11) and bookworm (12) . I am also using edits as described in issue #249 so that I can download both edited media (the ones we see on the phone and in iCloud) and original media in a regular cron backup script to my pi-based NAS, which makes this script critical to me. thanks again and congrats for keeping it working !

@kohldavido
Copy link

Is there a update for the PIP (windows) Version? I don't use docker.
Or do I have to wait for next friday?

Not sure on timing for the official update. Unfortunately for my fix to be pulled in, there are a few things that need to be in place on my branch for it to be accepted. I've been working through updating the various tests and code coverage and in doing so I am finding some other parts of the pyicloud_ipd/base.py auth flow that are no longer working correctly due to (possibly) recent changes in how the responses are coming through.

I'm working through it but it will take some time - especially with the holidays coming up.

The only option I'm aware of at this time is to consume the auth_fix branch directly if your environment and setup allows you - with the risk that it has yet to be fully tested and vetted.

Works on Debian 12 with docker for me until some period of time where the Apple servers either:
hang up, as icloudpd appears to idle in the middle of downloading
attempt negotiate 2fa and send me an authentication code while icloudpd is in the middle of downloading
In both cases there isn't any debug or info logging printed indicating why icloudpd paused, but my network traffic goes to near zero, so I'm guessing the sessions is closed by apple.

Additionally, I'm attempting to download over 180k photos, so that might be why apple is hanging up on me periodically. It doesn't explain the 2fa prompt while icloudpd is running and actively downloading though.

The expected behavior where re-running icloupd picks up where it left off results in diminishing returns:
1st session, 4k downloads.
2nd session: 4k skipped and 2k additional downloads.
3rd session: 6k skipped and 500 additional completed downloads.
And so on until no additional downloads complete at all. This might be from pyicloud json handling?

@yrahul3910
Copy link

yrahul3910 commented Dec 17, 2023

Works! For those using the above solution, a minor change is necessary (I'm on macOS): in the Shell script, change /mnt/iCloud to $(pwd). Then, in Docker Desktop, add the current directory under Settings --> Resources --> File sharing.

ETA: The $(pwd) comes from this documentation.

Edit 2: Don't use $(pwd)! The documentation version works because it refers to where you are, but in the script, it cds to a temp directory so it would download there. Change it to the directory you want the downloads to go to!

@tanookiben
Copy link

Is saving to the keychain not working for anyone else with the auth_fix branch?

Save password in keyring?  [y/N]: y
Two-step authentication required. Your trusted devices are:
  0: SMS to ********##
Which device would you like to use? [0]:

Previously I was able to see an option to use a 2FA code instead of SMS

@yrahul3910
Copy link

Is saving to the keychain not working for anyone else with the auth_fix branch?

Save password in keyring?  [y/N]: y
Two-step authentication required. Your trusted devices are:
  0: SMS to ********##
Which device would you like to use? [0]:

Previously I was able to see an option to use a 2FA code instead of SMS

Steps to reproduce? With the branch I just got a 2FA code on my devices like normal, and it prompted me in the terminal. I don't remember it giving me an option between SMS or 2FA code, though.

@tanookiben
Copy link

tanookiben commented Dec 18, 2023

Is saving to the keychain not working for anyone else with the auth_fix branch?

Save password in keyring?  [y/N]: y
Two-step authentication required. Your trusted devices are:
  0: SMS to ********##
Which device would you like to use? [0]:

Previously I was able to see an option to use a 2FA code instead of SMS

Steps to reproduce? With the branch I just got a 2FA code on my devices like normal, and it prompted me in the terminal. I don't remember it giving me an option between SMS or 2FA code, though.

With icloudpd --username "USERNAME" --password "PASSWORD" it asks me for a 2FA right away

When I try to save in the keyring it only offers SMS instead of 2FA:

docker run --rm -it --name icloud \
  -v /tmp/icloudpd/data:/data \
  -v /tmp/icloudpd/cookies:/cookies \
  -e "TZ=America/Los_Angeles" \
  icloudpd:auth_fix \
  icloud --username USERNAME
Enter iCloud password for USERNAME: 
Save password in keyring?  [y/N]: y
Two-step authentication required. Your trusted devices are:
  0: SMS to ********##
Which device would you like to use? [0]

Edit: for now I've just resolved myself to running icloudpd with a username and password until everything gets sorted out in the main branch

@scaraebeus
Copy link
Contributor

It appears the command line flow for the base pyicloud_ipd service (which you are invoking with icloud --username USERNAME) was never updated in the pyicloud - 1.0.0 release to use the new 2FA feature, so it prompts as if it is expecting 2SA (choosing a device to send the code to first).

It also appears, from what I can tell, the icloudpd flow doesn't offer to save in the keyring directly.

Based on the above, as you've saved the password in the keyring now, try using the icloudpd directly without the --password option and it should retrieve from the keyring - and ask for the 2FA without prompting for a device selection.

The option to offer saving in the keyring could be moved into the icloudpd flow if needed. Not sure if there is already an issue for that or not.

@boredazfcuk
Copy link
Contributor Author

boredazfcuk commented Dec 18, 2023

Previous behaviour was that saving to the keychain required SMS MFA, it did not have an option for Apple's built in MFA.

Performing a download would then trigger a second MFA prompt, which had two options for MFA; SMS and Apple.

I noticed this behaviour when attempting to relocate the MFA cookie in my container from $HOME/.local to /config. Took me a lot of deletes/recreates to get it working reliably.

Edit: I think it's also worth mentioning that I actually had two SMS numbers I could use for three of my containers. I have added my phone number as a trusted number to the three accounts of my family members. This allows me to re-authenticate their containers without me needing access to their devices. Very useful feature.

@scaraebeus
Copy link
Contributor

Previous behaviour was that saving to the keychain required SMS MFA, it did not have an option for Apple's built in MFA.

Performing a download would then trigger a second MFA prompt, which had two options for MFA; SMS and Apple.

I noticed this behaviour when attempting to relocate the MFA cookie in my container from $HOME/.local to /config. Took me a lot of deletes/recreates to get it working reliably.

Edit: I think it's also worth mentioning that I actually had two SMS numbers I could use for three of my containers. I have added my phone number as a trusted number to the three accounts of my family members. This allows me to re-authenticate their containers without me needing access to their devices. Very useful feature.

The current pyicloud - 1.0.0 2SA/2FA implementation doesn't appear to take this into account. The command line implementation for pyicloud doesn't even test for 2FA (hence why it drops straight into asking which device to use). The underlying hooks in the pyicloud 2FA implementation also limit the flexibility for the above use case.

On top of this, there are two separate endpoints used to validate codes depending on whether the 2SA or 2FA flow is triggered. It's possible the endpoints don't really matter and the same endpoint can be used regardless - I'll have to test.

Proposed solution:

  • Leave the default behavior as is and add a switch to list devices to alternatively send a code to
    • When initiating icloudpd if you have a default iDevice and the account is enabled for HSA2 (I believe all accounts are now enabled for HSA2 at this point), prompt for 2FA code directly as it should be auto sent to the device
      • This would likely cover a majority of use cases
      • There is an edge case where an account may not have an iDevice associated with it - in this case, there is likely still an SMS number to send to (potentially Android instead of iDevice). This is already covered in the default behavior and automatically triggers the list device flow.
    • Adding a 'list devices' switch (icloudpd --list-devices) could trigger the flow to collect and show the available devices associated with the account, allowing the user to select a different device to send a code to (and displaying the option to bypass this and enter the code if you decide to go with the code auto sent to the default device)
      • With the above switch, a code would still likely be auto sent to an iDevice - choosing a different device (or SMS) to send a code to would need to be tested to verify which validation endpoint accepts the second code (manual request) - (or if both do)

@tanookiben
Copy link

Based on the above, as you've saved the password in the keyring now, try using the icloudpd directly without the --password option and it should retrieve from the keyring - and ask for the 2FA without prompting for a device selection.

I think in theory this should work but the code I get via SMS can never be verified successfully:

Enter iCloud password for USERNAME: 
Save password in keyring?  [y/N]: y
Two-step authentication required. Your trusted devices are:
  0: SMS to ********##
Which device would you like to use? [0]: 0
Please enter validation code: ######
Failed to verify verification code

@scaraebeus
Copy link
Contributor

I retract my statement about the pyicloud - 1.0.0 not taking the 2SA/2FA into account, it does, the minimal fix in the auth_fix branch does not include the updated cmdline.py to expose this in the pyicloud_ipd direct invoking.

@tanookiben - when you attempt to verify the SMS code, have you also already received an auto-generated code to one of your iDevices?

If so, what happens if you enter that code at the prompt instead of the one sent through SMS?

This would at give me a bit of a clue as to where there is something end point related and/or if it's still expecting the first code.

@boredazfcuk
Copy link
Contributor Author

I'm not near my laptop to test, but IIRC, mine crashed at the point where it attempted to list the phone numbers.

@scaraebeus
Copy link
Contributor

Update: All test are now passing for the auth_fix branch with 99% code coverage (matching the master branch). Hopefully this will allow for the branch to be pulled into the master.

The 2SA/2FA cases noted above will still need to be looked into but I'm hopeful these can be treated as separate issues and allow the core fix to be merged for those that are waiting on this due to various package distribution requirements.

Happy Holidays!

@AndreyNikiforov
Copy link
Collaborator

1.17.0 released and has the fix

@boredazfcuk
Copy link
Contributor Author

I've built a new container with iCloud 1.17.0, but the behaviour is not the same. I am no longer presented with the option to perform SMS based multifactor authentication. I also receive a warning about it not being able to parse JSON, which I've not seen before but I'm not too worried about, as the application downloads the photos regardless:

2023-12-20 12:08:01 DEBUG    Configure password
2023-12-20 12:08:01 DEBUG    Adding password to keyring file: /config/python_keyring/keyring_pass.cfg
2023-12-20 12:08:01 DEBUG    Switched to icloudpd: 1.17.0
Enter iCloud password for my@email.com:
Save password in keyring? [y/N]: y

Two-step authentication required.
Please enter validation code
(string) --> 137446

2023-12-20 12:08:46 INFO     Starting container initialisation
2023-12-20 12:08:46 DEBUG    Generate MFA cookie using password stored in keyring file
2023-12-20 12:08:47 DEBUG    Switched to icloudpd: 1.17.0
2023-12-20 12:08:47 ERROR    Authentication required for Account. (421)
Please enter two-factor authentication code: 119033
2023-12-20 12:09:20 WARNING  Failed to parse response with JSON mimetype

Is SMS based multifactor authentication no longer possible?

@scaraebeus
Copy link
Contributor

scaraebeus commented Dec 20, 2023

Is SMS based multifactor authentication no longer possible?

With the new method used in pyicloud - 1.0.0 it splits the flow between 2SA (using SMS) and 2FA (Auto-sent code) based on whether the iCloud account itself is set for HSA version 2 (it appears all accounts may now be set for this).

The way the base pyicloud does this check, assuming all accounts are now set as HSA version 2, there is actually no way the 2SA flow would ever be called. I uncovered this awhile ago doing some other testing with an account that doesn't have an iDevice (using the free iCloud service as standalone). In this scenario, I identified a flag noting if the account has a qualifying device. Adding this in to the 2FA check (requiring this to be true) now allows the flow to fall back on the 2SA - but only if there isn't a qualifying device on the account.

I've started working on an enhancement to allow selecting an SMS for 2sa validation. A 2fa code will still be sent to the iDevice automatically, but you should be able to choose an SMS, get a code there, and validate with that instead.

If you want to check that out, see the 2fa_enhancements branch under scaraebeus/icloud_photos_downloader - keep in mind this is experimental though. In that branch, you can use the --list-devices option to present the list of SMS devices and choose to send a code there (or choose to enter the auto sent 2FA code).

It may be worth starting a separate issue for this to be tracked outside of this thread.

@kmcphillips
Copy link

I depend on this tool and have been following this issue. Thanks so much for all the hard work and getting this fixed so quickly. You are appreciated! 🙏

@tanookiben
Copy link

I've built a new container with iCloud 1.17.0, but the behaviour is not the same. I am no longer presented with the option to perform SMS based multifactor authentication. I also receive a warning about it not being able to parse JSON, which I've not seen before but I'm not too worried about, as the application downloads the photos regardless:

2023-12-20 12:08:01 DEBUG    Configure password
2023-12-20 12:08:01 DEBUG    Adding password to keyring file: /config/python_keyring/keyring_pass.cfg
2023-12-20 12:08:01 DEBUG    Switched to icloudpd: 1.17.0
Enter iCloud password for my@email.com:
Save password in keyring? [y/N]: y

Two-step authentication required.
Please enter validation code
(string) --> 137446

2023-12-20 12:08:46 INFO     Starting container initialisation
2023-12-20 12:08:46 DEBUG    Generate MFA cookie using password stored in keyring file
2023-12-20 12:08:47 DEBUG    Switched to icloudpd: 1.17.0
2023-12-20 12:08:47 ERROR    Authentication required for Account. (421)
Please enter two-factor authentication code: 119033
2023-12-20 12:09:20 WARNING  Failed to parse response with JSON mimetype

Is SMS based multifactor authentication no longer possible?

Can you share the command you used here? I'm able to save credentials to the keyring with icloud --username "USERNAME" but icloudpd --username "USERNAME" without a --password flag always prompts me for a password

@dustinziska
Copy link

Just checking in on this issue, tried the lastest release today and it seems like it is still broken. Was #734, supposed to resolve this issue?

@AndreyNikiforov
Copy link
Collaborator

Just checking in on this issue, tried the lastest release today and it seems like it is still broken

If you see the same issue then post here all details. Otherwise open new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests