`upload_to_testflight` action hangs with "Waiting for App Store Connect to finish processing the new build ..." #20645

mathaeus · 2022-09-12T09:06:18Z

New Issue Checklist

Updated fastlane to the latest version
I read the Contribution Guidelines
I read docs.fastlane.tools
I searched for existing GitHub issues

Issue Description

upload_to_testflight action hangs with the following message although the build has been processed

Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

Since up until recently this worked just fine it could also just be an App Store Connect issue 🤷‍♂️.
Tried several times with different apps.

Command executed

I basically have such a lane.

  lane :submit_build_for_beta_review_and_distribute do 
      app_store_connect_api_key(
        key_id: "keyid",
        issuer_id: "issuerid",
        key_content: "keycontent",
        is_key_content_base64: true,
        duration: 1200,
        in_house: false
      )

    upload_to_testflight(
      beta_app_review_info: {
        contact_email: "foo@bar.com",
        contact_first_name: "foo",
        contact_last_name: "bar",
        contact_phone: "+1234",
        demo_account_name: "foo",
        demo_account_password: "bar",
        notes: "foo bar"
      },
      app_version: version_name,
      build_number: build_number,
      submit_beta_review: true, 
      distribute_only: true,
      notify_external_testers: true,
      groups: ["foo", "bar"]
    )
end

Complete output when running fastlane, including the stack trace and command used

Here's our output from CI with some data anonymised

#!/bin/bash --login -eo pipefail
bundle exec fastlane submit_build_for_beta_review_and_distribute
[⠋] 🚀 [⠙] 🚀 [⠹] 🚀 [⠸] 🚀 [⠼] 🚀 [⠴] 🚀 [⠦] 🚀 [⠧] 🚀 [⠇] 🚀 [✔] 🚀 
+----------------------------------+---------+------------------+
|                         Used plugins                          |
+----------------------------------+---------+------------------+
| Plugin                           | Version | Action           |
+----------------------------------+---------+------------------+
| fastlane-plugin-aws_s3           | 2.1.0   | aws_s3           |
| fastlane-plugin-update_xcodeproj | 1.0.1   | update_xcodeproj |
+----------------------------------+---------+------------------+
[07:57:38]: Sending anonymous analytics information

[07:57:38]: Learn more at https://docs.fastlane.tools/#metrics

[07:57:38]: No personal or sensitive data is sent.

[07:57:38]: You can disable this by adding opt_out_usage at the top of your Fastfile

[07:57:39]: ----------------------------------------

[07:57:39]: --- Step: Verifying fastlane version ---

[07:57:39]: ----------------------------------------

[07:57:39]: Your fastlane version 2.209.1 matches the minimum requirement of 2.170.0  ✅

[07:57:39]: ------------------------------

[07:57:39]: --- Step: default_platform ---

[07:57:39]: ------------------------------

[07:57:39]: Driving the lane 'ios submit_build_for_beta_review_and_distribute' 🚀

[07:57:39]: -----------------------------

[07:57:39]: --- Step: setup_circle_ci ---

[07:57:39]: -----------------------------

[07:57:39]: Creating temporary keychain: "fastlane_tmp_keychain".

[07:57:39]: $ security list-keychains -d user

[07:57:39]: ▸ "/Users/distiller/Library/Keychains/fastlane_tmp_keychain-db"

[07:57:39]: Found keychain '/Users/distiller/Library/Keychains/fastlane_tmp_keychain-db' in list-keychains, adding to search list skipped

[07:57:39]: Enabling match readonly mode.

[07:57:39]: Trying to create AppStoreConnectAPIKey via CI ENV variables...

[07:57:39]: ---------------------------------------

[07:57:39]: --- Step: app_store_connect_api_key ---

[07:57:39]: ---------------------------------------

[07:57:39]: --------------------------------------

[07:57:39]: --- Step: get_ipa_info_plist_value ---

[07:57:39]: --------------------------------------

[07:57:39]: --------------------------------------

[07:57:39]: --- Step: get_ipa_info_plist_value ---

[07:57:39]: --------------------------------------

[07:57:39]: Submitting foobar 1.0.0/1234 for review and distributing to external testers.

[07:57:39]: ----------------------------------

[07:57:39]: --- Step: upload_to_testflight ---

[07:57:39]: ----------------------------------

[07:57:39]: Creating authorization token for App Store Connect API

[07:57:41]: Waiting for processing on... app_id: 12345678, app_version: 1.0.0, build_version: 1234, platform: IOS

[07:57:43]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

[07:58:14]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

[07:58:45]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

[07:59:17]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

[07:59:48]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

[08:00:19]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

[08:00:50]: Waiting for App Store Connect to finish processing the new build (1.0.0 - 1234) for IOS

...

The text was updated successfully, but these errors were encountered:

fastlane-bot · 2022-09-12T09:07:02Z

It seems like you have not included the output of fastlane env
To make it easier for us help you resolve this issue, please update the issue to include the output of fastlane env 👍

hendych · 2022-09-12T09:16:16Z

Same here,
My build always Waiting for App Store Connect to finish processing the new build (xxx - xxx009) for IOS even the build is already done processing.

Baza207 · 2022-09-12T09:17:18Z

Having the same issue. Seems to have started last Friday 9th Sept as builds from the Thursday 8th were fine.

Builds this morning having the same issues, I can see them in App Store Connect but DevOps ends up timing out due to stuck at this step where Fastlane isn't able to see the build has finished processing. 😞

baderbenzribia · 2022-09-12T09:31:19Z

Same here too.
We can see the build on TestFlight but Fastlane keep saying Waiting for App Store Connect to finish processing the new build...

mathaeus · 2022-09-12T09:39:19Z

Thanks @hendych @Baza207 @baderbenzribia for confirming that you have the same issue 👍

mathaeus · 2022-09-12T10:01:32Z

ℹ️ Submitting the same build for App Store review with deliver works without any problem.

unixvb · 2022-09-12T10:20:34Z

The same, using latest fastlane (2.209.1)

readyruncode · 2022-09-12T10:40:54Z

We see the same issue.
Build stays at "Waiting for App Store Connect to finish processing the new build" and times out after 5 hours.
Builds show up in TestFlight as Ready to Submit, but they seem to be missing tester groups and change log.
When we try to submit for review the build gets labeled Invalid Binary.

Baza207 · 2022-09-12T10:56:01Z

ℹ️ Submitting the same build for App Store review with deliver works without any problem.

Using 2.209.1 here as well, but via the Azure DevOps Apple App Store task, so can't switch to deliver without re-writing our pipelines. 😞

Pretty sure it's not Fastlane though as using the same version before Friday last week worked fine. Feels very much like something on Apple's end that got pushed last thing Thursday in the US that might have broken the API. Though I don't have the ability (at the moment) to test it myself, the API endpoints don't look like they've changed in Apple's docs. https://developer.apple.com/documentation/appstoreconnectapi/internalbetastate

mathaeus · 2022-09-12T11:06:32Z

When we try to submit for review the build gets labeled Invalid Binary.

@readyruncode you mean if you try to submit manually?

I was able to submit the build for BETA review manually without any problem (it disrupts our CI pipeline, but well ... 😅)

mathaeus · 2022-09-12T11:13:08Z

Pretty sure it's not Fastlane though as using the same version before Friday last week worked fine

Yes, it worked without any problem last week for us as well. My first thought also was that (parts of the) ASC APIs are either temporarily broken or Apple changed some things under the hood and now the fastlane code needs to be adapted or something.

I have too little knowledge about the internals of fastlane to debug the communication with ASC API. Guess that would show the underlying problem.

Baza207 · 2022-09-12T11:24:57Z

Pretty sure it's not Fastlane though as using the same version before Friday last week worked fine

Yes, it worked without any problem last week for us as well. My first thought also was that (parts of the) ASC APIs are either temporarily broken or Apple changed some things under the hood and now the fastlane code needs to be adapted or something.

I have too little knowledge about the internals of fastlane to debug the communication with ASC API. Guess that would show the underlying problem.

I don't have much either, but from what I can see the Fastlane model here:
https://github.com/fastlane/fastlane/blob/e8d30e3494024b6d4ee81bf21448991779ca278d/spaceship/lib/spaceship/connect_api/models/build_beta_detail.rb

Maps directly to the App Store Connect API docs about build state here:
https://developer.apple.com/documentation/appstoreconnectapi/internalbetastate
https://developer.apple.com/documentation/appstoreconnectapi/externalbetastate

Sadly I don't have the time right now to look any further into this and see what the endpoints from Apple actually return (https://developer.apple.com/documentation/appstoreconnectapi/read_build_beta_detail_information).

Posting the links here for anyone else who might have the time/ability to test and see what does/doesn't get returned. 😅

christianYoopies · 2022-09-12T12:18:45Z

Same issue , started around Friday 9th Sept too. Fastlane version 2.209.1

readyruncode · 2022-09-12T12:29:53Z

When we try to submit for review the build gets labeled Invalid Binary.

@readyruncode you mean if you try to submit manually?

I was able to submit the build for BETA review manually without any problem (it disrupts our CI pipeline, but well ... 😅)

@mathaeus Yes, I mean when trying to manually submit the build to beta review using the TestFlight interface on App Store Connect, the status changed to Invalid Binary.

Edit: I should maybe mention that our troubles also started on Friday, September 9.

vveidi · 2022-09-12T12:54:13Z

Same here. Started last Friday approximately. I'm able to deliver new app version to App Store, however, the CI always fails because the processing lasts too long

ivan-gaydamakin · 2022-09-12T13:49:29Z

fastlane 2.209.0.
Still "waiting" on during 6 hours... Today I saw that problem.
On appstore connect, have "Ready to submit" state

DimaElka · 2022-09-12T15:08:41Z

same issue

SSheldon · 2022-09-12T16:13:14Z

This looks like the same issue that @bassrock opened #20644 to fix. It sounds like that change solved the issue for him.

saudeon · 2022-09-12T18:54:25Z

Confirming we are also seeing this issue with the latest version of Fastlane.
My assumption is that perhaps Apple changed a response?

benjamin-es-hall · 2022-09-12T19:09:31Z

Confirming also seeing same issue with fastlane 2.209.1. Not sure when started but I've just about gone crazy trying to distribute to internal groups with no luck today (same as reported by @bassrock in #20644 )

LizCira · 2022-09-12T20:17:19Z

Also seeing the same using Bamboo and yes, for us it started sometime over the weekend. I have been able to locally reproduce the behavior running Fastlane from my terminal

mathaeus · 2022-09-12T20:46:22Z

Thanks @SSheldon for linking that other issue. The issues do indeed sound like they could have the same cause.

@joshdholtz since quite a few people are affected, may I drag you into this to get your expert opinion on this?

minhthong095 · 2022-09-13T04:45:35Z

Same here :(

Here is the script

fastlane run pilot \
	api_key_path: \
	distribute_external: true \
	distribute_only:true \
	notify_external_testers: true \
	groups: \
	changelog:$CHANGELOG \
	app_platform:ios \
	app_identifier:com.xx.xx

3a4oT · 2022-09-13T06:31:42Z

The same issue is on our side, is there any workaround?

crazymanish · 2022-09-13T07:03:19Z

Dear all,

@bassrock PR #20644 looks like fix this issue.
Adding steps to switch your Fastlane version to this PR

Switch your Fastlane version to PR

Update Gemfile to 👇 and run bundle install, bundle update fastlane, or bundle update

gem "fastlane", :git => "https://github.com/bassrock/fastlane.git", :branch => "beta-build"

Please let us know if this PR fixed the issue for you or not and provide us your valuable feedback!

Ibni-Amin · 2022-09-13T07:15:43Z

On 9th September 2022 I faced the same issue on testFlight through pipeline, before it was fine and work smoothly, when
I run pipeline, maximum within an hour build would available on testFlight.

Baza207 · 2022-09-13T07:27:22Z

Dear all,

@bassrock PR #20644 looks like fix this issue. Adding steps to switch your Fastlane version to this PR

Switch your Fastlane version to PR

Update Gemfile to 👇 and run bundle install, bundle update fastlane, or bundle update
gem "fastlane", :git => "https://github.com/bassrock/fastlane.git", :branch => "beta-build"
Please let us know if this PR fixed the issue for you or not and provide us your valuable feedback!

If this works that's great, but I have 2 questions:

Does anyone know how we'd use this version when using Azure DevOps tasks when use Fastlane where we don't use Fastlane directly? The fastlane version seems to only work with versions, not branches like this. 😞
Would there be any timeline for a new version being released (assuming this fix works)?

DimaElka · 2022-09-13T07:35:14Z

Dear all,

@bassrock PR #20644 looks like fix this issue. Adding steps to switch your Fastlane version to this PR

Switch your Fastlane version to PR

Update Gemfile to 👇 and run bundle install, bundle update fastlane, or bundle update
gem "fastlane", :git => "https://github.com/bassrock/fastlane.git", :branch => "beta-build"
Please let us know if this PR fixed the issue for you or not and provide us your valuable feedback!

after raising version to beta I have "No build_beta_detail included" error with the same upload step that worked previously

Julien-Roger · 2022-09-14T08:03:21Z

🩹 Workarounds

To prevent this crash you could set skip_waiting_for_build_processing: true to prevent the crash from happening while waiting for the build to process

Sleep the process for like an hour or just wait an hour 😛

After some amount of time you could run upload_to_testflight again with distribute_only: true

Problem is that method upload_to_testflight will never exit or timeout. Need to add some watchdog to kill the process after 1 hour.

…sion (bassrock, beta-build) gem "fastlane", :git => "https://github.com/bassrock/fastlane.git", :branch => "beta-build" For more info, please refer to fastlane/fastlane#20645.

joshdholtz · 2022-09-14T15:21:52Z

✨Official Update

Got confirmation that the App Store Connect API team is looking into the issue ✅

Will report back when I hear more and/or when it's fixed!

joshdholtz · 2022-09-14T23:51:21Z

✨Official Update

App Store Connect API is back to working for https://api.appstoreconnect.apple.com/v1/builds 💯 🥳

If you are able to test and report back that its working for you too that would be greatly appreciated 🙏

M1ZoN · 2022-09-15T00:19:37Z

I think now it is stuck on

Waiting for the build to show up in the build list - this may take a few minutes (check your email for processing issues if this continues)

Because previously it was doing this stage for 4-6 min, and now it is doing it for 12 min, and I am hypothesising that it will be stuck now :(

Will update shortly, because script has timeout in one hour.

P.S: tried without workaround: @bassrock PR #20644 looks like fix this issue.

nickeliason-motive · 2022-09-15T01:02:21Z

The API is working for us again - no workarounds or changes since prior to the issue 🙌

joshdholtz · 2022-09-15T01:29:20Z

@M1ZoN Huh... it really should have worked for you! Are you using App Store Connect API key, right? Did you check your email for any processing issues?

I don't know if that PR will get merged because the waiting for processing is needed for all the proper external distribution stuff to take place 🤔

bassrock · 2022-09-15T01:49:03Z

I can confirm my upload action is working again without #20644 ! 🥳 thanks @joshdholtz

Baza207 · 2022-09-15T06:10:56Z

Working again for us as well via Azure DevOps Apple App Store task. 🥳 Thanks @joshdholtz and everyone for the help. 😄

elitalon · 2022-09-15T08:16:25Z

It's working for us as well. Thanks a lot!

DimaElka · 2022-09-15T08:19:53Z

It's working again! Thanks!

bonnyfone · 2022-09-15T08:20:57Z

Working for me again without any change.

ghost · 2022-09-15T08:22:47Z

@joshdholtz saves us once again. You're a goddamn legend, man! ❤️

MrChrisBarker · 2022-09-15T08:41:20Z

Working for me too, including distribution to Groups!!
Thanks everyone & @joshdholtz !!!!

MatthewTHFisher · 2022-09-15T08:52:25Z

Amazing work!!! Thanks @joshdholtz and everyone 🎉

mathaeus · 2022-09-15T10:18:08Z

I haven't had the chance yet to test it, but since it's working for everyone now I'll close the issue.

Great work everyone ❤️

readyruncode · 2022-09-15T10:46:41Z

Confirming it's working again for our team as well! No changes needed.

Thanks @joshdholtz and everyone else who helped resolve this! ❤️

trongnhan68 · 2022-11-09T15:02:20Z

Hello, I still got an issue like this. Can any have an idea?
ENV:

Xcode 13.4.1
Fastlane 2.210.0

mathaeus changed the title ~~upload_to_testflight` action hangs with "Waiting for App Store Connect to finish processing the new build ..."~~ upload_to_testflight action hangs with "Waiting for App Store Connect to finish processing the new build ..." Sep 12, 2022

crazymanish added the tool: pilot label Sep 13, 2022

dinsen mentioned this issue Sep 14, 2022

Pilot: `ready_for_beta_submission?': No build_beta_detail included (RuntimeError) #20654

Closed

4 tasks

rajeshbeats mentioned this issue Sep 14, 2022

force set wait_for_build_beta_detail_processing to false rajeshbeats/fastlane#1

Closed

jeheon92 mentioned this issue Sep 14, 2022

Authentication credentials are missing or invalid. #20270

Closed

defagos mentioned this issue Sep 14, 2022

Handle buffering SRGSSR/pillarbox-apple#74

Merged

5 tasks

narin mentioned this issue Sep 14, 2022

Automation - Increase the ios build worker timeout department-of-veterans-affairs/va-mobile-app#3763

Closed

9 tasks

bassrock mentioned this issue Sep 15, 2022

fix(beta): updating to use previous wait behavior for ios #20644

Closed

5 tasks

crazymanish added status: released topic: ConnectAPI labels Sep 15, 2022

mathaeus closed this as completed Sep 15, 2022

emiliopavia mentioned this issue Sep 15, 2022

[IAI-242] Update fastlane to fix a timeout when uploading builds to App Store Connect pagopa/io-app#4117

Merged

narin mentioned this issue Sep 15, 2022

Increase timeout for iOS fastlane step department-of-veterans-affairs/va-mobile-app#3800

Merged

6 tasks

upload_to_testflight action hangs with "Waiting for App Store Connect to finish processing the new build ..." #20645

upload_to_testflight action hangs with "Waiting for App Store Connect to finish processing the new build ..." #20645

Comments

mathaeus commented Sep 12, 2022 • edited

New Issue Checklist

Issue Description

Command executed

Complete output when running fastlane, including the stack trace and command used

fastlane-bot commented Sep 12, 2022

hendych commented Sep 12, 2022

Baza207 commented Sep 12, 2022

baderbenzribia commented Sep 12, 2022

mathaeus commented Sep 12, 2022

mathaeus commented Sep 12, 2022

unixvb commented Sep 12, 2022

readyruncode commented Sep 12, 2022

Baza207 commented Sep 12, 2022

mathaeus commented Sep 12, 2022

mathaeus commented Sep 12, 2022

Baza207 commented Sep 12, 2022

christianYoopies commented Sep 12, 2022 • edited

readyruncode commented Sep 12, 2022 • edited

vveidi commented Sep 12, 2022

ivan-gaydamakin commented Sep 12, 2022 • edited

DimaElka commented Sep 12, 2022

SSheldon commented Sep 12, 2022

saudeon commented Sep 12, 2022

benjamin-es-hall commented Sep 12, 2022

LizCira commented Sep 12, 2022

mathaeus commented Sep 12, 2022

minhthong095 commented Sep 13, 2022 • edited

3a4oT commented Sep 13, 2022

crazymanish commented Sep 13, 2022

Switch your Fastlane version to PR

Ibni-Amin commented Sep 13, 2022

Baza207 commented Sep 13, 2022 • edited

Switch your Fastlane version to PR

DimaElka commented Sep 13, 2022

Switch your Fastlane version to PR

Julien-Roger commented Sep 14, 2022

joshdholtz commented Sep 14, 2022

joshdholtz commented Sep 14, 2022 • edited

M1ZoN commented Sep 15, 2022 • edited

nickeliason-motive commented Sep 15, 2022

joshdholtz commented Sep 15, 2022

bassrock commented Sep 15, 2022 • edited

Baza207 commented Sep 15, 2022

elitalon commented Sep 15, 2022

DimaElka commented Sep 15, 2022

bonnyfone commented Sep 15, 2022

ghost commented Sep 15, 2022

MrChrisBarker commented Sep 15, 2022

MatthewTHFisher commented Sep 15, 2022

mathaeus commented Sep 15, 2022

readyruncode commented Sep 15, 2022

trongnhan68 commented Nov 9, 2022

`upload_to_testflight` action hangs with "Waiting for App Store Connect to finish processing the new build ..." #20645

`upload_to_testflight` action hangs with "Waiting for App Store Connect to finish processing the new build ..." #20645

mathaeus commented Sep 12, 2022 •

edited

christianYoopies commented Sep 12, 2022 •

edited

readyruncode commented Sep 12, 2022 •

edited

ivan-gaydamakin commented Sep 12, 2022 •

edited

minhthong095 commented Sep 13, 2022 •

edited

Baza207 commented Sep 13, 2022 •

edited

joshdholtz commented Sep 14, 2022 •

edited

M1ZoN commented Sep 15, 2022 •

edited

bassrock commented Sep 15, 2022 •

edited