spaceship ITC client hardening (#150). #235

lacostej · 2016-02-07T10:34:42Z

The following set of patches allow to make ITC communication more resilient.

It:

handles better ITC timeouts (by extending the default timeout)
handles HTTP/SSL failures (by retrying them) those are probably openssl&client dependent
handles ITC 401 issues (by login again)
handles better temporary issue to save (by retrying again)

Several things are to be discussed, in particular, how to implement re-login in the case of spaceship client#login being used with 2 parameters (password is currently not remembered in that case)

Last run:

49 'Loops' for delivering 30 screenshots
71 retries
ITC temporary save error received: 'We're temporarily unable to save your changes. Please try again later.'. Retrying after 60 seconds (remaining: 1)...
ESC[37m[06:58:46]: ESC[0mFailed / completed after 9 hour(s) 38 min(s) 11 sec(s)
/Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:218:in `handle_itc_response': We're temporarily unable to save your changes. Please try again later. (Spaceship::TunesClient::ITunesConnectTemporaryError)
        from /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:361:in `block in update_app_version!'
        from /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:795:in `call'

It is resilient but slow.

Updated with unit tests.

Note that the issue mentioned here, is tentatively covered by df4b3b8. This is one solution, but I am unsure we want to solve it that way.

houndci-bot · 2016-02-07T10:34:53Z

lib/spaceship/client.rb

      raise ex # re-raise the exception
    end

    private

+    # this mostly for troubleshooting while harderning the client
+    def debug_count_recovered
+      @recoveriesCount = (@recoveriesCount || 0 ) + 1


Space inside parentheses detected.
Use snake_case for variable names.

KrauseFx · 2016-02-07T18:39:07Z

lib/spaceship/tunes/tunes_client.rb

+      return block.call
+    rescue Spaceship::TunesClient::ITunesConnectTemporaryError => ex
+      unless (tries -= 1).zero?
+        msg = "ITC temporary save error received: '#{ex.message}'. Retrying after 60 seconds (remaining: #{tries})..."


Do we have access to the tries variable here? If yes, we could start by waiting just like a few seconds, and then go up to 1-2 minutes

Yes we could. Right now I am aiming for resilience, then fine tuning. In the end I would love to move some of this into either a faraday middleware, or using a custom more declarative "retrier".

For example, out of the last 5 runs, they all ended up failing here. I increased it to 60 seconds and it still fails sometimes.

Here are the last 2, with a 60 second sleep in between retries:

ITC temporary save error received: 'We're temporarily unable to save your changes. Please try again later.'. Retrying after 60 seconds (remaining: 1)... ESC[37m[19:47:24]: ESC[0mFailed / completed after 5 hour(s) 16 min(s) 32 sec(s) /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:218:in `handle_itc_response': We're temporarily unable to save your changes. Please try again later. (Spaceship::TunesClient::ITunesConnectTemporaryError) from /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:361:in `block in update_app_version!'

ITC temporary save error received: 'We're temporarily unable to save your changes. Please try again later.'. Retrying after 60 seconds (remaining: 1)... ESC[37m[21:20:34]: ESC[0mFailed / completed after 1 hour(s) 33 min(s) 8 sec(s) /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:218:in `handle_itc_response': We're temporarily unable to save your changes. Please try again later. (Spaceship::TunesClient::ITunesConnectTemporaryError) from /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:361:in `block in update_app_version!' from /Users/lacostej/.rvm/gems/ruby-2.2.4/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:795:in `call'

In the one that lasted 5 h 30 min, I had

0 timeout / connection failed (they appear more seldomly now given the 300 seconds

20+ auth lost

6 unable to save (last one fatal)
connection timeout)

KrauseFx · 2016-02-07T18:47:00Z

This looks really promising, great job. Pinging @snatchev, as he's also a spaceship expert 👍

houndci-bot · 2016-02-12T11:30:46Z

spec/client_spec.rb

+
+describe Spaceship::Client do
+  class TestClient < Spaceship::Client
+


Extra empty line detected at class body beginning.

KrauseFx · 2016-02-12T17:39:30Z

TODO_client_hardening

+
+ideas
+* parametrize default timeout ?
+* make default timeout client specific ?


Before merging this needs to be moved out into separate GitHub issues. I can add a stability tag for you 👍

I mean the whole file

Sure I didn't intend to submit it as it.

KrauseFx · 2016-02-12T18:51:02Z

This looks great, thanks again for this PR, I added some comments 👍

houndci-bot · 2016-02-13T11:53:20Z

lib/spaceship/tunes/tunes_client.rb

+        msg = "ITC temporary save error received: '#{ex.message}'. Retrying after 60 seconds (remaining: #{tries})..."
+        puts msg
+        logger.warn msg
+        sleep 60 unless defined?SpecHelper # unless FastlaneCore::Helper.is_test?


Space after keyword defined? is missing.

snatchev · 2016-02-16T15:09:11Z

@lacostej looks good to me.

KrauseFx · 2016-02-19T19:28:33Z

I just did the final review, this looks perfect from my side. Let's squash the commits and I'll ship a new version 🚀

* handles better ITC timeouts (by extending the default timeout to 5 min) * handles HTTP/SSL failures (by retrying them) those are probably openssl&client dependent * handles ITC 401 issues (by login again) * handles better temporary ITC issue to save ('try again later') (by ... retrying again) Also * cache some of the data to avoid querying the server too much Assisted by Krzysztof Daniszewski <krzysztof@daniszewski.it>

spaceship ITC client hardening (#150).

KrauseFx · 2016-02-20T15:46:49Z

Thanks again for this 👍

houndci-bot reviewed Feb 7, 2016
View reviewed changes

lacostej force-pushed the client_hardening branch from 5ffb76f to a61c1d0 Compare February 7, 2016 10:43

KrauseFx reviewed Feb 7, 2016
View reviewed changes

lacostej force-pushed the client_hardening branch from a61c1d0 to a579667 Compare February 11, 2016 10:25

houndci-bot reviewed Feb 12, 2016
View reviewed changes

spec/client_spec.rb

describe Spaceship::Client do

class TestClient < Spaceship::Client

Copy link

houndci-bot Feb 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra empty line detected at class body beginning.

lacostej force-pushed the client_hardening branch from 6823514 to df4b3b8 Compare February 12, 2016 11:54

KrauseFx reviewed Feb 12, 2016
View reviewed changes

This was referenced Feb 13, 2016

log_request loses information if parameter url_or_path passed using block to Faraday #240

Closed

log_request isn't performed for retried requests. #241

Closed

403 should/could be treated differently than 401 under tunes_client #242

Closed

houndci-bot reviewed Feb 13, 2016
View reviewed changes

lacostej mentioned this pull request Feb 13, 2016

Unauthorized access #105

Closed

theMeatloaf mentioned this pull request Feb 17, 2016

Waiting for iTunes Connect to process the new build... fastlane/fastlane#685

Closed

vpolouchkine assigned KrauseFx Feb 17, 2016

lacostej force-pushed the client_hardening branch from 4215ff3 to 0010193 Compare February 19, 2016 19:55

lacostej added a commit to lacostej/spaceship that referenced this pull request Feb 19, 2016

ITC Client hardening fastlane-old#235

0010193

lacostej force-pushed the client_hardening branch from 0010193 to cf588f5 Compare February 19, 2016 20:21

lacostej force-pushed the client_hardening branch from cf588f5 to 1104727 Compare February 19, 2016 20:23

KrauseFx added a commit that referenced this pull request Feb 20, 2016

Merge pull request #235 from lacostej/client_hardening

7a539e1

spaceship ITC client hardening (#150).

KrauseFx merged commit 7a539e1 into fastlane-old:master Feb 20, 2016

fastlane-bot mentioned this pull request Mar 7, 2016

Long processing times cause an "Unauthorized Access" error fastlane/fastlane#1557

Closed

This was referenced Mar 8, 2016

Improve error message when auth token expires while waiting for "iTunes Connect to process the new build" fastlane/fastlane#3247

Closed

Exception occurred when deploying new testflight build fastlane/fastlane#3249

Closed

fastlane-old locked and limited conversation to collaborators Mar 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spaceship ITC client hardening (#150). #235

spaceship ITC client hardening (#150). #235

lacostej commented Feb 7, 2016

houndci-bot Feb 7, 2016

KrauseFx Feb 7, 2016

lacostej Feb 7, 2016

KrauseFx commented Feb 7, 2016

houndci-bot Feb 12, 2016

KrauseFx Feb 12, 2016

KrauseFx Feb 12, 2016

lacostej Feb 13, 2016

KrauseFx commented Feb 12, 2016

houndci-bot Feb 13, 2016

snatchev commented Feb 16, 2016

KrauseFx commented Feb 19, 2016

KrauseFx commented Feb 20, 2016


		describe Spaceship::Client do
		class TestClient < Spaceship::Client

spaceship ITC client hardening (#150). #235

spaceship ITC client hardening (#150). #235

Conversation

lacostej commented Feb 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrauseFx commented Feb 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrauseFx commented Feb 12, 2016

Choose a reason for hiding this comment

snatchev commented Feb 16, 2016

KrauseFx commented Feb 19, 2016

KrauseFx commented Feb 20, 2016