Investigate top 40% of test failures

[lawrence-forooghian]

This will be our first wave of work towards stabilising the test suite (#1279).

My data source is this point-in-time snapshot:

You are currently only viewing uploads belonging to the ably/ably-cocoa repo and from branches 1279-run-tests-in-loop, 1279-run-tests-in-loop-lots-of-times, main and which were uploaded before 2022-04-07T14:21:31.000Z and which were uploaded after 2022-03-03T00:27:24.182Z.

The "uploaded after" is to only include uploads from after we merged #1282.

I've picked 40% as a relatively arbitrary number. It gives us 12 test cases to investigate, compared to 21 for 50%. 12 seemed like a reasonable number to start with - gives us enough to split between developers without being overwhelming. Then after we've done these 12, maybe we'll discover some issues that will fix some of the long tail too.

You can find documentation with some suggestions about how to investigate a test failure in #1327.

Position	GitHub issue	Test case ID	Test class	Test case	Fixed in PR
1	#1321	84c9fbc5-8882-4010-ba99-3e44d1793397	RealtimeClientPresenceTests	test__015__Presence__subscribe__with_no_arguments_should_subscribe_a_listener_to_all_presence_messages()	#1321
2	#1331	1e5beffd-b21f-4281-85c9-b1b6ab02471d	RealtimeClientPresenceTests	test__078__Presence__enter__optional_data_can_be_included_when_entering_a_channel()	#1391
3	#1309	114e01a8-8971-4c3a-9cb7-0bb128e2749e	RealtimeClientPresenceTests	test__037__Presence__update__should_update_the_data_for_the_present_member_with_a_value()	#1391
4	#1332	4f837671-6233-41f1-94e8-01174d1da7b8	PushTests	test__008__LocalDevice__has_a_device_method_that_returns_a_LocalDevice()	#1392
5	#1333	ac44f09c-f521-4bfc-a3d7-58d4f7428c74	RealtimeClientPresenceTests	test__032__Presence__enter__entering_without_an_explicit_PresenceMessage_clientId_should_implicitly_use_the_clientId_of_the_current_connection()	#1391
6	#1334	b747a0db-dfdb-497b-ab81-bddc93dbaf5a	RealtimeClientConnectionTests	test__092__Connection__Host_Fallback__should_retry_hosts_in_random_order_after_checkin_if_an_internet_connection_is_available()	#1361

|7 | #1335 | d17ec17b-78b4-4e1d-a0d3-2dba60cc50f6 | RestClientStatsTests | test__002__RestClient__stats__result__should_match_hour_level_inbound_and_outbound_fixture_data__forwards_()||
|8 | #1336 | 181c0d92-7356-4faf-acd0-7cc9b78a0aa2 | AuthTests | test__148__JWT_and_realtime__when_token_expires_and_has_a_means_to_renew__reconnects_using_authCallback_and_obtains_a_new_token()||
|9 | #1337 | 752111fe-1e41-4187-bddf-62cbfbece196 | RealtimeClientConnectionTests | test__064__Connection__connection_failures_once_CONNECTED__if_a_Connection_transport_is_disconnected_unexpectedly_or_if_a_token_expires__then_the_Connection_manager_will_immediately_attempt_to_reconnect()|#1381|
|10 | #1338 | c2b81ea2-d06c-41a7-b217-36a144da77e5 | RealtimeClientTests | test__039__RealtimeClient__Auth_authorize_should_upgrade_the_connection_with_current_token__when_the_connection_is_in_the_DISCONNECTED_state_when_auth_authorize_is_called__after_obtaining_a_token_the_library_should_move_to_the_CONNECTING_state_and_initiate_a_connection_attempt_using_the_new_token()||
|11 | #1339 | f724070b-5530-4962-a4e0-20688f220bf1 | AuthTests | test__122__authorize__two_consecutive_authorizations__using_Realtime_and_connection_is_CONNECTING__should_call_each_Realtime_authorize_callback()||
|12 | #1340 | eaebbab2-8b9f-4204-9c48-17e3cd9c4ca6 | AuthTests | test__144__JWT_and_realtime__when_using_authUrl__when_token_expires__receives_a_40142_error_from_the_server()||

┆Issue is synchronized with this Jira Task by Unito

[lawrence-forooghian]

Since I've already started investigating a couple of the RealtimeClientPresenceTests failures, I'll assign the rest of those ones to myself, too. I suspect they may be related.