crash in AWSMobileClient.getTokens

[paulz]

Describe the bug

A clear and concise description of what the bug is. Please be as detailed as possible and help us help you.

Crashlytics reported crash on iOS device

Stack Trace

Fatal Exception: NSInternalInconsistencyException
*** -[NSHashTable NSHashTable { [0] <AWSMobileClientXCF.FetchUserPoolTokensOperation: 0x103c09570> } ] count underflow

Fatal Exception: NSInternalInconsistencyException
0  CoreFoundation                 0x9e88 __exceptionPreprocess
1  libobjc.A.dylib                0x178d8 objc_exception_throw
2  Foundation                     0x555d60 -[NSConcreteHashTable rehash]
3  Foundation                     0xc680 hashProbe
4  Foundation                     0x129f4 -[NSConcreteHashTable addObject:]
5  AWSMobileClientXCF             0x3ae68 specialized AWSMobileClient.getTokens(_:)
6  AWSMobileClientXCF             0x20538 AWSMobileClient.logins() (<compiler-generated>)
7  AWSMobileClientXCF             0x20ce4 @objc AWSMobileClient.logins() (<compiler-generated>)
8  AWSCore                        0x3192c -[AWSCognitoCredentialsProviderHelper getIdentityId] + 301 (AWSIdentityProvider.m:301)
9  AWSCore                        0x31558 -[AWSCognitoCredentialsProviderHelper logins] + 274 (AWSIdentityProvider.m:274)
10 AWSCore                        0x55fb4 -[AWSCognitoCredentialsProvider credentialsWithCancellationToken:] + 630 (AWSCredentialsProvider.m:630)
11 AWSMobileClientXCF             0x252a8 closure #1 in AWSMobileClient.getAWSCredentials(_:) + 30 (AWSMobileClient+Credentials.swift:30)
12 AWSMobileClientXCF             0x1c0a4 @objc AWSAsyncBlockOperation.main() (<compiler-generated>)
13 AWSMobileClientXCF             0x489c0 @objc AWSAsyncOperation.start() (<compiler-generated>)
14 Foundation                     0x3c50c __NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__
15 Foundation                     0x41c20 __NSOQSchedule_f
16 libdispatch.dylib              0x13114 _dispatch_block_async_invoke2
17 libdispatch.dylib              0x3fdc _dispatch_client_callout
18 libdispatch.dylib              0x746c _dispatch_continuation_pop
19 libdispatch.dylib              0x6ad4 _dispatch_async_redirect_invoke
20 libdispatch.dylib              0x15a6c _dispatch_root_queue_drain
21 libdispatch.dylib              0x16284 _dispatch_worker_thread2
22 libsystem_pthread.dylib        0xdbc _pthread_wqthread
23 libsystem_pthread.dylib        0xb98 start_wqthread

while another thread:

com.apple.root.default-qos
0  libsystem_kernel.dylib         0xb48 mach_msg2_trap + 8
1  libsystem_kernel.dylib         0x13008 mach_msg2_internal + 80
2  libsystem_kernel.dylib         0x13248 mach_msg_overwrite + 388
3  libsystem_kernel.dylib         0x108c mach_msg + 24
4  libdispatch.dylib              0x1f364 _dispatch_mach_send_and_wait_for_reply + 540
5  libdispatch.dylib              0x1f6ec dispatch_mach_send_with_result_and_wait_for_reply + 60
6  libxpc.dylib                   0x102e4 xpc_connection_send_message_with_reply_sync + 240
7  Security                       0x79b4 securityd_message_with_reply_sync + 124
8  Security                       0x11aa4 securityd_send_sync_and_do + 88
9  Security                       0xb8598 cftype_to_bool_cftype_error_request + 160
10 Security                       0x1d4c __SecItemCopyMatching_block_invoke_2 + 200
11 Security                       0x4dc4 __SecItemAuthDoQuery_block_invoke + 532
12 Security                       0x3898 SecItemAuthDoQuery + 1236
13 Security                       0x4a94 __SecItemCopyMatching_block_invoke + 140
14 Security                       0xa99c SecOSStatusWith + 56
15 Security                       0x495c SecItemCopyMatching + 400
16 AWSCore                        0x459c0 -[AWSUICKeyChainStore dataForKey:error:] + 457 (AWSUICKeyChainStore.m:457)
17 AWSCore                        0x45550 -[AWSUICKeyChainStore stringForKey:error:] + 377 (AWSUICKeyChainStore.m:377)
18 AWSCognitoIdentityProvider     0x8c48 -[AWSCognitoIdentityUser getSession] + 235 (AWSCognitoIdentityUser.m:235)
19 AWSMobileClientXCF             0x164a8 protocol witness for CognitoIdentityUserBehavior.getUserPoolToken(completion:) in conformance AWSCognitoIdentityUser + 26 (CognitoIdentityUserBehavior.swift:26)
20 AWSMobileClientXCF             0x3628c FetchUserPoolTokensOperation.fetchToken() + 3688 (<compiler-generated>:3688)
21 AWSMobileClientXCF             0x36f80 partial apply for closure #1 in closure #1 in FetchUserPoolTokensOperation.acceptEvent(_:) + 88 (<compiler-generated>:88)
22 AWSMobileClientXCF             0x1f0fc thunk for @escaping @callee_guaranteed () -> () + 6748 (<compiler-generated>:6748)
23 libdispatch.dylib              0x24b4 _dispatch_call_block_and_release + 32
24 libdispatch.dylib              0x3fdc _dispatch_client_callout + 20
25 libdispatch.dylib              0x70c8 _dispatch_queue_override_invoke + 788
26 libdispatch.dylib              0x15a6c _dispatch_root_queue_drain + 396
27 libdispatch.dylib              0x16284 _dispatch_worker_thread2 + 164
28 libsystem_pthread.dylib        0xdbc _pthread_wqthread + 228
29 libsystem_pthread.dylib        0xb98 start_wqthread + 8

Code Snippet
Please provide a snippet of the code causing the issue or how you are using the service that has the bug.

Unique Configuration
Cognito with user pools standard configuration:

AWSMobileClient(
      configuration: [
        "Version": "1.0",
        "CredentialsProvider": [
          "CognitoIdentity": [
            "Default": [
              "PoolId": <REDACTED>,
              "Region": "us-east-1"
            ]
          ]
        ],
        "IdentityManager": [
          "Default": [:]
        ],
        "CognitoUserPool": [
          "Default": [
            "PoolId": <REDACTED>,
            "AppClientId": <REDACTED>,
            "Region": "us-east-1"
          ]
        ]
      ]
    )

where real id have been replaced with fake.

Areas of the SDK you are using (AWSMobileClient, Cognito, Pinpoint, IoT, etc)?

AWSMobileClient

Environment(please complete the following information):

• SDK Version: 2.27.12
• Dependency Manager: SPM
• Swift Version : 5.0
• Xcode Version: 14.2

Device Information (please complete the following information):
Model:iPad (10.9-inch, 10th generation)
Orientation: Landscape
RAM free: 329.19 MB
Disk free: 171.67 GB
Operating System
Version:16.1.1
Orientation: Landscape
Jailbroken:No

Additional context
App likely be refreshing token

[ameter]

@paulz Thanks for reporting this. Are you able to reproduce it locally?

We recommend upgrading to the most recent version of the aws-sdk-ios, version 2.29.1. We have released some bug fixes for ASWMobileClient in recent versions that may be related.

[paulz]

thank you, we did update SDK for the next release of the app. Do not even know how would we reproduce this locally. Is there a test that verifies safety of concurrent access to the NSHashTable used in AWSMobileClient.getTokens?

[ameter]

This may be related to #4346

[royjit]

A fix #4563 to make the usage of NSHashtable has been released in https://github.com/aws-amplify/aws-sdk-ios/releases/tag/2.30.3 . Could you please take a look and verify if that fixes your issue.