[cloud_functions]: Getting DEADLINE_EXCEEDED but no trace of the call on the backend

[orestesgaolin]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

Cloud Functions

Which platforms are affected?

Android, iOS

Description

We are happy users of cloud_functions SDK but sometimes it happens that the library just stops executing calls to callable functions, even though:

• it was working just fine few minutes before
• Firebase Auth token is valid
• other users can access the function just fine, even from the same network

The error thrown is just DEADLINE_EXCEEDED after the specified time

I/flutter (15750): [CloudFunctionsClient] [firebase_functions/deadline-exceeded] DEADLINE_EXCEEDED
I/flutter (15750): [CloudFunctionsClient] 
I/flutter (15750): [CloudFunctionsClient] #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:648:7)
I/flutter (15750): [CloudFunctionsClient] #1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:334:18)
I/flutter (15750): [CloudFunctionsClient] <asynchronous suspension>
I/flutter (15750): [CloudFunctionsClient] #2      MethodChannelHttpsCallable.call (package:cloud_functions_platform_interface/src/method_channel/method_channel_https_callable.dart:22:24)
I/flutter (15750): [CloudFunctionsClient] <asynchronous suspension>
I/flutter (15750): [CloudFunctionsClient] #3      HttpsCallable.call (package:cloud_functions/src/https_callable.dart:49:37)
I/flutter (15750): [CloudFunctionsClient] <asynchronous suspension>
I/flutter (15750): [CloudFunctionsClient] #4      CloudFunctionsClient.appLaunch (package:cloud_functions_client/src/cloud_functions_client.dart:775:22)
I/flutter (15750): [CloudFunctionsClient] <asynchronous suspension>
...
I/flutter (15750): [CloudFunctionsClient] 
I/flutter (15750): [CloudFunctionsClient] #0   StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:648:7)
I/flutter (15750): [CloudFunctionsClient] #1   MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:334:18)
I/flutter (15750): [CloudFunctionsClient] #2   <asynchronous suspension>
I/flutter (15750): [CloudFunctionsClient] #3   MethodChannelHttpsCallable.call (package:cloud_functions_platform_interface/src/method_channel/method_channel_https_callable.dart:22:24)
I/flutter (15750): [CloudFunctionsClient] #4   <asynchronous suspension>
I/flutter (15750): [CloudFunctionsClient] #5   HttpsCallable.call (package:cloud_functions/src/https_callable.dart:49:37)
I/flutter (15750): [CloudFunctionsClient] #6   <asynchronous suspension>
I/flutter (15750): [CloudFunctionsClient] #7   CloudFunctionsClient.appLaunch (package:cloud_functions_client/src/cloud_functions_client.dart:775:22)
I/flutter (15750): [CloudFunctionsClient] ⛔ appLaunch failed

Reproducing the issue

It's quite tricky to reproduce the issue, it may happen during debugging, but when it occurs it's just difficult to capture any debug traces outside of the above exception.

My implementation typically is quite simple

      final user = firebaseAuth.currentUser;
      if (user == null) {
        throw Exception('User is not authenticated');
      }

      // check if token is valid
      final tokenResult = await user.getIdTokenResult();
      if (tokenResult.token == null) {
        throw Exception('User token is not valid');
      }

      final result = await functions
          .httpsCallable(
        'nameOfTheFunction',
        options: HttpsCallableOptions(timeout: const Duration(seconds: 30)),
      )
          .call<Map<String, dynamic>>(
        {
          'appBuild': appBuild,
          'userAgent': userAgent,
        },
      );

Firebase Core version

2.31.1

Flutter Version

3.24.4

Relevant Log Output

No response

Flutter dependencies

Expand Flutter dependencies snippet

Dart SDK 3.5.4
Flutter SDK 3.24.4
visible_app 24.11.3

dependencies:

- cloud_functions_client 1.0.0+1 [cloud_functions equatable firebase_auth flutter json_annotation log watcher]
- cloud_storage_client 1.0.0+1 [equatable file firebase_auth firebase_storage flutter log path rxdart]
- firebase_auth 4.19.6 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta]
- firebase_core 2.31.1 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_crashlytics 3.4.18 [firebase_core firebase_core_platform_interface firebase_crashlytics_platform_interface flutter stack_trace]
- firebase_messaging 14.9.0 [firebase_core firebase_core_platform_interface firebase_messaging_platform_interface firebase_messaging_web flutter meta]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]


transitive dependencies:

- cloud_functions 4.7.5 [cloud_functions_platform_interface cloud_functions_web firebase_core firebase_core_platform_interface flutter]
- cloud_functions_platform_interface 5.5.27 [firebase_core flutter meta plugin_platform_interface]
- cloud_functions_web 4.9.5 [cloud_functions_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins]
- firebase_auth_platform_interface 7.2.7 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_auth_web 5.11.6 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser meta web]
- firebase_core_platform_interface 5.0.0 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.17.0 [firebase_core_platform_interface flutter flutter_web_plugins meta web]
- firebase_crashlytics_platform_interface 3.6.25 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_dynamic_links 5.4.17 [firebase_core firebase_core_platform_interface firebase_dynamic_links_platform_interface flutter meta plugin_platform_interface]
- firebase_dynamic_links_platform_interface 0.2.6+25 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_messaging_platform_interface 4.5.33 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_messaging_web 3.8.3 [_flutterfire_internals firebase_core firebase_core_web firebase_messaging_platform_interface flutter flutter_web_plugins meta web]
- firebase_storage 11.7.6 [firebase_core firebase_core_platform_interface firebase_storage_platform_interface firebase_storage_web flutter]
- firebase_storage_platform_interface 5.1.21 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_storage_web 3.9.6 [_flutterfire_internals async firebase_core firebase_core_web firebase_storage_platform_interface flutter flutter_web_plugins http meta web]
- http_multi_server 3.2.1 [async]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- http_profile 0.1.0

Additional context and comments

I know that my firebase sdk is quite old compared to the current version, but maybe something comes to your mind that could potentially explain the issue? Do you recall any fixes between 2.31.1 and current version that could resolve it? Migrating to the current version would be a bit inconvenient for us right now.

[SelaseKay]

Hi @orestesgaolin, thanks for the report. I wasn't able to reproduce the issue on my end. It might be related to a timeout, so increasing the timeout could be a potential solution, though I'm not entirely sure. You might also find it helpful to check out this Reddit thread, as some of the suggested solutions there might address your issue.

[orestesgaolin]

The person in the reddit thread tries to explain it by their MacBook network issues, and they may be onto something. But in my case:

• I can see this issue appearing among our users, every day there will be few dozen who cannot access something via the cloud functions
• it affects both V1 and V2 callable functions
• it starts after a while, restarting the app typically solves it

So my gut feeling is that it may be related to some internal cache in the context of the application. Whether it's DNS or auth session I think it's common enough that probably some other people have it regularly. In the last 7 days I can see 164 unique users having this issue on Android and 134 on iOS.

[orestesgaolin]

I don't think I'm able to provide more repro steps but today it happened again. I can see all my colleagues are able to call the endpoint but from my Android device I see the same exception.

I'm using now 5.2.0 of the cloud_functions package as opposed to 4.7.5 last time.

The issue does not occur on iOS and another Samsung device on the same network.

The Firebase Auth token gets refreshed normally.

Restarting the app or network connection does not help here.

Restarting the phone didn't help too.

Issue occurs both in debug and release mode.

I also noticed following error message when trying to access DevTools

Log message:

A Dart VM Service on SM G991B is available at: http://127.0.0.1:49445/tMJnW2EYSmU=/

and clicking on the link leads to:

No DevTools instance is registered with the Dart Development Service (DDS).

[orestesgaolin]

What is quite annoying this issue appears in production to dozens of users as well so it's not only dev-ex issue

[russellwheatley]

Hey @orestesgaolin - a few thoughts on this.

We have tried to match the exceptions on iOS and web (good argument against trying to unify platform exceptions) which unfortunately might be obfuscating the true nature of the exception: https://github.com/firebase/flutterfire/blob/main/packages/cloud_functions/cloud_functions/android/src/main/java/io/flutter/plugins/firebase/functions/FlutterFirebaseFunctionsPlugin.java#L157-L167

So you might be getting "cancelled" or "timeout" but it is coming back as "deadline exceeded".

The next thing; what exactly are you doing in the function? Are you doing firestore writes for example? You could be hitting a quota: firebase/firebase-android-sdk#744 (comment)

Last thing; the nuclear option is to point your cloud_functions to a local src, and update the android code I've linked and use crashlytics to log the crash. At least you get the untampered native crash report which might be more illuminating to your issue.