[firebase_storage]: OOM native when uploading a large file when app is on relatively high memory usage

[Tom3652]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

Storage

Which platforms are affected?

Android, iOS

Description

I have previously created #16753 but the OOM was on the dart side.
However, i have managed to reproduce in my real app a crash that i have got 2 days ago in my Crashlytics dashboard.

This is the native OOM stack trace that some of my live users are experiencing currently.
Of course they have the version using the putData instead of putFile but it's not a dart OOM...

-> What i believe is happening is that for example the app has 300MB RAM available for it to run. Let's say my app is consuming 250MB to run with everything inside / UI / Code etc...
That means there is a 50MB left free memory to use, but if i upload a 200MB file at this moment, it's like firebase_storage is reducing the 200MB by chunks to perform the upload, but the initial chunk is anyway > 50MB which cause the crash.

For example in #13460 and #13385 it is probably fixed that firebase_storage itself handles perfectly the large files now, but i am not sure it takes in consideration the total available free memory before starting the upload, otherwise i would not have the above stack trace.

Note : I have noticed it on Android because my phone has a low RAM available compared to my iPhone, but the same behavior may exist on iOS as well simply at a higher degree, but it is easier to reproduce it on Android than iOS.

Reproducing the issue

I have not managed yet to produce a sample code, but here are the steps :

1. Have an app that runs with a relatively high memory usage around (200MB) (by displaying few very large images / videos for example)
2. Try to upload a very large file
3. See the crash

I am working on the reproductible example, i save the issue to not lose it but i will update with the reproductible sample code.

Firebase Core version

3.8.0

Flutter Version

3.24.5

Relevant Log Output

Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 255746824 byte allocation with 25165824 free bytes and 127MB until OOM, target footprint 160196792, growth limit 268435456
       at io.flutter.plugin.common.StandardMessageCodec.readBytes(StandardMessageCodec.java:320)
       at io.flutter.plugin.common.StandardMessageCodec.readValueOfType(StandardMessageCodec.java:386)
       at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApiCodec.readValueOfType(GeneratedAndroidFirebaseStorage.java:692)
       at io.flutter.plugin.common.StandardMessageCodec.readValue(StandardMessageCodec.java:340)
       at io.flutter.plugin.common.StandardMessageCodec.readValueOfType(StandardMessageCodec.java:424)
       at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApiCodec.readValueOfType(GeneratedAndroidFirebaseStorage.java:692)
       at io.flutter.plugin.common.StandardMessageCodec.readValue(StandardMessageCodec.java:340)
       at io.flutter.plugin.common.StandardMessageCodec.decodeMessage(StandardMessageCodec.java:89)
       at io.flutter.plugin.common.BasicMessageChannel$IncomingMessageHandler.onMessage(BasicMessageChannel.java:262)
       at io.flutter.embedding.engine.dart.DartMessenger.invokeHandler(DartMessenger.java:292)
       at io.flutter.embedding.engine.dart.DartMessenger.lambda$dispatchMessageToQueue$0(DartMessenger.java:319)
       at android.os.Handler.handleCallback(Handler.java:958)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loopOnce(Looper.java:224)
       at android.os.Looper.loop(Looper.java:318)
       at android.app.ActivityThread.main(ActivityThread.java:8780)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:561)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1013)

Flutter dependencies

Expand Flutter dependencies snippet

Dart SDK 3.5.4
Flutter SDK 3.24.5
test_database 1.0.0+1

dependencies:
- firebase_core 3.8.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_storage 12.3.6 [firebase_core firebase_core_platform_interface firebase_storage_platform_interface firebase_storage_web flutter]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]
- image_picker 1.1.2 [flutter image_picker_android image_picker_for_web image_picker_ios image_picker_linux image_picker_macos image_picker_platform_interface image_picker_windows]
- video_player 2.9.2 [flutter html video_player_android video_player_avfoundation video_player_platform_interface video_player_web]

dev dependencies:
- flutter_lints 4.0.0 [lints]
- flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math leak_tracker_flutter_testing async boolean_selector characters collection leak_tracker leak_tracker_testing material_color_utilities meta source_span stream_channel string_scanner term_glyph vm_service]

transitive dependencies:
- _flutterfire_internals 1.3.46 [collection firebase_core firebase_core_platform_interface flutter meta]
- async 2.11.0 [collection meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- characters 1.3.0
- clock 1.1.1
- collection 1.18.0
- cross_file 0.3.4+2 [meta web]
- csslib 1.0.2 [source_span]
- fake_async 1.3.1 [clock collection]
- file_selector_linux 0.9.3+2 [cross_file file_selector_platform_interface flutter]
- file_selector_macos 0.9.4+2 [cross_file file_selector_platform_interface flutter]
- file_selector_platform_interface 2.6.2 [cross_file flutter http plugin_platform_interface]
- file_selector_windows 0.9.3+3 [cross_file file_selector_platform_interface flutter]
- firebase_core_platform_interface 5.3.0 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.18.1 [firebase_core_platform_interface flutter flutter_web_plugins meta web]
- firebase_storage_platform_interface 5.1.33 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_storage_web 3.10.5 [_flutterfire_internals async firebase_core firebase_core_web firebase_storage_platform_interface flutter flutter_web_plugins http meta web]
- flutter_plugin_android_lifecycle 2.0.23 [flutter]
- flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math]
- html 0.15.5 [csslib source_span]
- http 1.2.2 [async http_parser meta web]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- image_picker_android 0.8.12+18 [flutter flutter_plugin_android_lifecycle image_picker_platform_interface]
- image_picker_for_web 3.0.6 [flutter flutter_web_plugins image_picker_platform_interface mime web]
- image_picker_ios 0.8.12+1 [flutter image_picker_platform_interface]
- image_picker_linux 0.2.1+1 [file_selector_linux file_selector_platform_interface flutter image_picker_platform_interface]
- image_picker_macos 0.2.1+1 [file_selector_macos file_selector_platform_interface flutter image_picker_platform_interface]
- image_picker_platform_interface 2.10.0 [cross_file flutter http plugin_platform_interface]
- image_picker_windows 0.2.1+1 [file_selector_platform_interface file_selector_windows flutter image_picker_platform_interface]
- leak_tracker 10.0.5 [clock collection meta path vm_service]
- leak_tracker_flutter_testing 3.0.5 [flutter leak_tracker leak_tracker_testing matcher meta]
- leak_tracker_testing 3.0.1 [leak_tracker matcher meta]
- lints 4.0.0
- matcher 0.12.16+1 [async meta stack_trace term_glyph test_api]
- material_color_utilities 0.11.1 [collection]
- meta 1.15.0
- mime 2.0.0
- path 1.9.0
- plugin_platform_interface 2.1.8 [meta]
- sky_engine 0.0.99
- source_span 1.10.0 [collection path term_glyph]
- stack_trace 1.11.1 [path]
- stream_channel 2.1.2 [async]
- string_scanner 1.2.0 [source_span]
- term_glyph 1.2.1
- test_api 0.7.2 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph]
- typed_data 1.4.0 [collection]
- vector_math 2.1.4
- video_player_android 2.7.16 [flutter video_player_platform_interface]
- video_player_avfoundation 2.6.3 [flutter video_player_platform_interface]
- video_player_platform_interface 6.2.3 [flutter plugin_platform_interface]
- video_player_web 2.3.3 [flutter flutter_web_plugins video_player_platform_interface web]
- vm_service 14.2.5
- web 1.1.0

Additional context and comments

No response

[Tom3652]

Hi @SelaseKay !
I have managed to make a reproductible code sample :

import 'dart:io';

import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_storage/firebase_storage.dart';
import 'package:flutter/material.dart';
import 'package:image_picker/image_picker.dart';
import 'package:video_player/video_player.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  runApp(const TestApp());
}

class TestApp extends StatefulWidget {
  const TestApp({super.key});

  @override
  State<TestApp> createState() => _TestAppState();
}

class _TestAppState extends State<TestApp> {
  Future<void> testStorage() async {
    XFile? file = await ImagePicker().pickVideo(source: ImageSource.gallery);
    if (file != null) {
      UploadTask task =
          FirebaseStorage.instance.ref().child("test").putFile(File(file.path));
      task.snapshotEvents.listen((event) {
        print(event.state);
      });
    }
  }

  final List<VideoPlayerController> _controllers = [];

  @override
  void initState() {
    for (int i = 0; i <= 7; i++) {
      _controllers.add(VideoPlayerController.asset("assets/Butterfly-209.mp4"));
    }
    for (var c in _controllers) {
      c.initialize().then((_) {
        c.setLooping(true);
        c.play();
        c.setVolume(0);
      });
    }
    super.initState();
  }

  @override
  void dispose() {
    for (var c in _controllers) {
      c.dispose();
    }
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        floatingActionButton: FloatingActionButton(onPressed: () {
          testStorage();
        }),
        body: ListView(
          scrollDirection: Axis.vertical,
            children: List.generate(_controllers.length, (index) {
          return SizedBox(height:100, child: VideoPlayer(_controllers[index]));
        })),
      ),
    );
  }
}

Steps :

1. Run the sample code
2. Download the flutter video_player ButterFly.mp4 file from the official package (it's the one i have been using) and put it as asset in the test project where you run the sample code
3. Adjust the number of video player according to your phone's threshold memory (until it doesn't crash at app start because of OOM), in my sample code it's 8 because at 9 the app is crashing with around 500MB of RAM used
4. Click on the floating button and add a large file (to see the crash)
5. See the stack trace :

I/e.test_databas(16898): Alloc young concurrent copying GC freed 17(1568B) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 99us total 9.891ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 54.664ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 9.764ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Clamp target GC heap from 215MB to 192MB
I/e.test_databas(16898): Alloc concurrent copying GC freed 42(1784B) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 95us total 38.029ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 48.148ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): WaitForGcToComplete blocked Background on HeapTrim for 48.332ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 88.656ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 135.320ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Clamp target GC heap from 215MB to 192MB
I/e.test_databas(16898): Alloc concurrent copying GC freed 45(2168B) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 112us total 37.440ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 75.406ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): Forcing collection of SoftReferences for 9256B allocation
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Clamp target GC heap from 215MB to 192MB
I/e.test_databas(16898): Alloc concurrent copying GC freed 44(3008B) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 122us total 43.361ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 119.926ms
I/e.test_databas(16898): Starting a blocking GC Alloc
W/e.test_databas(16898): Throwing OutOfMemoryError "Failed to allocate a 9256 byte allocation with 3056 free bytes and 3056B until OOM, target footprint 201326592, growth limit 201326592" (VmSize 20925380 kB)
W/StorageTask(16898): unable to change internal state to: INTERNAL_STATE_CANCELED isUser: false from state:INTERNAL_STATE_IN_PROGRESS
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Waiting for a blocking GC Alloc
I/e.test_databas(16898): Clamp target GC heap from 209MB to 192MB
I/e.test_databas(16898): Alloc concurrent copying GC freed 1996(7027KB) AllocSpace objects, 0(0B) LOS objects, 3% free, 185MB/192MB, paused 96us total 48.968ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 130.597ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): WaitForGcToComplete blocked Background on HeapTrim for 132.202ms
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 132.775ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 88.685ms
I/e.test_databas(16898): Starting a blocking GC Alloc
I/e.test_databas(16898): WaitForGcToComplete blocked Alloc on HeapTrim for 36.005ms
I/e.test_databas(16898): Starting a blocking GC Alloc
E/StorageException(16898): StorageException has occurred.
E/StorageException(16898): An unknown error occurred, please check the HTTP result code and inner exception for server response.
E/StorageException(16898):  Code: -13000 HttpResult: 0
E/AndroidRuntime(16898): FATAL EXCEPTION: Firebase Blocking Thread #1
E/AndroidRuntime(16898): Process: com.example.test_database, PID: 16898
E/AndroidRuntime(16898): java.lang.OutOfMemoryError: Failed to allocate a 9256 byte allocation with 3056 free bytes and 3056B until OOM, target footprint 201326592, growth limit 201326592
E/AndroidRuntime(16898): 	at java.util.Arrays.copyOf(Arrays.java:3257)
E/AndroidRuntime(16898): 	at com.android.internal.util.LineBreakBufferedWriter.ensureCapacity(LineBreakBufferedWriter.java:266)
E/AndroidRuntime(16898): 	at com.android.internal.util.LineBreakBufferedWriter.appendToBuffer(LineBreakBufferedWriter.java:249)
E/AndroidRuntime(16898): 	at com.android.internal.util.LineBreakBufferedWriter.write(LineBreakBufferedWriter.java:215)
E/AndroidRuntime(16898): 	at java.io.PrintWriter.write(PrintWriter.java:473)
E/AndroidRuntime(16898): 	at java.io.PrintWriter.print(PrintWriter.java:603)
E/AndroidRuntime(16898): 	at java.io.PrintWriter.println(PrintWriter.java:756)
E/AndroidRuntime(16898): 	at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:778)
E/AndroidRuntime(16898): 	at java.lang.Throwable.printStackTrace(Throwable.java:670)
E/AndroidRuntime(16898): 	at java.lang.Throwable.printStackTrace(Throwable.java:735)
E/AndroidRuntime(16898): 	at android.util.Log.printlns(Log.java:459)
E/AndroidRuntime(16898): 	at android.util.Log.w(Log.java:211)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.network.NetworkRequest.performRequestStart(NetworkRequest.java:248)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.network.NetworkRequest.performRequest(NetworkRequest.java:263)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.network.NetworkRequest.performRequest(NetworkRequest.java:282)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.UploadTask.send(UploadTask.java:519)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.UploadTask.delaySend(UploadTask.java:452)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.UploadTask.uploadChunk(UploadTask.java:478)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.UploadTask.run(UploadTask.java:248)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.StorageTask.lambda$getRunnable$7$com-google-firebase-storage-StorageTask(StorageTask.java:1078)
E/AndroidRuntime(16898): 	at com.google.firebase.storage.StorageTask$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
E/AndroidRuntime(16898): 	at com.google.firebase.concurrent.LimitedConcurrencyExecutor.lambda$decorate$0$com-google-firebase-concurrent-LimitedConcurrencyExecutor(LimitedConcurrencyExecutor.java:65)
E/AndroidRuntime(16898): 	at com.google.firebase.concurrent.LimitedConcurrencyExecutor$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
E/AndroidRuntime(16898): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/AndroidRuntime(16898): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/AndroidRuntime(16898): 	at com.google.firebase.concurrent.CustomThreadFactory.lambda$newThread$0$com-google-firebase-concurrent-CustomThreadFactory(CustomThreadFactory.java:47)
E/AndroidRuntime(16898): 	at com.google.firebase.concurrent.CustomThreadFactory$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
E/AndroidRuntime(16898): 	at java.lang.Thread.run(Thread.java:923)
E/StorageException(16898): StorageException has occurred.
E/StorageException(16898): An unknown error occurred, please check the HTTP result code and inner exception for server response.
E/StorageException(16898):  Code: -13000 HttpResult: 0
I/Process (16898): Sending signal. PID: 16898 SIG: 9

Note : i am using video_player because it's the fastest way IMO to get a constant high memory usage.
The StackTrace differs from the one i am getting in Crashlytics report from my live app but i believe the behavior is here 🙏

[SelaseKay]

Hi @tom365, thanks for reporting this and providing detailed insights. Firebase Storage attempts to upload large files in chunks to prevent memory overload. However, if there isn’t enough memory available to handle even the initial chunk, this becomes a limitation outside the scope of the plugin. To address this, I recommend structuring your app to account for available memory, perhaps by checking the memory before attempting to upload large files. That being said, I'll go ahead and close this for now. Feel free to reach out if you have anymore questions.