[firebase_storage]: Out of memory when uploading large file using `putData` on Android

[Tom3652]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

Storage

Which platforms are affected?

Android, iOS

Description

When uploading a large file > 500MB, the app crashes because of Out of memory issue using putData.
I have tested on Android, not on iOS yet but i have seen an open issue on iOS #13532 so there is no need for me to create a duplicated one.

I have seen some closed issues but that don't seem to be resolved on my end unfortunately.
So this issue could reopen #13385

Reproducing the issue

1. Run the sample code
2. Upload a large video file > 500MB
3. See the crash log

import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_storage/firebase_storage.dart';
import 'package:flutter/material.dart';
import 'package:image_picker/image_picker.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  runApp(const TestApp());
}

class TestApp extends StatefulWidget {
  const TestApp({super.key});

  @override
  State<TestApp> createState() => _TestAppState();
}

class _TestAppState extends State<TestApp> {

  Future<void> testStorage() async {
    XFile? file = await ImagePicker().pickVideo(source: ImageSource.gallery);
    if (file != null) {
      FirebaseStorage.instance.ref().child("test").putData(await file.readAsBytes());
    }
  }

  @override
  void initState() {
    testStorage();
    super.initState();
  }

  @override
  Widget build(BuildContext context) {
    return const Placeholder();
  }
}

Firebase Core version

3.8.0

Flutter Version

3.24.5

Relevant Log Output

E/flutter (11883): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Out of Memory
E/flutter (11883): #0      new Uint8List (dart:typed_data-patch/typed_data_patch.dart:2307:3)
E/flutter (11883): #1      WriteBuffer._resize (package:flutter/src/foundation/serialization.dart:67:33)
E/flutter (11883): #2      WriteBuffer._add (package:flutter/src/foundation/serialization.dart:39:7)
E/flutter (11883): #3      WriteBuffer.putUint8 (package:flutter/src/foundation/serialization.dart:75:5)
E/flutter (11883): #4      _FirebaseStorageHostApiCodec.writeValue (package:firebase_storage_platform_interface/src/pigeon/messages.pigeon.dart:256:14)
E/flutter (11883): #5      StandardMessageCodec.writeValue (package:flutter/src/services/message_codecs.dart:454:9)
E/flutter (11883): #6      _FirebaseStorageHostApiCodec.writeValue (package:firebase_storage_platform_interface/src/pigeon/messages.pigeon.dart:265:13)
E/flutter (11883): #7      StandardMessageCodec.encodeMessage (package:flutter/src/services/message_codecs.dart:331:5)
E/flutter (11883): #8      BasicMessageChannel.send (package:flutter/src/services/platform_channel.dart:218:71)
E/flutter (11883): #9      FirebaseStorageHostApi.referencePutData (package:firebase_storage_platform_interface/src/pigeon/messages.pigeon.dart:604:52)
E/flutter (11883): #10     MethodChannelPutTask._getTask (package:firebase_storage_platform_interface/src/method_channel/method_channel_task.dart:323:55)
E/flutter (11883): #11     new MethodChannelPutTask (package:firebase_storage_platform_interface/src/method_channel/method_channel_task.dart:319:13)
E/flutter (11883): #12     MethodChannelReference.putData (package:firebase_storage_platform_interface/src/method_channel/method_channel_reference.dart:153:12)
E/flutter (11883): #13     Reference.putData (package:firebase_storage/src/reference.dart:112:44)
E/flutter (11883): #14     _TestAppState.testStorage (package:test_database/storage_test.dart:24:70)
E/flutter (11883): #15     _FutureListener.handleValue (dart:async/future_impl.dart:163:18)
E/flutter (11883): #16     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:861:45)
E/flutter (11883): #17     Future._propagateToListeners (dart:async/future_impl.dart:890:13)
E/flutter (11883): #18     Future._completeWithValue (dart:async/future_impl.dart:666:5)
E/flutter (11883): #19     Future._asyncCompleteWithValue.<anonymous closure> (dart:async/future_impl.dart:736:7)
E/flutter (11883): #20     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
E/flutter (11883): #21     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)

Flutter dependencies

Expand Flutter dependencies snippet

Dart SDK 3.5.4
Flutter SDK 3.24.5
test_database 1.0.0+1

dependencies:
- firebase_auth 5.3.3 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta]
- firebase_core 3.8.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_database 11.1.6 [firebase_core firebase_core_platform_interface firebase_database_platform_interface firebase_database_web flutter]
- firebase_storage 12.3.6 [firebase_core firebase_core_platform_interface firebase_storage_platform_interface firebase_storage_web flutter]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]
- image_picker 1.1.2 [flutter image_picker_android image_picker_for_web image_picker_ios image_picker_linux image_picker_macos image_picker_platform_interface image_picker_windows]

dev dependencies:
- flutter_lints 4.0.0 [lints]
- flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math leak_tracker_flutter_testing async boolean_selector characters collection leak_tracker leak_tracker_testing material_color_utilities meta source_span stream_channel string_scanner term_glyph vm_service]

transitive dependencies:
- _flutterfire_internals 1.3.46 [collection firebase_core firebase_core_platform_interface flutter meta]
- async 2.11.0 [collection meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- characters 1.3.0
- clock 1.1.1
- collection 1.18.0
- cross_file 0.3.4+2 [meta web]
- fake_async 1.3.1 [clock collection]
- file_selector_linux 0.9.3+1 [cross_file file_selector_platform_interface flutter]
- file_selector_macos 0.9.4+2 [cross_file file_selector_platform_interface flutter]
- file_selector_platform_interface 2.6.2 [cross_file flutter http plugin_platform_interface]
- file_selector_windows 0.9.3+3 [cross_file file_selector_platform_interface flutter]
- firebase_auth_platform_interface 7.4.9 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_auth_web 5.13.4 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser meta web]
- firebase_core_platform_interface 5.3.0 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.18.1 [firebase_core_platform_interface flutter flutter_web_plugins meta web]
- firebase_database_platform_interface 0.2.5+46 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_database_web 0.2.6+4 [collection firebase_core firebase_core_web firebase_database_platform_interface flutter flutter_web_plugins]
- firebase_storage_platform_interface 5.1.33 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_storage_web 3.10.5 [_flutterfire_internals async firebase_core firebase_core_web firebase_storage_platform_interface flutter flutter_web_plugins http meta web]
- flutter_plugin_android_lifecycle 2.0.23 [flutter]
- flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math]
- http 1.2.2 [async http_parser meta web]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- image_picker_android 0.8.12+17 [flutter flutter_plugin_android_lifecycle image_picker_platform_interface]
- image_picker_for_web 3.0.6 [flutter flutter_web_plugins image_picker_platform_interface mime web]
- image_picker_ios 0.8.12+1 [flutter image_picker_platform_interface]
- image_picker_linux 0.2.1+1 [file_selector_linux file_selector_platform_interface flutter image_picker_platform_interface]
- image_picker_macos 0.2.1+1 [file_selector_macos file_selector_platform_interface flutter image_picker_platform_interface]
- image_picker_platform_interface 2.10.0 [cross_file flutter http plugin_platform_interface]
- image_picker_windows 0.2.1+1 [file_selector_platform_interface file_selector_windows flutter image_picker_platform_interface]
- leak_tracker 10.0.5 [clock collection meta path vm_service]
- leak_tracker_flutter_testing 3.0.5 [flutter leak_tracker leak_tracker_testing matcher meta]
- leak_tracker_testing 3.0.1 [leak_tracker matcher meta]
- lints 4.0.0
- matcher 0.12.16+1 [async meta stack_trace term_glyph test_api]
- material_color_utilities 0.11.1 [collection]
- meta 1.15.0
- mime 2.0.0
- path 1.9.0
- plugin_platform_interface 2.1.8 [meta]
- sky_engine 0.0.99
- source_span 1.10.0 [collection path term_glyph]
- stack_trace 1.11.1 [path]
- stream_channel 2.1.2 [async]
- string_scanner 1.2.0 [source_span]
- term_glyph 1.2.1
- test_api 0.7.2 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph]
- typed_data 1.4.0 [collection]
- vector_math 2.1.4
- vm_service 14.2.5
- web 1.1.0

Additional context and comments

No response

[russellwheatley]

Hey @Tom3652 - the android and iOS issues you linked are different and have been addressed, they occur on the native side (e.g. Fatal Exception: java.lang.OutOfMemoryError:) which is not what is happening in your stack trace. It is happening on the Flutter side.

Have you tried to steam the data as opposed to writing to memory which is causing the problem (i.e. await file.readAsBytes())? Something like this:

XFile? file = await ImagePicker().pickImage(source: ImageSource.gallery);

    if (file != null) {
      File fileToUpload = File(file.path);

      Reference storageRef =
          FirebaseStorage.instance.ref().child('tests');

      UploadTask uploadTask = storageRef.putFile(fileToUpload);

      // Await the completion of the upload
      await uploadTask.whenComplete(() {
        print('Upload complete!');
      });
    }

[Tom3652]

Hey @russellwheatley - Thank you for the quick reply and you are absolutely right i feel guilty i have not thought about it earlier !
It's working very fine using putFile instead of trying to read the bytes in memory first on the dart side, so there is no issue at all now i have managed to upload a very large file without problem.

Thanks for your time !

[russellwheatley]

No worries, @Tom3652. Glad it worked for you. I'll close this out 🙏