Storage.put fail with large files on Android with react-native

[luisfuertes]

Storage.put need a blob to upload file.

But react-native has a fetch bug on android with "big" files. You can see here

When you try to uplaod video with size of 160mb or more, const response = await fetch(videoUri) get next error: TypeError: Network request failed(It isnt network error, with small sizes it work).

I have tried to generate the blob with https://github.com/joltup/rn-fetch-blob but I have not been able to make it works.

Is there a way to get it working or an alternative way to upload large files?

[sammartinez]

@luisfuertes are you still looking for support on this issue? If so, please provide us information on your environment by running the following command: npx envinfo --system --binaries --browsers --npmPackages --npmGlobalPackages. Also, if you can provide us a code snippet this will help us further assist you. Thanks ahead of time.

[walvekarnikhil]

I am also getting the same error, here is the requested information:
envinfo
System:
OS: macOS 10.15.6
CPU: (4) x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
Memory: 42.13 MB / 16.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 14.8.0 - /usr/local/bin/node
Yarn: 1.22.4 - /usr/local/bin/yarn
npm: 6.14.7 - /usr/local/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
Browsers:
Chrome: 85.0.4183.102
Firefox: 79.0
Firefox Developer Edition: 44.0a2
Safari: 14.0
npmPackages:
@apollo/client: ^3.0.0 => 3.1.3
@babel/core: ^7.10.3 => 7.11.4
@babel/runtime: ^7.10.5 => 7.11.2
@react-native-community/async-storage: ^1.11.0 => 1.12.0
@react-native-community/datetimepicker: ^2.6.0 => 2.6.2
@react-native-community/eslint-config: ^2.0.0 => 2.0.0
@react-native-community/masked-view: ^0.1.10 => 0.1.10
@react-native-community/netinfo: ^5.9.4 => 5.9.6
@react-native-community/toolbar-android: ^0.1.0-rc.2 => 0.1.0-rc.2
@react-native-firebase/app: ^8.4.3 => 8.4.3
@react-native-firebase/crashlytics: ^8.3.2 => 8.4.2
@react-navigation/bottom-tabs: ^5.7.3 => 5.8.0
@react-navigation/material-top-tabs: ^5.2.15 => 5.2.16
@react-navigation/native: ^5.0.9 => 5.7.3
@react-navigation/stack: ^5.1.1 => 5.9.0
@twotalltotems/react-native-otp-input: 1.3.7 => 1.3.7
amazon-cognito-identity-js: ^4.3.3 => 4.3.4
aws-amplify: ^3.0.19 => 3.0.24
aws-amplify-react-native: ^4.2.1 => 4.2.5
axios: ^0.19.2 => 0.19.2
babel-jest: ^26.1.0 => 26.3.0
babel-preset-react-native-stage-0: ^1.0.1 => 1.0.1
buffer: ^5.6.0 => 5.6.0
date-fns: ^2.15.0 => 2.16.1
enzyme: ^3.11.0 => 3.11.0
enzyme-adapter-react-16: ^1.15.2 => 1.15.3
eslint: ^7.3.1 => 7.7.0
graphql: ^15.3.0 => 15.3.0
graphql-tag: ^2.10.3 => 2.11.0
i18n-js: ^3.7.0 => 3.7.1
jest: ^26.1.0 => 26.4.2
lodash: ^4.17.15 => 4.17.20
md5: ^2.3.0 => 2.3.0
metro-react-native-babel-preset: ^0.59.0 => 0.59.0
moment: ^2.27.0 => 2.27.0
native-base: ^2.13.12 => 2.13.14
native-base-autocomplete: ^1.3.2 => 1.3.2
prop-types: ^15.7.2 => 15.7.2
react: 16.11.0 => 16.11.0
react-dom: ^16.13.1 => 16.13.1
react-native: 0.62.2 => 0.62.2
react-native-actionsheet: ^2.4.2 => 2.4.2
react-native-app-intro-slider: ^4.0.4 => 4.0.4
react-native-autocomplete-input: ^4.2.0 => 4.2.0
react-native-autocomplete-search: ^1.0.8 => 1.0.8
react-native-aws3: 0.0.9 => 0.0.9
react-native-check-box: ^2.1.7 => 2.1.7
react-native-fast-image: ^8.3.2 => 8.3.2
react-native-floating-action: ^1.21.0 => 1.21.0
react-native-gesture-handler: ^1.7.0 => 1.7.0
react-native-image-crop-picker: ^0.30 => 0.30.0
react-native-image-picker: ^2.3.3 => 2.3.3
react-native-keyboard-aware-scroll-view: ^0.9.1 => 0.9.2
react-native-keyboard-aware-scrollview: ^2.1.0 => 2.1.0
react-native-keyboard-spacer: ^0.4.1 => 0.4.1
react-native-loading-spinner-overlay: ^1.1.0 => 1.1.0
react-native-localize: ^1.4.1 => 1.4.1
react-native-media-controls: ^2.0.3 => 2.1.0
react-native-modal: ^11.5.6 => 11.5.6
react-native-modal-datetime-picker: ^8.7.1 => 8.9.3
react-native-picker-select: ^7.0.0 => 7.0.0
react-native-reanimated: ^1.10.1 => 1.13.0
react-native-safe-area-context: ^3.1.1 => 3.1.7
react-native-safe-area-view: ^1.1.1 => 1.1.1
react-native-screens: ^2.9.0 => 2.10.1
react-native-simple-radio-button: ^2.7.4 => 2.7.4
react-native-slider: ^0.11.0 => 0.11.0
react-native-splash-screen: ^3.2.0 => 3.2.0
react-native-sticky-parallax-header: ^0.3.0 => 0.3.1
react-native-svg: ^12.1.0 => 12.1.0
react-native-svg-charts: ^5.4.0 => 5.4.0
react-native-switch: ^2.0.0 => 2.0.0
react-native-tab-view: ^2.15.0 => 2.15.1
react-native-toast-message: ^1.3.2 => 1.3.3
react-native-vector-icons: ^7.0.0 => 7.0.0
react-native-video: ^4.4.5 => 4.4.5
react-native-video-player: ^0.10.1 => 0.10.1
react-native-video-processing: ^2.0.0 => 2.0.0
react-redux: ^7.2.0 => 7.2.1
react-test-renderer: 16.11.0 => 16.11.0
redux: ^4.0.5 => 4.0.5
redux-devtools-extension: ^2.13.8 => 2.13.8
redux-logger: ^3.0.6 => 3.0.6
redux-thunk: ^2.3.0 => 2.3.0
remote-redux-devtools: ^0.5.16 => 0.5.16
rn-fetch-blob: ^0.12.0 => 0.12.0
stream: 0.0.2 => 0.0.2
subscriptions-transport-ws: ^0.9.17 => 0.9.18
util: ^0.12.3 => 0.12.3
validator: ^13.1.1 => 13.1.1
npmGlobalPackages:
@angular/cli: 8.3.2
@aws-amplify/cli: 4.27.3
@dojo/cli: 0.5.0
@nestjs/cli: 6.12.9
@nrwl/cli: 8.7.0
appcenter-cli: 2.6.7
eslint: 6.0.1
expo-cli: 3.24.2
firebase-tools: 8.1.1
generator-code: 1.1.50
gh-pages: 2.0.1
http-server: 0.11.1
istanbul-combine: 0.3.0
mockserver: 3.1.0
nginx: 0.0.0
node: 12.9.1
npkill: 0.7.1
npm: 6.14.7
polymer-cli: 1.9.10
source-map-cli: 1.0.0
sql-cli: 0.6.2
ts-node: 8.6.2
typeorm: 0.2.22
uglify-js: 3.4.9
vsce: 1.62.0
webpack-bundle-analyzer: 3.3.2
yo: 2.0.6

Code Sample

uploadToStorage = async pathToVideoFile => {
  try {
    const response = await fetch(pathToVideoFile)

    const blob = await response.blob()

    Storage.put('yourKeyHere.mp4', blob, {
      contentType: 'video/mp4',
    })
  } catch (err) {
    console.log(err)
  }
}

Here in the above code

    const response = await fetch(pathToVideoFile)

fails with TypeError: Network request failed on Android

Try to upload a file more than 300mb. Logcat shows OutOfMemory exception.

[luisfuertes]

My workaround was upload with formdata and upload to bucket url generated by my backend.

    formData.append('file', {
      type: `video/${type}`,
      name: name,
      uri: videoUri,
    })

    const onProgress = percentCompleted => dispatch(videosActions.setFetching(true, translate('videoUpload.uploadVideoSucccessProgress', { percentCompleted })))
    const uploadVideoRes = await api.uploadVideo(postVideoUrl, formData, onProgress)

And upload video with axios:

export const uploadVideo = (url, formData, onProgress) => {
  return axios.post(url, formData, {
    headers: { 'Content-Type': 'multipart/form-data' },
    onUploadProgress: progressEvent => {
      const percentCompleted = Math.round((progressEvent.loaded * 100) / progressEvent.total)
      onProgress && onProgress(percentCompleted)
    },
  })
}

[stale]

This issue has been automatically closed because of inactivity. Please open a new issue if are still encountering problems.

[luisfuertes]

@sammartinez can you check this?

[CuongNguyenLD]

Hi, I'm facing the same issue, Storage.put not working with video file > 5mb using blob. Any update here?

[luisfuertes]

This part is the bug in react-native:

 const response = await fetch(pathToVideoFile)

[PatrykMilewski]

I'm having the same issue. Uploading for small files works, for file of size 5.7 MB it fails.

Using Amplify 4.19.0

[walvekarnikhil]

Here is my observation

const response = await fetch(pathToImageFile)

Fails with Out of memory exception, can be found using adb logcat
Storage.put excepts complete file to be read in the memory, there is no way to read it chunk by chunk.

[imsaurabhdixit]

I have the same issue. Did anybody solve this ?

[walvekarnikhil]

I have the same issue. Did anybody solve this?

Here is a small wrapper I created, using the Storage plugin. It supports reading files in chunks. https://github.com/walvekarnikhil/amplify-s3-chunk-upload

[jamesaucode]

Hey folks, I started doing some research on this issue and here's what I found.

The issue likely stems from here: https://github.com/aws-amplify/amplify-js/blob/main/packages/storage/src/providers/AWSS3ProviderManagedUpload.ts#L316
As a work around for using Axios on React Native (They don't support React native blobs). We implemented a change where we will transform the blob into an arrray bufer first. When the file is large enough it will hog all the memory which causes an OOM exception.

On top of that, during the transformation it freezes the UI.

Currently working on a fix, will give more updates soon.

[jamesaucode]

Update: I have a working implementation locally now that doesn't give the OOM error and should solve the UI freezing problem, its also significantly faster than before. Once I have properly tested it I will submit a PR.

[Jun711]

@jamesaucode
Would your fix work on Cordova apps too? we face the same issue on Cordova.

[majirosstefan]

@jamesaucode The problem is still present (although indirectly) when e.g. trying to upload 100MB+ media on Android. According to the docs here https://docs.amplify.aws/lib/storage/upload/q/platform/js/#react-native-uploads, we should still use fetch before uploading.

This part with fetch is, simply, broken in React Native. So Storage.put() can support uploading big files, chunks, etc,
but if it still needs to use fetch() that loads the whole file in memory, it does not help us.

Would be really nice, if Storage.put method supported something like fileUrl parameter instead of fileContent blob (as it is now).

[chintannp]

Hi Everyone,
Thanks for providing the valuable feedback. We are working to provide a solution for this. As this is a duplicate of #9736 , I will go ahead and close this issue. If you have anymore feedback, please direct it to the before mentioned issue.

[github-actions]

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server amplify-help forum.