Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

React Native Issue uploading smaller files #306

Open
paulsizer opened this issue Mar 7, 2021 · 28 comments · Fixed by #307 or #308
Open

React Native Issue uploading smaller files #306

paulsizer opened this issue Mar 7, 2021 · 28 comments · Fixed by #307 or #308
Assignees
Labels
bug Something isn't working

Comments

@paulsizer
Copy link

I have added this library to my ReactNative project. It works great for larger file sizes but trying to use it for smaller files doesn't seem to be working.

I get the following error when trying to upload smaller files:

UploadClientError: [400] Request does not contain files.

This works fine with 3-minute videos, I have no issues with it at all.

This is the code I am using to upload the files:

import UploadClient from '@uploadcare/upload-client'

const client = new UploadClient({ publicKey: 'XXX' })

const resp = await fetch(image.uri);
const blob = await resp.blob();
const uriParts = image.uri.split('.');
const ext = uriParts[uriParts.length - 1];
const filename = `file.${ext}`;

const uploadResponse = await client.uploadFile(blob, { fileName: filename });

Expo CLI 4.1.4 environment info:
System:
OS: macOS 10.15.7
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 12.16.3 - /usr/local/bin/node
Yarn: 1.22.4 - /usr/local/bin/yarn
npm: 6.14.4 - /usr/local/bin/npm
SDKs:
iOS SDK:
Platforms: iOS 14.4, DriverKit 20.2, macOS 11.1, tvOS 14.3, watchOS 7.2
IDEs:
Android Studio: 3.6 AI-192.7142.36.36.6392135
Xcode: 12.4/12D4e - /usr/bin/xcodebuild
npmPackages:
expo: ^40.0.1 => 40.0.1
react: 16.13.1 => 16.13.1
react-dom: 16.13.1 => 16.13.1
react-native: https://github.com/expo/react-native/archive/sdk-40.0.1.tar.gz => 0.63.2
react-native-web: ~0.13.12 => 0.13.14
npmGlobalPackages:
expo-cli: 4.1.4
Expo Workflow: managed

@paulsizer paulsizer added the bug Something isn't working label Mar 7, 2021
@paulsizer
Copy link
Author

We have changed to using the direct upload API for smaller file uploads (smaller than 100mb).

We now need a way to upload files greater than 100mb. I have tried to use the Javascript library above but that now also appears to be blowing up in my ReactNative app. I have tried changing the maxConcurrentRequests to 1 thinking it might be a resource issue but that still hasn't fixed it.

@paulsizer
Copy link
Author

I am now getting file size mismatch when calling complete on the multipart upload. Not sure if this is something to do with the way I do the chunking in my ReactNative (Expo) app.

const resp = await fetch(image.uri);
const blob = await resp.blob();
const uriParts = image.uri.split('.');
const ext = uriParts[uriParts.length - 1];
const filename = `file.${ext}`;

if (blob.size > 99000000) {
  multipartUpload(blob, filename);
} else {
  directUpload(image, filename);
}
const getChunk = (file, index, filesize, chunkSize) => {
  const start = chunkSize * index
  const end = Math.min(start + chunkSize, filesize)

  return file.slice(start, end)
}
const multipartStart = async(blob, filename) => {

  const body = new FormData
  body.append("UPLOADCARE_PUB_KEY", "XXX")
  body.append("filename", filename)
  body.append("size", blob.data.size)
  body.append("content_type", blob.type)
  body.append("UPLOADCARE_STORE", "auto")

  const response = await fetch("https://upload.uploadcare.com/multipart/start/?jsonerrors=1", {
    method: "POST",
    body,
    headers: {
      "Content-Type": "multipart/form-data"
    }
  })

  return response.json();

};
const uploadParts = async(blob, parts, uuid) => {
  const CHUNK_SIZE = 5 * 1024 * 1024;
  const chunks = [];
  for (let i = 0; i < parts.length; i++) {
    chunks.push(getChunk(blob, i, blob.data.size, CHUNK_SIZE));
  }
  console.log(chunks);
  let completed = 0;
  for (let i = 0; i < parts.length; i++) {
    let endpoint = parts[i];
    axios(endpoint, {
      method: 'PUT',
      headers: {
        'Content-Type': blob.type
      },
      data: chunks[i]
    })
    .then(res => {
      if (res.status == 200) {
        completed++;
      }
      console.log('Parts uploaded: ', completed);
      if (completed == parts.length) {
        multipartComplete(uuid);
      }
    })
    .catch(function (error) {
      console.log(error);
    });
  }

};
const multipartComplete = async(uuid) => {

  const body = new FormData
  body.append("UPLOADCARE_PUB_KEY", "XXX")
  body.append("uuid", uuid)

  const response = await fetch("https://upload.uploadcare.com/multipart/complete/?jsonerrors=1", {
    method: "POST",
    body,
    headers: {
      "Content-Type": "multipart/form-data"
    }
  })

  const result = await response.json();
  console.log(result);
};

@nd0ut
Copy link
Member

nd0ut commented Mar 11, 2021

I'm not quite understand why you're doing your own implementation?

Until we release a fixed version, you can import needed methods directly and use react-native file-like object ({ name, type, uri, size }). It will work for both direct and multipart uploads.

Example:

import defaultSettings from '@uploadcare/upload-client/lib/defaultSettings';
import uploadFile from '@uploadcare/upload-client/lib/uploadFile';
import uploadMultipart from '@uploadcare/upload-client/lib/uploadFile/uploadMultipart';

const uri = image.uri;
const resp = await fetch(uri);
const blob = await resp.blob();
const name = 'filename.ext';
const size = blob.size
const type = blob.type

const populateOptionsWithSettings = (options) => ({
  ...defaultSettings,
  ...options,
});

const options = { publicKey: 'demopublickey' }
const asset = { size, name, type, uri }

uploadFile(asset, populateOptionsWithSettings(options));
uploadMultipart(asset, populateOptionsWithSettings(options))

@paulsizer
Copy link
Author

@nd0ut using that approach the call to getChuck within uploadMultipart doesn't like the file.

This is the error I get using your example above.

Error - TypeError: file.slice is not a function

@paulsizer
Copy link
Author

paulsizer commented Mar 11, 2021

@nd0ut trying your approach, passing in the blob and extra options seems to do what I want but I get issues like a mentioned previously.

The whole reason for me trying my own approach was that it seems to be hit and miss and the app just completely crashes during an upload, doesn't drop in the the catch block just dumps out.

Like I said above I didn't know if it was something to do with the concurrent requests, this is why I get a failure without anyway to catch the error.

const options = {publicKey: 'XXX', fileName: name, contentType: type};

    uploadMultipart(blob, populateOptionsWithSettings(options))
    .then(value => 
      console.log(`Success - ${value}`)
    )
    .catch(error => 
      console.log(`Error - ${error}`)
    )

@nd0ut
Copy link
Member

nd0ut commented Mar 12, 2021

@paulsizer

using that approach the call to getChuck within uploadMultipart doesn't like the file.

Yep, I was wrong, uploadMultipart requires a Blob, sorry.

I was able to reproduce both problems with size mismatch and crashes on large files, I'll take a look into it. Thanks!

@paulsizer
Copy link
Author

@nd0ut

I'm glad you were able to reproduce the errors and it wasn't me doing something wrong my end. I have been banging my head against this for a while now!

Any timescales when you think the fix can be rolled out? We are looking at getting this in the app asap as we were doing our own transcoding and uploading on our own server but it makes no sense for us to do it. We are better offloading this onto your service.

@nd0ut
Copy link
Member

nd0ut commented Mar 15, 2021

@paulsizer
This is the first time I've encountered with react-native and I can't figure out what's causing these crashes. There is nothing that can highlight the cause - no errors in the system log or in the console. Since your simplified approach works without crashes, the cause is clearly somewhere in our codebase. We need a lot more time to find out the problem.

Now to the good stuff. I ran your code and it seems that axios doesn't work well in react-native and doesn't upload the Blob to the server, it's just sends an empty request instead, which resulted in a size mismatch. I replaced it with fetch and that's worked for me.

fetch(endpoint, {
  method: 'PUT',
  headers: {
    'Content-Type': 'image/jpeg'
  },
  body: chunks[i]
})

@nd0ut
Copy link
Member

nd0ut commented Mar 15, 2021

Ha, I turned off the queue and retry mechanics and the crashes are gone. I'm close to a clue.

@nd0ut
Copy link
Member

nd0ut commented Mar 15, 2021

It will not crash if you set maxConcurrentRequests to value greater than parts number, it will upload all the parts at once without concurrency. Still can't figure out why concurrent queue causes crashes at react-native.

@paulsizer
Copy link
Author

@nd0ut What would be the best way to do this? Is there a way to set that on the options for that you pass into populateOptionsWithSettings?

@nd0ut
Copy link
Member

nd0ut commented Mar 15, 2021

Catched it.

facebook/react-native#27543

When any of sliced blobs get collected, react-native will deallocate the whole blob, which is leading to the app crash.

@paulsizer for now you can temporarily set maxConcurrentRequests to some high enough value like 100 anywhere you want. I'll prepare a PR with the fix soon.

@paulsizer
Copy link
Author

@nd0ut Excellent thanks Alex, I'll give it a go!

@paulsizer
Copy link
Author

@nd0ut Hi Alex I tried the following but it seems to be even worse now! Crashes every single time, pretty much instantly now.

const options = {publicKey: 'XXX', fileName: name, contentType: type, maxConcurrentRequests: 100};

    uploadMultipart(blob, populateOptionsWithSettings(options))
    .then(value => 
      console.log(`Success - ${value}`)
    )
    .catch(error => 
      console.log(`Error - ${error}`)
    )

@nd0ut nd0ut reopened this Mar 17, 2021
@nd0ut
Copy link
Member

nd0ut commented Mar 17, 2021

@paulsizer I'll release a new version soon, it should work. #314

@nd0ut
Copy link
Member

nd0ut commented Mar 23, 2021

@paulsizer Does version 1.1.3 work as expected?

@paulsizer
Copy link
Author

@nd0ut Sorry it's taken a while to get back to you.

Myself and @stevesizer have tried to use the uploadFile method in the following ways:

const options = { publicKey: '882f02a0ca6451ab4795', fileName: name, contentType: type };
const asset = { size, name, type, uri };
var body = new FormData();
body.append('photo', asset);
uploadResponse = await uploadFile(body, populateOptionsWithSettings(options));

const options = { publicKey: '882f02a0ca6451ab4795', fileName: name, contentType: type };
const asset = { size, filename, type, uri };
uploadResponse = await uploadFile(asset, populateOptionsWithSettings(options));

But both result in the following:

TypeError: File uploading from "[object Object]" is not supported.

I have also tried to use the multipartUpload and I am still having issues...

const resp = await fetch(image.uri);
const blob = await resp.blob();
const options = { publicKey: 'XXX', fileName: name, contentType: type };
uploadResponse = await uploadMultipart(blob, populateOptionsWithSettings(options));

The above seemed to work the first time round. Tried it again with the same file and then it crashed as it has done in the past without any error.

I then remembered you mentioned about setting the maxConcurrentRequests. So I tried the following.

const resp = await fetch(image.uri);
const blob = await resp.blob();
 const options = { publicKey: 'XXX', fileName: name, contentType: type, maxConcurrentRequests: 100 };
uploadResponse = await uploadMultipart(blob, populateOptionsWithSettings(options));

As before this just dumps straight out no errors, just a complete crash...

Definitely using the new version 1.1.3

 "dependencies": {
    "@react-native-community/datetimepicker": "3.0.4",
    "@react-native-community/masked-view": "0.1.10",
    "@react-native-community/netinfo": "5.9.7",
    "@react-native-community/segmented-control": "2.2.1",
    "@react-native-community/slider": "3.0.3",
    "@react-navigation/bottom-tabs": "^5.9.1",
    "@react-navigation/native": "^5.7.5",
    "@react-navigation/stack": "^5.9.2",
    "@uploadcare/upload-client": "^1.1.3",
    "axios": "^0.21.1",
    "expo": "^40.0.1",

@nd0ut
Copy link
Member

nd0ut commented Mar 29, 2021

@paulsizer
Starting from 1.1.3 version, uploadFile doesn't support { size, filename, type, uri } input format, blobs only.

The above seemed to work the first time round. Tried it again with the same file and then it crashed as it has done in the past without any error.

Are you trying to upload the same blob or a new one a second time? React-native deallocates blob from the memory after upload complete, so you can't use the same blob after uploading and need to recreate it from uri again.

@paulsizer
Copy link
Author

@nd0ut I am creating a new one again from uri

@nd0ut
Copy link
Member

nd0ut commented Mar 31, 2021

@paulsizer Is it the same uri, or a new one?

Strange thing. Probably, fresh blobs are created by reference, not by value and react-native clears memory that blob and uri pointing on.

I was tested double uploads using expo-asset's fromModule with no errors. But I don't know how it's works, seems that it creates new files in memory every time.

I'll dig into it.

@paulsizer
Copy link
Author

paulsizer commented Mar 31, 2021 via email

@nd0ut
Copy link
Member

nd0ut commented Apr 5, 2021

@paulsizer According to this comment, new Blob([blob]) should create a copy of data in memory. I did some tests and it seems to work.

Example:

let blobCopy = new Blob([blob])
await client.uploadFile(blobCopy, options)
...

Please, check it on your end and I'll add this workaround to the upload-client codebase if it would work.

@nd0ut
Copy link
Member

nd0ut commented May 11, 2021

Hey @paulsizer,

did this workaround work?

@paulsizer
Copy link
Author

Hey sorry @nd0ut I have been super busy trying to get other pieces of the app out.

I have just tested what you suggested and it dumped me straight out. No errors just a fatal crash.

@stevesizer
Copy link

stevesizer commented May 18, 2021 via email

@nd0ut
Copy link
Member

nd0ut commented May 24, 2021

@paulsizer could you provide a minimal snippet? I can't reproduce it :(

@paulsizer
Copy link
Author

@nd0ut I have been testing this morning and everything appears to be working for the larger files now if we set

maxConcurrentRequests = 1

We are doing the following to call the different APIs depending on the size, as there is a size limit for multipart uploads

if (size > 10000000) {
      uploadResponse = await uploadMultipart(blob, populateOptionsWithSettings(options));
      console.log(uploadResponse);
    } else {
      uploadResponse = await uploadFile(blob, populateOptionsWithSettings(options));
      console.log(uploadResponse)
    }

This works as expected on iOS but on Android for images that are under the 10Mb threshold we get an error that seems to come from using the uploadFile method.

Cannot create URL for blob!
- node_modules/react-native/Libraries/Blob/URL.js:120:12 in createObjectURL
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147755:18 in Object.exports.transformFile
- node_modules/@uploadcare/upload-client/lib/tools/buildFormData.js:15:12 in _loop_1
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147734:6 in Object.buildFormData [as default]
- node_modules/@uploadcare/upload-client/lib/api/base.js:19:38 in retryIfThrottled_1._default$argument_0
- node_modules/@uploadcare/upload-client/lib/tools/retryIfThrottled.js:14:34 in retry_1._default$argument_0
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147925:13 in runAttempt
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147931:11 in Object.retrier [as default]
- node_modules/@uploadcare/upload-client/lib/tools/retryIfThrottled.js:12:9 in retryIfThrottled
- node_modules/@uploadcare/upload-client/lib/api/base.js:16:438 in base

We do not get this error on iOS.

@nd0ut
Copy link
Member

nd0ut commented Apr 18, 2022

@paulsizer Try to use react-native-url-polyfill on Android to create URL for blobs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment