New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OutOfMemory when uploading a huge file. #108
Comments
Hi all, I've experienced the same issue. Thanks in advance! |
@isabsent @FaustoMoon Sorry to see that nobody has gotten back to you yet regarding these questions. There's some documentation regarding the limitations of this endpoint that's worth calling out here: mainly, that it supports payloads up to 4MB in size. From graph.microsoft.io
I'm not aware of any support for large uploads yet in this SDK, perhaps the maintainer can chime in with any insight here... /cc @daboxu |
hey guys, sorry for late reply, here is a PR for supporting large file upload: #111 |
Thanks a lot! Would you be so kind to provide code snippets for your chunked upload in synchronous and asynchronous cases? |
Nice news, thanks! |
@isabsent here you go, I only applied callback pattern with synchronous case, curious how to apply async one? The operation is tightly combined with File IO and network so haven't found a good way to snitch into async way in the code. Any good ideas? The code I tested so far looks like: final AsyncTask<Void, Void, Void> uploadFile = new AsyncTask<Void, Void, Void>() {
@Override
protected Void doInBackground(final Void... params) {
try {
final ContentResolver contentResolver = getActivity().getContentResolver();
final ContentProviderClient contentProvider = contentResolver
.acquireContentProviderClient(data.getData());
contentProvider.release();
final String filename = FileContent.getValidFileName(contentResolver, data.getData());
final int fileSize = FileContent.getFileSize(contentProvider, data.getData());
final InputStream fileStream = contentResolver.openInputStream(data.getData());
final Option option = new QueryOption("@name.conflictBehavior", "fail");
oneDriveClient.getDrive()
.getRoot()
.getItemWithPath(<"Path to file">)
.getCreateSession(new ChunkedUploadSessionDescriptor())
.buildRequest()
.post()
.createUploadProvider(oneDriveClient, fileStream, fileSize, Item.class)
.upload(Collections.singletonList(option),
new IProgressCallback<Item>() {
@Override
public void progress(long current, long max) {
dialog.setProgress((int) current);
dialog.setMax((int) max);
}
@Override
public void success(Item item) {dialog.dismiss();
Toast.makeText(getActivity(),
application
.getString(R.string.upload_complete,
item.name),
Toast.LENGTH_LONG).show();
refresh();
}
@Override
public void failure(ClientException error) {
dialog.dismiss();
if (error.isError(OneDriveErrorCodes.NameAlreadyExists)) {
Toast.makeText(getActivity(),
R.string.upload_failed_name_conflict,
Toast.LENGTH_LONG).show();
} else {
Toast.makeText(getActivity(),
application
.getString(R.string.upload_failed,
filename),
Toast.LENGTH_LONG).show();
}
}
});
} catch (final Exception e) {
Log.e(getClass().getSimpleName(), e.getMessage());
Log.e(getClass().getSimpleName(), e.toString());
}
return null;
}
};
uploadFile.execute(); |
@isabsent @FaustoMoon it should be out with version 1.3.1, feel free to try it out and let me know if you have any questions. |
Thanks a lot! I will try right now. |
Works fine. I have used synchronous variant with Android IntentService. Does
fire after each 4 MB uploaded? If so, it seems too much because progress bar looks very intermittent . Other clouds use smaller portion. By the way, is there an option to interrupt the upload? |
@isabsent we do provide the the customization on the chunk you want to upload: so you can invoke like:
and due to service requirement, the number should be a multiple of 320KB. |
320 KB is good for smooth progress, thanks. Is there an option to cancel the upload? |
do you mean cancel during uploading? or cancel if the result returned is an exception? |
I mean cancel during uploading. |
that's a good point, I will add it to backlog, probably a Cancelalbe Interface is needed. |
hey guys, I've just tested the chunk upload and I find out an unexpected behaviour, after one hour of uploading I receive this message: Response code 401, Unauthorized hereafter the logcat in starting phase: `12-13 07:58:21.504 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 71: Intercepting request, https://api.onedrive.com/up/.... 12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 82: Found account information 12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 219: Starting to send request, URL https://api.onedrive.com/up/.... 12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 223: Request Method PUT 12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 229: Sending byte[] as request body 12-13 07:58:40.481 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 265: Response code 202, Accepted 12-13 07:58:40.481 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 270: StatefulResponse is handling the HTTP response. 12-13 07:58:40.486 2652-8018/com.onedfau.apiexplorer D/ChunkedUploadResponseHandler[generateResult] - 90: Chunk bytes has been accepted by the server. 12-13 07:58:40.489 2652-8018/com.onedfau.apiexplorer D/DefaultSerializer[deserializeObject] - 67: Deserializing type UploadSession 12-13 07:58:40.489 2652-8018/com.onedfau.apiexplorer D/PingMe: BigFile progress: 327680` then the logcat after one hour: `12-13 08:56:29.876 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 71: Intercepting request, https://api.onedrive.com/up/... 12-13 08:56:29.876 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 76: Found an existing authorization header! 12-13 08:56:29.877 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 219: Starting to send request, URL https://api.onedrive.com/up/... 12-13 08:56:29.877 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 223: Request Method PUT 12-13 08:56:29.878 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 229: Sending byte[] as request body 12-13 08:58:13.416 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 265: Response code 401, Unauthorized 12-13 08:58:13.417 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 270: StatefulResponse is handling the HTTP response. 12-13 08:58:13.417 2652-8018/com.onedfau.apiexplorer D/ChunkedUploadResponseHandler[generateResult] - 107: Receiving error during upload, see detail on result error 12-13 08:58:21.425 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 71: Intercepting request, https://api.onedrive.com/up/.... 12-13 08:58:21.425 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 76: Found an existing authorization header! 12-13 08:58:21.426 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 219: Starting to send request, URL https://api.onedrive.com/up/... 12-13 08:58:21.427 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 223: Request Method PUT 12-13 08:58:21.427 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 229: Sending byte[] as request body 12-13 08:58:59.558 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 265: Response code 401, Unauthorized 12-13 08:58:59.558 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 270: StatefulResponse is handling the HTTP response. 12-13 08:58:59.559 2652-8018/com.onedfau.apiexplorer 12-13 08:58:59.560 2652-8018/com.onedfau.apiexplorer D/PingMe: BigFile: upload error` Do you think that I've made a mistake in my implementation? thanks in advance for any suggestions. |
hi @FaustoMoon , did your uploading job exists for more than one hour? The oauth token is valid for 1 hour only so that's why you got unauthorized. Try with larger chunk size and you can see my example code above. |
Hi @daboxu, |
I am trying to upload media file with size of 141 MB to OneDrive by means of your code snippet:
(IOUtils is from org.apache.commons.io version 2.4) and get OOM error immediately:
How about a promised wrapper on chunked upload? :)
The text was updated successfully, but these errors were encountered: