New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Upload is too slow for a large file #693

Closed
Laeyoung opened this Issue Jun 16, 2017 · 10 comments

Comments

Projects
None yet
8 participants
@Laeyoung

Laeyoung commented Jun 16, 2017

  • What service are you using?
    S3 (APNortheast-1)

  • In what version of SDK are you facing the problem?
    2.5.8

  • Is the issue limited to Simulators / Actual Devices?
    Actual devices.

  • Can your problem be resolved if you bump to a higher version of SDK?
    Already using the latest version.

  • Is this problem related to specific iOS version?
    Generic

  • How are you consuming the SDK? CocoaPods / Carthage / Prebuilt frameworks?
    CocoaPods

  • Can you give us steps to reproduce with a minimal, complete, and verifiable example? Please include any specific network conditions that might be required to reproduce the problem.

When I try to upload a large size file using S3TransferManager or S3TransferUtiltity, uploading speed is really slow. Some users have reported similar issues.

Now, I am using 3rd party library(https://github.com/AnthonyMDev/AmazonS3RequestManager) which uses http request to upload and is 3 times faster than AWS iOS SDK. I hope for this issue to be solved soon.

@kvasukib

This comment has been minimized.

Show comment
Hide comment
@kvasukib

kvasukib Jun 20, 2017

Contributor

Hi @Laeyoung,

Thanks for reporting to this.

S3TransferManager does multi-part upload which does concurrent part uploads to improve the performance.

  1. Have you tried using S3 Transfer Acceleration? Does AmazonS3RequestManager use acceleration by default?
  2. Is the upload speed slow for every large file? Can you specify the file sizes in which the upload speed is slow?
  3. Can you post a snippet of code that invokes the upload function?
Contributor

kvasukib commented Jun 20, 2017

Hi @Laeyoung,

Thanks for reporting to this.

S3TransferManager does multi-part upload which does concurrent part uploads to improve the performance.

  1. Have you tried using S3 Transfer Acceleration? Does AmazonS3RequestManager use acceleration by default?
  2. Is the upload speed slow for every large file? Can you specify the file sizes in which the upload speed is slow?
  3. Can you post a snippet of code that invokes the upload function?
@Laeyoung

This comment has been minimized.

Show comment
Hide comment
@Laeyoung

Laeyoung Jun 21, 2017

Hi @kvasukib

I tested translate acceleration feature with AWSS3TransferUtility. Upload speed didn't increase.
AmazonS3RequestManager doesn't support acceleration feature.

I am sorry, that's my mistake. It doesn't depend on file size.

I tested on Android and iOS via two WIFI APs, one is fast, another one is slow.

Platform / WIFI AP / Filesize(Bytes) / elapsed time(seconds) / MBps
iOS/Fast/18860030/21.4/0.88
iOS/Fast/49663141/25.1/1.97
iOS/Fast/89717412/44.9/1.99
iOS/Fast/108048235/57.9/1.86
iOS/Fast/289532082/141.2/2.05
Android/Fast/67813793/12.18/5.56

iOS/Slow/102349723/496.8/0.20
iOS/Slow/26585616/187.5/0.14
iOS/Slow/6052769/43.6/0.13
Android/Slow/11900000/25.6/0.46
Android/Slow/53400000/98/0.54

iOS-AmazonS3RequestManager/Fast/267834688/54/4.95
iOS-AmazonS3RequestManager/Slow/42178449/78.4/0.53
iOS-AmazonS3RequestManager/Slow/144240714/459.4/0.31

Uploading on Android SDK is 3 times faster than iOS SDK.

private func uploadFile() {
    let mTransferUtility = AWSS3TransferUtility.default()
    let expression = AWSS3TransferUtilityUploadExpression()
    expression.progressBlock = mProgressBlock
    
    mTransferUtility.uploadFile(
        fileUrl,
        bucket: Const.S3_BUCKET,
        key: bucketKey,
        contentType: "video/mp4",
        expression: expression,
        completionHandler: mCompletionHandler).continueWith { (task) -> AnyObject! in
            if let error = task.error {
                Log.error("Error: \(error.localizedDescription)")
                self.uploadFailed()
                return nil
            }
            
            if let result = task.result {
                Log.info("Result: \(result)")
                return nil
            }
            
            return nil
    }
}

My Test Env

  • iPhone 5 (10.3.2)
  • Xcode 8.3.3
  • AWS SDK 2.5.8

Laeyoung commented Jun 21, 2017

Hi @kvasukib

I tested translate acceleration feature with AWSS3TransferUtility. Upload speed didn't increase.
AmazonS3RequestManager doesn't support acceleration feature.

I am sorry, that's my mistake. It doesn't depend on file size.

I tested on Android and iOS via two WIFI APs, one is fast, another one is slow.

Platform / WIFI AP / Filesize(Bytes) / elapsed time(seconds) / MBps
iOS/Fast/18860030/21.4/0.88
iOS/Fast/49663141/25.1/1.97
iOS/Fast/89717412/44.9/1.99
iOS/Fast/108048235/57.9/1.86
iOS/Fast/289532082/141.2/2.05
Android/Fast/67813793/12.18/5.56

iOS/Slow/102349723/496.8/0.20
iOS/Slow/26585616/187.5/0.14
iOS/Slow/6052769/43.6/0.13
Android/Slow/11900000/25.6/0.46
Android/Slow/53400000/98/0.54

iOS-AmazonS3RequestManager/Fast/267834688/54/4.95
iOS-AmazonS3RequestManager/Slow/42178449/78.4/0.53
iOS-AmazonS3RequestManager/Slow/144240714/459.4/0.31

Uploading on Android SDK is 3 times faster than iOS SDK.

private func uploadFile() {
    let mTransferUtility = AWSS3TransferUtility.default()
    let expression = AWSS3TransferUtilityUploadExpression()
    expression.progressBlock = mProgressBlock
    
    mTransferUtility.uploadFile(
        fileUrl,
        bucket: Const.S3_BUCKET,
        key: bucketKey,
        contentType: "video/mp4",
        expression: expression,
        completionHandler: mCompletionHandler).continueWith { (task) -> AnyObject! in
            if let error = task.error {
                Log.error("Error: \(error.localizedDescription)")
                self.uploadFailed()
                return nil
            }
            
            if let result = task.result {
                Log.info("Result: \(result)")
                return nil
            }
            
            return nil
    }
}

My Test Env

  • iPhone 5 (10.3.2)
  • Xcode 8.3.3
  • AWS SDK 2.5.8
@ManishBTC

This comment has been minimized.

Show comment
Hide comment
@ManishBTC

ManishBTC Jul 6, 2017

I am also facing the same problem. Upload process is fast in android as compared to iOS. I am using AWS SDK 2.5.8 added via Frameworks, Xcode 8.3.2. Also background upload hangs after some time without throwing any error for large files. Yes, I am using AWSTransferUtility and the below function has also been added:

func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @Escaping () -> Void) {

    AWSS3TransferUtility.interceptApplication(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler)
}

ManishBTC commented Jul 6, 2017

I am also facing the same problem. Upload process is fast in android as compared to iOS. I am using AWS SDK 2.5.8 added via Frameworks, Xcode 8.3.2. Also background upload hangs after some time without throwing any error for large files. Yes, I am using AWSTransferUtility and the below function has also been added:

func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @Escaping () -> Void) {

    AWSS3TransferUtility.interceptApplication(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler)
}
@hernangonzalez

This comment has been minimized.

Show comment
Hide comment
@hernangonzalez

hernangonzalez Aug 23, 2017

Hi, I am seeing really slow uploads as well on iOS. Anyone have found a workaround it?

hernangonzalez commented Aug 23, 2017

Hi, I am seeing really slow uploads as well on iOS. Anyone have found a workaround it?

@mbesnili

This comment has been minimized.

Show comment
Hide comment
@mbesnili

mbesnili Aug 26, 2017

I'm having this problem as well, uploading a video to s3 takes a long time.

mbesnili commented Aug 26, 2017

I'm having this problem as well, uploading a video to s3 takes a long time.

@rohandubal rohandubal added the S3 label Sep 12, 2017

@cbommas

This comment has been minimized.

Show comment
Hide comment
@cbommas

cbommas Jan 30, 2018

@mbesnili @hernangonzalez @ManishBTC @Laeyoung

As mentioned by @kvasukib, the iOS Transfer Utility does not use multi-part for uploads, where as the Android Transfer Utility uses multi-part. That is the reason why the Android uploads are faster.

We are working on adding multi-part functionality to the iOS Transfer Utility. Please check back on this thread [or look for general forum announcements] for updates on this topic.

cbommas commented Jan 30, 2018

@mbesnili @hernangonzalez @ManishBTC @Laeyoung

As mentioned by @kvasukib, the iOS Transfer Utility does not use multi-part for uploads, where as the Android Transfer Utility uses multi-part. That is the reason why the Android uploads are faster.

We are working on adding multi-part functionality to the iOS Transfer Utility. Please check back on this thread [or look for general forum announcements] for updates on this topic.

@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot Apr 11, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale bot commented Apr 11, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@ericmgoeken

This comment has been minimized.

Show comment
Hide comment
@ericmgoeken

ericmgoeken Apr 11, 2018

@cbommas, any update? stale bot is trying to remove my hope that this will get fixed.

ericmgoeken commented Apr 11, 2018

@cbommas, any update? stale bot is trying to remove my hope that this will get fixed.

@cbommas

This comment has been minimized.

Show comment
Hide comment
@cbommas

cbommas Apr 11, 2018

Hello @ericmgoeken

Transfer Utility now supports multipart uploads [see https://docs.aws.amazon.com/aws-mobile/latest/developerguide/how-to-transfer-files-with-transfer-utility.html]. Please upgrade to the latest version and let us know if you run into any issues.

cbommas commented Apr 11, 2018

Hello @ericmgoeken

Transfer Utility now supports multipart uploads [see https://docs.aws.amazon.com/aws-mobile/latest/developerguide/how-to-transfer-files-with-transfer-utility.html]. Please upgrade to the latest version and let us know if you run into any issues.

@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot May 11, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale bot commented May 11, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment