Swift rewrite discussion

[MMasterson]

Re writing the library in swift in the swift-development branch
original gist

Proposed changes for TUSKit

Goals

• Ditch objective-c and adopt Swift
• Change blocks to delegates
• Replicate the flow of the Android TUS library to bring synergy to TUS mobile native development
• Make it easier for multiple file uploads
• Better support for background sessions

Proposed new usage

// Init and setup the file upload url
TUSConfig {
url: "UPLOAD URL HERE"
sessionConfig: SOME_URL_SESSION_CONFIG //Not required, if not set the default session config will be set
}

TUSClient.setup(config) //MUST BE FIRED FIRST, AppDelegate on launch suggested 

// To access the client, use the singlegton
var client = TUSClient.shared

//Set the delegate 
client.delegate = self

//You can also change the file upload URL at anytime
client.uploadURL = "https://newUploadURL.com/files"

//Create a TUS Upload per file you want to upload
//On creation of an upload, the data of an object will be saved to the device until completion of the upload
var newUpload = TUSUpload(withId: "Some Id to refrence a file", andFile: "FilePathHere")
var anotherNewUpload = TUSUpload(withId: "Another Id to refrence a file", andData: DataObject)
var previousUpload = TUSUpload(withId: "Just an ID of a past upload")

//Misc functions for TUSUpload
newUpload.cancel()
newUpload.retry()


//Misc functions for client
client.currentUploads() //an array of TUSUpload objects of uploads unfinished
client.uploadStatus //An enum TUSStatus - either set to `uploading` `paused` `finished`
client.retryAll()
client.resumeAll()
client.cancelAll()
client.cleanUp() //Deletes local files of canceled or failed uploads - Files cannot be resumed after this is fired

//Now upload
client.createOrResumeUpload(TUSUploadObjectHere)
//or
client.createOrResumeUpload(TUSUploadObjectHere, withRetries: 3)


//TUSKitDelegate

func TUSProgress(bytesUploaded, bytesRemaining) //overall current upload progress

func TUSProgress(TUSUpload, bytesUploaded, bytesRemaining) //Per file upload progress

func TUSSuccess(TUSUpload, TUSResponse)

func TUSFailure(TUSUpload, TUSResponse, Error)

Remaining tasks

• Custom Session Configs to be passed to clients
• • Sub task: SSL Pinning
• TUS Create Post Call
• TUS Upload Via Chunking
• TUS Get call for file
• TUS Resuming
• • State persistence
• • Resuming from last offset/chunk
• TUS Response handling
• • Success file creation
• • Success file upload
• • Failed file storage
• • Bad server response

[foxware00]

@MMasterson Just had a read over this, I'm all for a swift transition how long do you estimate this taking?

I'm going to play around and cleanup my branch for delegates and open that to see if I was going down the wrong path. I've been learning objective-c (coming from Android straight into Swift) so I may have made a few mistakes. It sounds like we'll can the objective-c delegates anyway but would be good to test out our assumptions.

[foxware00]

@MMasterson Another idea I had over the weekend, possibly not worth the first pass on the re-write is adding support for iOS 13 Background tasks, now that we'll be using a background session I would hope it will be a simple addition to increase the lifetime of background uploads. I don't have any experience with the API myself.

https://developer.apple.com/documentation/backgroundtasks

[Acconut]

Amazing initiative, Mark! I just have two comments

Ditch objective-c and adopt Swift

How will this impact current users? Are folks developing in Objective-C able to use the Swift version without problems? I am not sure what the current share of Swift projects is.

Also, I am wondering if and how this could impact TransloaditKit, which also uses TusKit internally.

Replicate the flow of the Android TUS library to bring synergy to TUS mobile native development

Very good idea! Just want to add that we are currently planing some changes to tus-js-client which will then also trickle down to tus-java-client and tus-android-client. :)

[foxware00]

Just reading over your proposed API covered above, looks like some good improvements there I have some additional code that after successful upload I need to mark some stuff as finished on my side. Thus the TUSResponse or TUSUpload would be really useful. Something I've noted is I can have a 401 during one of the upload chunks so I need to refresh the oauth token retry, however, 401 falls under a normal "checkingFile" flow and I can't update the Authorization header..
I can either use the cookie store and update it that way, or I can add a block giving me information of the 401 allowing me to pause, add the header and trigger the retry. Just a thought as to whether this sort of custom retry/error handling might make sense, otherwise the cookie route might make more sense if I can get that to work.

This also highlights another question, when we go through the retry flow, instead of stopping when retries are reached. Shouldn't we be calling the error callback to let the caller know it's failed?

After having a play I can see that [request setHTTPShouldHandleCookies:NO]; Is applied everywhere not allowing me to manage the cookie store. This would be a nice thing to be configurable on the TUSSession

[MMasterson]

Amazing initiative, Mark! I just have two comments

Ditch objective-c and adopt Swift

How will this impact current users? Are folks developing in Objective-C able to use the Swift version without problems? I am not sure what the current share of Swift projects is.

Objective-C and Swift are able to be used with each other. The API will change though, so the only impact is users will have to use the new API.

Also, I am wondering if and how this could impact TransloaditKit, which also uses TusKit internally.

Yes, updates will have to be made to TransloaditKit, that's being kept in mind - The old version of TUSKit will still be available so the release of Swift TUSKit won't kill TransloaditKit

Replicate the flow of the Android TUS library to bring synergy to TUS mobile native development

Very good idea! Just want to add that we are currently planing some changes to tus-js-client which will then also trickle down to tus-java-client and tus-android-client. :)

Can you share those changes?

[MMasterson]

Re: @foxware00's comments

Looking to have this wrapped by Saturday/Sunday

@MMasterson Another idea I had over the weekend, possibly not worth the first pass on the re-write is adding support for iOS 13 Background tasks, now that we'll be using a background session I would hope it will be a simple addition to increase the lifetime of background uploads. I don't have any experience with the API myself.

https://developer.apple.com/documentation/backgroundtasks

We will implement something, and provide some documentation for background tasks. Just remember iOS13 is very strict on background processes

Just reading over your proposed API covered above, looks like some good improvements there I have some additional code that after successful upload I need to mark some stuff as finished on my side. Thus the TUSResponse or TUSUpload would be really useful. Something I've noted is I can have a 401 during one of the upload chunks so I need to refresh the oauth token retry, however, 401 falls under a normal "checkingFile" flow and I can't update the Authorization header..
I can either use the cookie store and update it that way, or I can add a block giving me information of the 401 allowing me to pause, add the header and trigger the retry. Just a thought as to whether this sort of custom retry/error handling might make sense, otherwise the cookie route might make more sense if I can get that to work.

This also highlights another question, when we go through the retry flow, instead of stopping when retries are reached. Shouldn't we be calling the error callback to let the caller know it's failed?

After having a play I can see that [request setHTTPShouldHandleCookies:NO]; Is applied everywhere not allowing me to manage the cookie store. This would be a nice thing to be configurable on the TUSSession

I think if you set a retry, and it failes, it should retry X times before firing an error. With this new setup you can have your own retry methodology, for example

Do a TUSUpload and don't set a retry amount, then if the upload does fail out, you can fire TUSUpload.resume() from the error delegate method based on whatever criteria you have.

And yes, a custom session configuration should be able to be passed to TUSClient this way you can configure how you want the cookies and cache to work

[shravanteegala]

Thank you.
rewrite to swift. In this multiple file upload feature is implemented or not ?

[MMasterson]

Yes @shravanteegala. A simple example of how multiple files could work in this rewrite is as follows

var arrayOfFiles = [URL(string: "someFilePath/fileName1.jpg"), URL(string: "someFilePath/fileName2.png")]

arrayOfFiles.forEach { path in
    fileName = path.lastPathComponent.deletingPathExtension
    var newUpload = TUSUpload(withId: fileName, andFile: path)
    TUSClient.shared.createOrResume(withUpload: newUpload)
}

and now the delegate methods will return a TUSUpload object when fired, and you can identify which upload is being referenced about by checking the id

func TUSProgress(upload,_ uploaded, remaining) {
    if (upload.id == "fileName1") {
        //fileName1 was referenced - do something for that upload
        fileName1ProgressLabel.text = String(format: "%@ out of %@", uploaded, remaining) 
    }

    if (upload.id == "fileName2") {
        //fileName2 was referenced - do something for that upload
        fileName2ProgressLabel.text = String(format: "%@ out of %@", uploaded, remaining) 
    }
}

[shravanteegala]

@MMasterson .
multiple uploads we must need to asynchronous instead of synchronous. bcz synchronous operations are block the UI until it complete.

[MMasterson]

@shravanteegala yes of course, the uploads will happen async/off the main thread to avoid blocking the UI. However that is not something you'll have to worry about when implementing the library, it'll be taken care of behind the scenes.

[foxware00]

@MMasterson do you have your latest code, would love to have a crack at it?

[StackHelp]

@MMasterson One more thing I want to share is that TUSResumableUpload doesn't gonna work on reboot an app as File Directory Path gonna change on a reboot of an app. Or Is there any solution? 🤔🤔🤔

[MMasterson]

@foxware00 you can check the swift-development branch

@StackHelp I'm sorry, I don't fully understand your issues. Can you provide another example of your problem?

[MMasterson]

Looking to wrap this up tomorrow & Sunday and hit all the points listed in the og post

[foxware00]

@MMasterson thank you. I had my own punt at migrating it to swift earlier in the week. Managed to get it all working in a similar vain to the current lib. Trying to as closely follow your proposed API. I did notice however the task of background processing is an interesting one. We'd need to chunk the file into smaller files and uploads those and the dataTask API doesn't properly run in the background. Using uploadTask with an actual file enables much better background running.

Im not sure I'll have time to look into it this weekend, but I didn't know if you'd tried tackling this problem previously?

[StackHelp]

@MMasterson Yes, Let's say when I create an upload, I have this path /var/mobile/Containers/Data/Application/<app id>/Documents/file.json for file.json. Then let's say I paused the upload and reboot the app. On reboot <app id> gonna change. So, the file path for paused upload to start from that point is not gonna work as upload data can't get the correct file path now. So, do you think we can archive this kind of scenario here where App Id changed doesn't affect resumable uploads? Or is it not possible?

[chandu4ugandhi]

When will I get the new Swift code? @MMasterson

[shravanteegala]

hi,
@MMasterson i am waiting for fully swift asynchronous multiple file uploaded TusKit library.
plz give it, asap.

[MMasterson]

@chandu4ugandhi @shravanteegala @StackHelp @foxware00
Sorry about the delay - been a busy week over here at my day job. The library is getting close and will have it done ASAP. Just have to finish the networking side of things. Will update you over the next few days - shouldn't take much longer.

@StackHelp
We'll be saving the data with the FileManager and will be able to retrieve it with the FileManager on reboot of the app. You won't lose your data on reboot of the app.

@foxware00
Will look into uploadTask for better background support. And yes, we will be chunking.

[sandeeppsmys]

@chandu4ugandhi @shravanteegala @StackHelp @foxware00
Sorry about the delay - been a busy week over here at my day job. The library is getting close and will have it done ASAP. Just have to finish the networking side of things. Will update you over the next few days - shouldn't take much longer.

Cool.. Appreciate your passion and commitment.

@StackHelp
We'll be saving the data with the FileManager and will be able to retrieve it with the FileManager on reboot of the app. You won't lose your data on reboot of the app.

@foxware00
Will look into uploadTask for better background support. And yes, we will be chunking.

[MMasterson]

Committed a bunch of stuff today relating to the chunking & networking - getting it more ironed out. Feel free to take a look in the swift dev branch.

Will do the last bits and test over the next day or so

[chandu4ugandhi]

I have checked and most of the stuff completed in it but delegates and some other stuff remaining

[MMasterson]

Please check the readme of swift-development for what to expect from the published library. Let me know if there's anything you'd like to see or something you'd like to work differently.

[StackHelp]

@MMasterson I am trying to upload with /var/mobile/Containers/Data/Application/9457A41C-AD75-43BE-9606-2D2A3DC990D3/Documents/img_637229814909693312.heic file path.

        let fileName = "img_637229814909693312.heic"
        let newUpload = TUSUpload(withId: fileName, andFilePathString: filePath)
        TUSClient.shared.createOrResume(forUpload: newUpload)

And I get this error Failed to get file attributes for local path: /var/mobile/Containers/Data/Application/9457A41C-AD75-43BE-9606-2D2A3DC990D3/Documents/TUS/img_637229814909693312.heicimg_637229814909693312.heic with error: Error Domain=NSCocoaErrorDomain Code=260 "The file “img_637229814909693312.heicimg_637229814909693312.heic” couldn’t be opened because there is no such file." UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/9457A41C-AD75-43BE-9606-2D2A3DC990D3/Documents/TUS/img_637229814909693312.heicimg_637229814909693312.heic, NSUnderlyingError=0x283fa6250 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

You can see it takes fileName 2 times here
img_637229814909693312.heicimg_637229814909693312.heic

[chandu4ugandhi]

@MMasterson when will you resolve this issue?

[StackHelp]

@MMasterson I got the same error with this update as well.

  URL: file:///private/var/mobile/Containers/Data/Application/ED57AC20-29B6-4CD9-8CB2-6C9D6B6D1D89/tmp/4ac98994-c304-47fc-b4c7-58fef1e370f2.jpg, STR: image/jpeg

  TUSKit: Creating file img_637260632099908480 on server

  Failed moving file file:///private/var/mobile/Containers/Data/Application/5DFFD609-D987-4D75-88E6-4347BE1C8D5E/tmp/197e5acb-8423-4b6d-b21a-a6102253cf36.jpg to /var/mobile/Containers/Data/Application/ED57AC20-29B6-4CD9-8CB2-6C9D6B6D1D89/Documents/TUS/img_637260631150321408.jpeg for TUS folder storage

  “197e5acb-8423-4b6d-b21a-a6102253cf36.jpg” couldn’t be moved to “TUS” because either the former doesn’t exist, or the folder containing the latter doesn’t exist.

[MMasterson]

TLPhotoViewController.swift should cover an implementation of the two libraries, I'm unsure of what the difference is. @chandu4ugandhi is sending me some code so I can see where the disconnect is, will report back once I take a look.

from TLPhotoViewController.swift

// From TLPhotosPickerViewControllerDelegate
func shouldDismissPhotoPicker(withTLPHAssets: [TLPHAsset]) -> Bool {

    // I set a global array to the array supplied here, for access later, but not super needed
    self.selectedAssets = withTLPHAssets
    
    /*
     temp copy media file is the only way I see to get the file path.
     I only use the first element of the array, but you can cycle though and do the same process to add multiple files
     */
    
    //Run this for each element, I only use the first
    var file = selectedAssets[0].tempCopyMediaFile { (url, string) in
        
        //Create an upload object
        var upload = TUSUpload(withId: "File", andFilePathURL: url, andFileType: ".jpeg")
        
        //  Pass it to TUSClient, this can be done multiple times, it'll add it to TUSKit's `currentUploads`
        TUSClient.shared.createOrResume(forUpload: upload)

    }
    return true
}

[shravanteegala]

TLPhotoViewController.swift should cover an implementation of the two libraries, I'm unsure of what the difference is. @chandu4ugandhi is sending me some code so I can see where the disconnect is, will report back once I take a look.

from TLPhotoViewController.swift

// From TLPhotosPickerViewControllerDelegate
func shouldDismissPhotoPicker(withTLPHAssets: [TLPHAsset]) -> Bool {

    // I set a global array to the array supplied here, for access later, but not super needed
    self.selectedAssets = withTLPHAssets
    
    /*
     temp copy media file is the only way I see to get the file path.
     I only use the first element of the array, but you can cycle though and do the same process to add multiple files
     */
    
    //Run this for each element, I only use the first
    var file = selectedAssets[0].tempCopyMediaFile { (url, string) in
        
        //Create an upload object
        var upload = TUSUpload(withId: "File", andFilePathURL: url, andFileType: ".jpeg")
        
        //  Pass it to TUSClient, this can be done multiple times, it'll add it to TUSKit's `currentUploads`
        TUSClient.shared.createOrResume(forUpload: upload)

    }
    return true
}

Its Working fine for images.😀

But when i choose video's 🎥
I am using below code some times getting asset values some times getting nil.
what is the issue here? how can i do ?

PHImageManager.default().requestAVAsset(forVideo: mPhasset, options: options, resultHandler: { (asset, audioMix, info) in
if let urlAsset = asset as? AVURLAsset {
let localVideoUrl = urlAsset.url
completionHandler(localVideoUrl)
} else {
completionHandler(nil)
}
})

[MMasterson]

@shravanteegala thank you for pointing that out! Working on a patch for other file types, will report back what the issue was and why it was triggered.

[shravanteegala]

Hi MMasterson.
When App background uploading mode App is crashed.
app is running in the background.

[MMasterson]

hey @shravanteegala

A new version was just uploaded - can you update. I don't think it'll fix your issue just yet, but just to stay up to date.

What is at line 2460 on ChattingViewController?

[MMasterson]

@chandu4ugandhi @StackHelp please update

try using
TUSUpload(withId: id, andFilePathURL: file, andFileType: "type" )

if you get an error, please try wrapping the file url with URL(fileURLWithPath: String)

If error persists, please let me know. You can still upload via data for now

let upload: TUSUpload = TUSUpload(withId: String(number), andData: fileData, andFileType: "jpeg")

You can grab the fileData with var fileData = try! Data(contentsOf: file)

[shravanteegala]

hey @shravanteegala

A new version was just uploaded - can you update. I don't think it'll fix your issue just yet, but just to stay up to date.

5 Days ago new updated version is implemented for multiple files uploading based on id and
Data. we implemented based on id using below one.

//When you have a file, create an upload, and give it a Id.
let upload: TUSUpload = TUSUpload(withId: String(format: "%@%@", "img", String(number)), andFilePathURL: file, andFileType: ".jpeg") //let upload: TUSUpload = TUSUpload(withId: String(format: "%@%@", "img", String(number)), andFilePathURL: file, andFileType: ".jpeg")
var fileData = try! Data(contentsOf: file)
let upload: TUSUpload = TUSUpload(withId: String(number), andData: fileData, andFileType: "jpeg")

But my problem is after uploading more than one images then immediately i put app it background at that time app is crashed.

Can you give me updated version number . for i'll updated to my project.

[Acconut]

@MMasterson Just wanted to ask if there are any updates on the Swift version? :)

[MMasterson]

Yes sorry, will commit some updates this week. My main gig has been pretty busy. PRs are always welcome!!

[jsahoo]

@MMasterson how do you specify custom headers and metadata for the upload?

I found this in the docs for the Objective-C version, but not seeing anything in the new Swift docs

TUSResumableUpload *upload = [self.tusSession createUploadFromFile:fileUrl retry:3 headers:@{} metadata:@{}];

[YazilimTuneli]

After the first file upload, I could not upload again and I found the reason for this.
upload.status = .ready
It should be done this way after every successful installation.

[MMasterson]

@jsahoo There isn't one currently, but can add to the next release.

@YazilimTuneli will check into

[MMasterson]

@jsahoo custom headers are in the 2.1.0

[jsahoo]

@jsahoo custom headers are in the 2.1.0

Thanks!

[sripberger]

I'm seeing that all the signatures for TusUpload require you to specify an id. Is there no planned support for allowing the server to generate an id, as specified here, and as seems to be supported in the Objective-C version?

https://tus.io/protocols/resumable-upload.html#creation

[MMasterson]

@sripberger there are plans to bring back support for an already created file/id. I have some early support code for it I need to clean up and test a bit. Are you trying to upload to Vimeo or your own TUS instance?

[sripberger]

@sripberger there are plans to bring back support for an already created file/id. I have some early support code for it I need to clean up and test a bit. Are you trying to upload to Vimeo or your own TUS instance?

We are trying to upload to our own tus instance, which happens to be tusd.

What I'm getting at is not re-using an already-created file id, but the Create extension, where you make a POST to the root of the TUS endpoint, and it gives you back a Location header which contains an ID generated by the server.

[sripberger]

So, after talking with my co-worker it looks like we're talking about two different things. The ids provided to TusUpload to be internal to the client, and don't line up with the upload id used to track the file on the server (i.e. the last term of the upload path).

Is there any way to get that server-side id after an upload completes?

[MMasterson]

@sripberger yeah we can definitely fix that up. I'm curious as to what ID TUSD uses instead of the one supplied by the client. I see that we're sending the ID to TUS is in base64, could that be the culprit of id's not matching?

cc'ing @Acconut here as well

[sripberger]

@sripberger yeah we can definitely fix that up. I'm curious as to what ID TUSD uses instead of the one supplied by the client. I see that we're sending the ID to TUS is in base64, could that be the culprit of id's not matching?

cc'ing @Acconut here as well

The id the server uses is generated with this:

https://github.com/tus/tusd/blob/master/internal/uid/uid.go

I'll try to clear up some confusion, though. The situation is that we have the server set up to create a database entry for a completed upload. The client then needs the id of this database entry in order to proceed with further operations on the file-- associating it with other database entities, like posting a video to a board of videos or something.

For simplicity, we are simply re-using the server-side id created by tusd, which works fine since it is both cryptographically secure and close enough to a v4 uuid. So we just need the client to be able to get that id so it can do stuff with files after uploads have been completed. Our iOS engineer was eventually able to get it from the uploadLocationURL property of the TUSUpload instance, which works fine. It might be nice to have a more convenient way to do this-- like a "serverId" property or something-- but no big deal if not.

When I asked this question, though, we were definitely mixing up the "id" property on TUSUpload instances and the one used on the server side. That was the cause of the mismatch, so this is largely just a misunderstanding. Thanks for your help, though. :)

[Acconut]

@sripberger: The id the server uses is generated with this

Exactly, that's all correct from tusd's point of view :) I cannot comment on TUSKit's part though.

[andreaslindahl]

After the first file upload, I could not upload again and I found the reason for this.
upload.status = .ready
It should be done this way after every successful installation.

I also noticed this, and for now have to manually set status to .ready for each upload

[MMasterson]

@andreaslindahl I believe I had patched this in an updated version of the library, but maybe I am mistaken. Can you check what version of the library you're using?

[deeplinkage]

What if we have dynamic upload URLs? I see upload URL is set in config in AppDelegate, but what if actually I want to do this inside Controller after I get response from API which contains my upload URL.

[andreaslindahl]

What if we have dynamic upload URLs? I see upload URL is set in config in AppDelegate, but what if actually I want to do this inside Controller after I get response from API which contains my upload URL.

You can create the config anywhere and don't have to do it in your AppDelegate

[andreaslindahl]

@andreaslindahl I believe I had patched this in an updated version of the library, but maybe I am mistaken. Can you check what version of the library you're using?

I'm using version 2.1.5.alpha

[deeplinkage]

Hi @andreaslindahl, I understand that but...

TUSClient.shared.delegate = self

This gets called on the initial ViewController, and inside TUSClient:

guard let config = TUSClient.config else {
            fatalError("Error - you must call setup before accessing TUSClient")
        }

Here we error out because config is not yet set.

[MMasterson]

@deeplinkage @andreaslindahl, I recommend putting the config in the app delegate. You'll need to set a URL there, however, you can change the upload URL at anytime using TUSClient.shared.uploadURL = URL(string: "")

For the next release we could add a better solution, such as adding property to the TUSUpload class that overrides the uploadURL that is currently set on the TUSKit singleton