Experiencing issues with Tus

[paulm17]

Hi.

To begin with, thanks for your works on the library, examples and documentation. Thanks to those, I was able to duplicate an existing uppy setup and get it partially working without too much stress 😓.

I'm using the tus supplied by https://github.com/tus/tusd.

The reason for the migration is that it's (uppy lib) huge and uploadly is much more efficient. I would really like to get a working solution. I've created a discussion rather than an issue in case I have made some errors or misunderstanding tus usage.

To get a working solution, I've tried using these two as the provider:

const tusEnhancer = getTusEnhancer({
    parallel: 1,
  });

return (
    <Uploady
      autoUpload={false}
      noPortal={true}
      destination={{
        url: `${process.env.NEXT_PUBLIC_DAM_STORAGE_UPLOAD_URL}/upload`,
      }}
      enhancer={tusEnhancer}
    >
      {children}
    </Uploady>
  );

and

<TusUploady
      debug
      noPortal={true}
      destination={{
        url: `${process.env.NEXT_PUBLIC_DAM_STORAGE_UPLOAD_URL}/upload`,
      }}
      chunkSize={2142880}
      sendDataOnCreate
    >
      {children}
    </TusUploady>

and then

const { upload } = useUploady();

as I'm using my own UI and don't need anything else 😄.

When I use the first option. I see 404 on the HEAD request. If I have the parallel greater than 1, I can see many more 404 instances.

When I increase parallel from 1 to another number. The process does not finish. Nor do I get 1 file. I see a bunch of chunked files.

Intermittently I see a range of errors and if I try another upload.

Sometimes if I look at the TUS console:

2023-09-15 12:13:36 [tusd] 2023/09/15 11:13:36.786623 event="ResponseOutgoing" status="400" method="POST" path="" error="one of the partial uploads is not finished" requestId=""

Additionally, when I a file with latin characters, I sometimes get the following error

Uncaught (in promise) DOMException: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.

---

When I use the 2nd option.

Uncaught (in promise) ChunkedSendError: start byte 8746 is invalid. File size: 8746

And I can't even start the upload process.

---

I've tried creating a sandbox also, but I keep on getting

Identifier 'process' has already been declared

If you want to duplicate my works, You can use the uppy supplied TUS endpoint: https://tusd.tusdemo.net/files/

---

Btw, this is uppy in action. It's able to take a 100mb file and using TUS chunk it in and then create the file at the end. Hopefully this is the right terminology.

https://streamable.com/2cdxe3

I would very much like to replicate this with react-uploadly!

Thanks! 😄

[yoavniran]

hey @paulm17 thanks!

Strange that you're facing an issue with uploading. I tested thoroughly with tusd while building this part of Uploady and it worked well. I will have to check again.

Almost missed the issue about latin chars, that sounds like a bug on its own. Please open an issue for that and give examples of the names you've seen fail.

Will try to get to this in the next few days.

[yoavniran]

I just downloaded the latest tusd binary and ran it locally (on mac).
I was able to upload a 134 MB file successfully

[tusd] 2023/09/17 10:50:12.187903 event="ResponseOutgoing" status="204" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.196285 event="RequestIncoming" method="OPTIONS" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.196327 event="ResponseOutgoing" status="200" method="OPTIONS" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.198094 event="RequestIncoming" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.199453 event="ChunkWriteStart" id="11b8b2ec54ea6f0bcb13151872f1f986" maxSize="524288" offset="130023424"
[tusd] 2023/09/17 10:50:12.201646 event="ChunkWriteComplete" id="11b8b2ec54ea6f0bcb13151872f1f986" bytesWritten="524288"
[tusd] 2023/09/17 10:50:12.201686 event="ResponseOutgoing" status="204" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.208599 event="RequestIncoming" method="OPTIONS" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.208635 event="ResponseOutgoing" status="200" method="OPTIONS" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.210330 event="RequestIncoming" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.211525 event="ChunkWriteStart" id="11b8b2ec54ea6f0bcb13151872f1f986" maxSize="524288" offset="130547712"
[tusd] 2023/09/17 10:50:12.212110 event="ChunkWriteComplete" id="11b8b2ec54ea6f0bcb13151872f1f986" bytesWritten="524288"
[tusd] 2023/09/17 10:50:12.212129 event="ResponseOutgoing" status="204" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.223451 event="RequestIncoming" method="OPTIONS" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.223486 event="ResponseOutgoing" status="200" method="OPTIONS" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.225560 event="RequestIncoming" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.244811 event="ChunkWriteStart" id="11b8b2ec54ea6f0bcb13151872f1f986" maxSize="524288" offset="131072000"
[tusd] 2023/09/17 10:50:12.245799 event="ChunkWriteComplete" id="11b8b2ec54ea6f0bcb13151872f1f986" bytesWritten="524288"
[tusd] 2023/09/17 10:50:12.245833 event="ResponseOutgoing" status="204" method="PATCH" path="11b8b2ec54ea6f0bcb13151872f1f986" requestId=""
[tusd] 2023/09/17 10:50:12.402427 event="UploadFinished" id="11b8b2ec54ea6f0bcb13151872f1f986" size="133999884"

I run tusd like this:

./tusd -base-path /upload -upload-dir tus-files -port 9009

important here is the -base-path param. I raise this as I see you also used the /upload path in your code but this isn't the default so you have to set it.

BTW, I definitely recommend using the TusUploady provider rather than setting the enhancer yourself.

[paulm17]

Thanks for taking a look so quickly!

and ran it locally (on mac).

Now I see. I'm running on a docker instance.

tusd:
    image: tusproject/tusd:latest
    container_name: tusd
    command: -upload-dir=/opt -base-path=/upload --hooks-http http://host.docker.internal:3000/api/dam/uploadHook
    restart: always
    ports:
      - 9450:1080
    volumes:
      - /Users/paul/development/src/media_server/public/temp/fileuploads:/opt

I wonder if this is the issue? I will investigate tomorrow and will respond when I find something.

BTW, I definitely recommend using the TusUploady provider rather than setting the enhancer yourself.

Thanks. I'll do that.

[paulm17]

Could you paste your TusUploady provider?

Also found the following: https://github.com/tus/tusd/blob/main/docs/locks.md

tusd does not support concurrent requests to the same upload resource

Shall I presume that parallel must be 1 then?

[yoavniran]

So it's a little confusing. I'll try and help sort it out:

1. Uploady has a grouped option. Meaning, uploading multiple files in the same request. This isn't supported at all by TUS and if is turned on, the tus-sender will not handle the upload, but delegate down to the chunked-sender / xhr-sender. I assume you're not using this, but want to be thorough.

2. Uploady has a parallel option. Outside of Tus, it will result in multiple upload requests (with 1 or more files, depending on the grouped option) being sent in parallel. For Tus, this option is treated differently as you're seeing. It results in multiple chunks being sent concurrently. However, based on the Tus protocol, parallel upload requires the use of the Concatenation extension.

I'm not sure how the tusd implementation handles this. Perhaps you need to turn on the extension to get it to work. If not, turning parallel off should also help.

This is the implementation I'm using for testing: https://github.com/rpldy/react-uploady/blob/master/packages/ui/tus-uploady/TusUploady.stories.js#L81

[paulm17]

Turns out, it was due to an older issue of TUS server. Once I had removed the docker version and re-installed. I was able to use the TusUploady provider (commenting out sendDataOnCreate) and upload fine.

There is an issue though: I intermittently get a 404 on the head. It's the same with a locally installed version as I was able to get that working as well.

[paulm17]

I was using 1.4.1. Upgraded to 1.5.0 and the header issue goes away.

[paulm17]

Marking this complete as all issues solved. 😓

[yoavniran]

glad to hear!
a bit curious though, as im not aware of any relevant changes between these versions, but not gonna complain 😄