Companion: support sending custom headers to the upload destination

[SamuraiT]

Steps to reproduce

1. Set up the standalone Companion server for file handling.
2. Attempt to upload a file from a URL using the Uppy URL plugin.
3. Initiate the file upload process through the tus protocol.

Expected behavior

The file should be successfully uploaded to the target endpoint http://localhost:54321/storage/v1/upload/resumable without any authentication issues, assuming proper configuration and valid access tokens.

Actual behavior

During the upload process, an error occurs indicating a failure in the authentication step. The system specifically reports a "jwt must be provided" error, suggesting that the required JWT token for authorization is missing or not properly included in the request. This results in a 400 Bad Request response from the server, halting the upload process.

details

companion: 2024-02-21T06:50:30.254Z [error] uploader.tus.error DetailedError: tus: unexpected response while creating upload, originated from request (method: POST, url: http://localhost:54321/storage/v1/upload/resumable, response code: 400, response text: {"message":"jwt must be provided","statusCode":"400","error":"jwt must be provided"}, request id: 1cb24d28-986d-4b67-945f-fa059de3f404)
    at Upload._emitHttpError (/node_modules/tus-js-client/lib.es5/upload.js:872:23)
    at node_modules/tus-js-client/lib.es5/upload.js:993:18
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  originalRequest: Request {
    _method: 'POST',
    _url: 'http://localhost:54321/storage/v1/upload/resumable',
    _headers: {
      'Tus-Resumable': '1.0.0',
      'X-Request-ID': '1cb24d28-986d-4b67-945f-fa059de3f404',
      'Upload-Length': 98377,
      'Upload-Metadata': 'filename c3BlYWtlcl95dWluYV8wMi5tcDM=,filetype YXVkaW8vbXBlZw==,name c3BlYWtlcl95dWluYV8wMi5tcDM=,type YXVkaW8vbXBlZw=='
    },
    _request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: false,
      chunkedEncoding: false,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      _header: 'POST /storage/v1/upload/resumable HTTP/1.1\r\n' +
        'Tus-Resumable: 1.0.0\r\n' +
        'X-Request-ID: 1cb24d28-986d-4b67-945f-fa059de3f404\r\n' +
        'Upload-Length: 98377\r\n' +
        'Upload-Metadata: filename c3BlYWtlcl95dWluYV8wMi5tcDM=,filetype YXVkaW8vbXBlZw==,name c3BlYWtlcl95dWluYV8wMi5tcDM=,type YXVkaW8vbXBlZw==\r\n' +
        'Host: localhost:54321\r\n' +
        'Connection: keep-alive\r\n' +
        'Content-Length: 0\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/storage/v1/upload/resumable',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'localhost',
      protocol: 'http:',
      [Symbol(shapeMode)]: false,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kChunkedBuffer)]: [],
      [Symbol(kChunkedLength)]: 0,
      [Symbol(kSocket)]: [Socket],
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    _progressHandler: [Function (anonymous)],
    _requestOptions: {}
  },
  originalResponse: Response {
    _response: IncomingMessage {
      _events: [Object],
      _readableState: [ReadableState],
      _maxListeners: undefined,
      socket: null,
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 400,
      statusMessage: 'Bad Request',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [ClientRequest],
      _eventsCount: 2,
      [Symbol(shapeMode)]: true,
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 16,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0
    },
    _body: '{"message":"jwt must be provided","statusCode":"400","error":"jwt must be provided"}'
  },
  causingError: null
}
companion: 2024-02-21T06:50:30.256Z [debug] 2851d7ab cleanup
companion: 2024-02-21T06:50:30.256Z [error] 2851d7ab uploader.error DetailedError: tus: unexpected response while creating upload, originated from request (method: POST, url: http://localhost:54321/storage/v1/upload/resumable, response code: 400, response text: {"message":"jwt must be provided","statusCode":"400","error":"jwt must be provided"}, request id: 1cb24d28-986d-4b67-945f-fa059de3f404)
    at Upload._emitHttpError (/node_modules/tus-js-client/lib.es5/upload.js:872:23)
    at node_modules/tus-js-client/lib.es5/upload.js:993:18
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  causingError: null
}

[Murderlon]

Hi, unfortunately this has nothing to do with Uppy. Apparently you've setup some kind of authentication on your tus server and this is not the place for help with tus servers. There is also not enough information to help you. Companion doesn't expect the tus server to have auth so I recommend removing it and seeing if it works.

I know for a fact that things will work with tus-node-server and tusd

[SamuraiT]

Apparently you've setup some kind of authentication on your tus server and this is not the place for help with tus servers.

@Murderlon
thanks for replay. I'm utilizing a Supabase endpoint for data transmission and need to include an access token in the headers, similar to how one can configure S3 in Companion. Therefore, I believe this issue is related to Uppy itself, as Companion is also provided with Uppy.

[Murderlon]

As far as I know it's not possible to send extra headers to the upload destination from Companion, only to Companion. But I can see the value in that, with Supabase, but also for anyone who wants to send auth headers to their destination.

[SamuraiT]

Thank you, I appreciate it !
Yes, it's worth addressing, as I believe many Supabase users might encounter a similar issue.

[ikupenov]

Just ran into this myself 😕. I utilize RLS in Supabase so I need to be able to pass the JWT token. Are there any workarounds for this?

[Murderlon]

Contributing or forking is the only workaround for now.

[SamuraiT]

@Murderlon
Firstly, thank you for addressing this issue! You have been assigned, along with other team members, to address the issue. When do you estimate the feature solution for this problem will be completed?

[Murderlon]

We're very light on hours from core contributors lately. Currently Google Photos support is being developed and a massive TS migration, in preparation for #4764.

However since this is a small feature, I can probably take a look in the next two weeks.

The quikfix would be to allow a Record<string, string> to be send. But the proper fix would be some async function to generate a token and also support to refresh the token on 401. Both cases would only work when you integrate Companion manually into Express, not in standalone.

Thoughts @mifi? Here is a client-side example from Supabase.

[mifi]

Looking at the code I dont understand why the supabase client side example wouldn’t work. As far as i can see, when a Tus upload happens in companion, the Uppy Tus plugin headers option does get passed in the request body (json) to companion. Companion will then take these headers and pass them to tus-js-client. So I’m not sure what is the exact problem here

[SamuraiT]

However since this is a small feature, I can probably take a look in the next two weeks.

Awesome, thank you! I'll definitely give it a try once it's implemented

Companion will then take these headers and pass them to tus-js-client. So I’m not sure what is the exact problem here

I understand, but it seems it didn't work out for both @ikupenov and me

[ikupenov]

I was using standalone companion in Docker if that matters.

[mifi]

I just tested locally with yarn dev:with-companion:

uppyDashboard.use(Tus, { endpoint: 'http://127.0.0.1:8001', limit: 6, headers: { 'X-MyHeader': 'hello' } })

I then ran a fake Tus TCP server:

nc -v -l 8001

and I uploaded a single file from google drive, and I got this on my fake Tus TCP

POST / HTTP/1.1
Tus-Resumable: 1.0.0
X-MyHeader: hello
X-Request-ID: 3d4f91c7-5829-4199-a521-80f0b6fb3e78
Upload-Defer-Length: 1
Upload-Metadata: filename ...
Host: 127.0.0.1:8001
Connection: close
Content-Length: 0

as you can see the header is included, so it seems to be working. Need some reproducible code or more detailed steps to reproduce if you think it's still an issue.

[mifi]

Closing this because it's seemingly already implemented