Taildrop: fails when sending many files from macOS

[dsnet]

What is the issue?

I'm sending from iOS (1.30.2) to a Windows (1.30.1) machine.

On the Windows machine, I see:

2022-10-07 17:54:41.9334572 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:41.9334572 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:41.9334572 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:41.935013 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:41.9373942 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:41.9560422 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:42.3082423 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:42.3154549 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:42.3278697 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:42.5259584 -0700 PDT: peerapi: put Copy error: unexpected EOF
2022-10-07 17:54:42.5514431 -0700 PDT: peerapi: put Copy error: unexpected EOF

On the iOS device, I see:

2022-10-08 00:54:41.099811 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59682->100.107.177.2:33542: write tcp 100.76.79.70:59682->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109393 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59789->100.107.177.2:33542: write tcp 100.76.79.70:59789->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109466 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59791->100.107.177.2:33542: write tcp 100.76.79.70:59791->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109519 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59790->100.107.177.2:33542: write tcp 100.76.79.70:59790->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109614 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59792->100.107.177.2:33542: write tcp 100.76.79.70:59792->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109695 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59793->100.107.177.2:33542: write tcp 100.76.79.70:59793->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109772 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59795->100.107.177.2:33542: write tcp 100.76.79.70:59795->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109839 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59794->100.107.177.2:33542: write tcp 100.76.79.70:59794->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.109919 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59796->100.107.177.2:33542: write tcp 100.76.79.70:59796->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.11153 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59688->100.107.177.2:33542: write tcp 100.76.79.70:59688->100.107.177.2:33542: write: no buffer space available
2022-10-08 00:54:41.111952 +0000 UTC: http: proxy error: readfrom tcp 100.76.79.70:59862->100.107.177.2:33542: write tcp 100.76.79.70:59862->100.107.177.2:33542: write: no buffer space available

It appears that we are creating a new TCP connection for every file upload? Perhaps we should switch to a multipart HTTP POST?

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

No response

OS version

No response

Tailscale version

No response

Bug report

No response

[bradfitz]

Even without HTTP/2, HTTP/1 can re-use a TCP connection between multiple requests.

@soniaappasamy, @nickoneill, @mihaip: do you know if the iOS/macOS client has a queue of files to send (like Windows and Linux) or does it start all of them concurrently?

[soniaappasamy]

I haven't looked at that code in a while, but I think it queues them up and sends individually. And looking at it now, it creates a new URLSession for each send, which may not be necessary?

https://github.com/tailscale/corp/blob/main/xcode/Shared/LocalAPIClient.swift#L141

[mihaip]

@dsnet how many files were you trying to send (and from which app)?

@soniaappasamy that's for the LocalAPI call, which should be ~instant. Once we end up in that handler, we use the Go HTTP stack to make the PeerAPI call with the actual files, so it should be the same on all platforms.

Though based on the fact that we make a separate URLSession for the local API call, it does mean that each gets its own dispatch queue, so they will all run at the same time (for all files). @bradfitz were you suggesting that we have a queue like on Windows?

[dsnet]

I initiated a transfer of 200 photos from the photos application on iOS.

we use the Go HTTP stack to make the PeerAPI call with the actual files, so it should be the same on all platforms.

Perhaps we should have a global semaphore on the maximum number of parallel POSTs?

[bradfitz]

I'm just saying that trying to upload too many/big files concurrently is worse than doing a smaller number at a time. They'll be competing for bandwidth and we don't have any resume-from-position mechanism on any failure.

[bradfitz]

Perhaps we should have a global semaphore on the maximum number of parallel POSTs?

But we'd still then have 200 from iOS to LocalAPI, even if we had a semaphore from Go (LocalBackend) to PeerAPI elsewhere.

We could do that, but I'd rather put the semaphore on the Swift side.

[dsnet]

It seems unfortunate putting a semaphore on the Swift side, as we would presumably want a semaphore for other platforms (e.g., Android), right? Having a semaphore on the Go side solves the issue for multiple platforms. Could the Go side put backpressure and block Swift calls instead?

[mihaip]

Using a shared URLSession is a cheap/easy way to impose a maximum amount of parallelism from the Swift side, will send out a PR with that (https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1407597-httpmaximumconnectionsperhost ends up kicking in, which limits it to 6 requests)