Amazon S3 tracker returns HTTP 400 for remaining byte estimates > 5GB

[jberkel]

If there's no metadata available, the remaining byte estimate is set to UINT64_MAX:

req->leftUntilComplete = tr_torrentHasMetadata(tor) ?
        tor->info.totalSize - tr_torrentHaveTotal(tor) 
        : ~(uint64_t)0;

(announcer.c)

Some trackers (e.g. the ones used by Amazon S3) return an error when the left= parameter is bigger than 5GB:

curl -v  'http://s3-tracker.eu-central-1.amazonaws.com:6969/announce?info_hash=%1fE%e5%8a%b1p5%7b%28e%f1%ba%1dv%90%f2m0%e5%9a&peer_id=-TR292Z-q2tk1lqg46m7&port=51413&uploaded=0&downloaded=0&left=18446744073709551615&numwant=80&compact=1&event=started'
*   Trying 54.239.54.85...
* TCP_NODELAY set
* Connected to s3-tracker.eu-central-1.amazonaws.com (54.239.54.85) port 6969 (#0)
> GET /announce?info_hash=%1fE%e5%8a%b1p5%7b%28e%f1%ba%1dv%90%f2m0%e5%9a&peer_id=-TR292Z-q2tk1lqg46m7&port=51413&uploaded=0&downloaded=0&left=18446744073709551615&numwant=80&key=2a201a75&compact=1&supportcrypto=1&event=started&ipv6=2001%3A4b98%3Adc0%3A41%3A216%3A3eff%3Afe39%3A30f HTTP/1.1
> Host: s3-tracker.eu-central-1.amazonaws.com:6969
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain;charset=ISO-8859-1
< Content-Length: 46
< Date: Fri, 21 Apr 2017 07:39:02 GMT
< Connection: close
< Server: AmazonS3
<
* Curl_http_done: called premature == 0
* Connection #0 to host s3-tracker.eu-central-1.amazonaws.com left intact
left(18446744073709551615) must be an integer.

[mikedld]

Isn't this a tracker issue then? How do they handle torrents greater than 4 GiB in size?

[jberkel]

Ah, it looks like a limitation on the S3 end:

Note
You can get torrent only for objects that are less than 5 GB in size.

http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html

Still, it would be great if transmission worked with these limited trackers.

[jberkel]

libtorrent-rasterbar sends -1 when it does not have any metadata:

https://github.com/arvidn/libtorrent/blob/f95f5b05c268b31c691e31b46b3f86e55f88f7cc/src/torrent.cpp#L3855

[mikedld]

Please check the latest nightly build to confirm.

[jberkel]

Looks like the value checked is actually a signed 64-bit integer, and -1 is not accepted :(

curl -v 'http://s3-tracker.eu-central-1.amazonaws.com:6969/announce?info_hash=Y%9f%85Mlf%ac%2fG%dc%1e%8c%cd%98%f2-%da.%f3%bf&peer_id=-TR292Z-beg8ugeou9dv&port=51413&uploaded=0&downloaded=0&numwant=80&key=8eaccab&compact=1&supportcrypto=1&left=-1&event=started&ipv6=2001%3A4b98%3Adc0%3A41%3A216%3A3eff%3Afe39%3A30f'
*   Trying 54.239.54.85...
* TCP_NODELAY set
* Connected to s3-tracker.eu-central-1.amazonaws.com (54.239.54.85) port 6969 (#0)
> GET /announce?info_hash=Y%9f%85Mlf%ac%2fG%dc%1e%8c%cd%98%f2-%da.%f3%bf&peer_id=-TR292Z-beg8ugeou9dv&port=51413&uploaded=0&downloaded=0&numwant=80&key=8eaccab&compact=1&supportcrypto=1&left=-1&event=started&ipv6=2001%3A4b98%3Adc0%3A41%3A216%3A3eff%3Afe39%3A30f HTTP/1.1
> Host: s3-tracker.eu-central-1.amazonaws.com:6969
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain;charset=ISO-8859-1
< Content-Length: 59
< Date: Sun, 30 Apr 2017 21:46:36 GMT
< nnCoection: close
< Server: AmazonS3
<
* Curl_http_done: called premature == 0
* Connection #0 to host s3-tracker.eu-central-1.amazonaws.com left intact
left(-1) was not in the valid range 0 - 9223372036854775807

[kholia]

Sending (1 << 63) - 1 or pow(2, 63) - 1) as the value of left seems to works fine with Amazon S3 tracker.

[mikedld]

Yeah, 9223372036854775807 is exactly 0x7fffffffffffffff = (1 << 63) - 1. So they're okay with the signed type it seems, unless the value it negative. Anyway, 9 EiB seems like a lot and is suitable as well.

[anacrolix]

I've just come across this while testing compatibility of anacrolix/torrent with other clients. I arrived at the same fix independently in anacrolix/torrent@27b7fbe. I can confirm that Transmission 2.94 doesn't currently work with S3 trackers.