Never-ending upload

[livanh]

BUG REPORT

I use Sia for renting storage space. I upload a few files (500 to 1500 MB usually), wait for them to finish, and then I upload some more files.
Even when all the uploads are complete (including redundancy) and the GUI shows no active transfers, Sia continues to generate sustained upload traffic. It can be seen through a system monitoring application (conky in my case), and it's around 500/600 KB/s, probably bottlenecked by wi-fi. If I stop siad, that traffic drops to almost zero.

Stack Trace or error message
No relevant stack trace or error messages. Instead, I see that many files have silently grown much more than 3x redundancy:

Expected Behavior
Sia should boost all files to 3x redundancy and then stop uploading data

How to reproduce it (as minimally and precisely as possible)
Upload some files, wait for the upload to complete, and observe network traffic with a monitoring application

Environment

• Sia version: 1.3.1
• OS: Linux

[mtlynch]

I saw this as well during load testing.

How to reproduce it (as minimally and precisely as possible)

1. Set 5000 SC renter allowance
2. Upload ~48,000 1-byte files

Expected Behavior

/renter/files shows that uploaded files have redundancy of ~3.0x

Actual Behavior

/renter/files shows that uploaded files have the following redundancy:

• Mean: 5.2x
• Median: 5.3x
• Min: 3.0x
• Max: 11.3x

• As Spreadsheet
• Logs

Environment

• Sia version: 1.3.1
• OS: Win10

[ChrisSchinnerl]

@mtlynch Do you know how many active hosts were in your hostdb when you started uploading? Peviously (before 1.3.2 RC3) the /renter/files endpoint also counted files in contracts that are not good for renew towards redundancy while the repair code did not. That means you could see a redundancy of >3x while the file is still being repaired simply because some of your contracts won't be renewed and therefore that data has to be moved to new contracts.
Starting from 1.3.2 Sia will only count healthy renewable contracts towards file redundancy. That way we should never see a >3x redundancy for files uploaded with 1.3.2 or higher. If we continue to encounter this issue in the future, we might actually duplicate the file over more hosts than necessary.

[mtlynch]

My hostdb.log has these lines:

2018/02/14 19:11:41.968534 scan.go:339: Performing scan on 200 online hosts and 200 offline hosts.

Full: https://gist.github.com/mtlynch/7726468eca7ffef0ff6ac9370b05024e#file-hostdb-log

Is that an accurate measure?

Otherwise, I don't have the number of hosts.

[DavidVorick]

It's likely that we will still see files over 3.0x exactly due to hosts going offline, being repaired, and then those hosts coming back online.

Closing this issue for now, I believe we have solved the problem in the released v1.3.2. Please feel free to re-open if files uploaded after upgrading to v1.3.2 continue to have this issue.