This repository has been archived by the owner. It is now read-only.

Never-ending upload #2813

Closed
livanh opened this Issue Mar 3, 2018 · 4 comments

Comments

Projects
None yet
5 participants
@livanh

livanh commented Mar 3, 2018

BUG REPORT

I use Sia for renting storage space. I upload a few files (500 to 1500 MB usually), wait for them to finish, and then I upload some more files.
Even when all the uploads are complete (including redundancy) and the GUI shows no active transfers, Sia continues to generate sustained upload traffic. It can be seen through a system monitoring application (conky in my case), and it's around 500/600 KB/s, probably bottlenecked by wi-fi. If I stop siad, that traffic drops to almost zero.

Stack Trace or error message
No relevant stack trace or error messages. Instead, I see that many files have silently grown much more than 3x redundancy:
screen

Expected Behavior
Sia should boost all files to 3x redundancy and then stop uploading data

How to reproduce it (as minimally and precisely as possible)
Upload some files, wait for the upload to complete, and observe network traffic with a monitoring application

Environment

  • Sia version: 1.3.1
  • OS: Linux

@livanh livanh changed the title from Never-ending upload to Never-ending upload [bug] Mar 8, 2018

@livanh livanh changed the title from Never-ending upload [bug] to Never-ending upload Mar 8, 2018

@mtlynch

This comment has been minimized.

Contributor

mtlynch commented Mar 11, 2018

I saw this as well during load testing.

How to reproduce it (as minimally and precisely as possible)

  1. Set 5000 SC renter allowance
  2. Upload ~48,000 1-byte files

Expected Behavior

/renter/files shows that uploaded files have redundancy of ~3.0x

Actual Behavior

/renter/files shows that uploaded files have the following redundancy:

  • Mean: 5.2x
  • Median: 5.3x
  • Min: 3.0x
  • Max: 11.3x

image

Environment

  • Sia version: 1.3.1
  • OS: Win10

@tbenz9 tbenz9 added the Bug label Mar 11, 2018

@ChrisSchinnerl

This comment has been minimized.

Contributor

ChrisSchinnerl commented Mar 12, 2018

@mtlynch Do you know how many active hosts were in your hostdb when you started uploading? Peviously (before 1.3.2 RC3) the /renter/files endpoint also counted files in contracts that are not good for renew towards redundancy while the repair code did not. That means you could see a redundancy of >3x while the file is still being repaired simply because some of your contracts won't be renewed and therefore that data has to be moved to new contracts.
Starting from 1.3.2 Sia will only count healthy renewable contracts towards file redundancy. That way we should never see a >3x redundancy for files uploaded with 1.3.2 or higher. If we continue to encounter this issue in the future, we might actually duplicate the file over more hosts than necessary.

@mtlynch

This comment has been minimized.

Contributor

mtlynch commented Mar 12, 2018

My hostdb.log has these lines:

2018/02/14 19:11:41.968534 scan.go:339: Performing scan on 200 online hosts and 200 offline hosts.

Full: https://gist.github.com/mtlynch/7726468eca7ffef0ff6ac9370b05024e#file-hostdb-log

Is that an accurate measure?

Otherwise, I don't have the number of hosts.

@DavidVorick

This comment has been minimized.

Member

DavidVorick commented Mar 15, 2018

It's likely that we will still see files over 3.0x exactly due to hosts going offline, being repaired, and then those hosts coming back online.

Closing this issue for now, I believe we have solved the problem in the released v1.3.2. Please feel free to re-open if files uploaded after upgrading to v1.3.2 continue to have this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.