Reliability Issues Overview

[braydonf]

Uploads

• Issues with timeouts for storage offers
  • People can write scripts that upload many tiny (and large) files at the same time that will cause network congestion for offers (DONE Rate limiter testing #410)
  • People with freeTier set to false do not have any rate limiting (DONE Rate limiter testing #410)
  • Current release of the GUI is unstable and may go offline (DONE [WIP] v5.0.0 storjshare-gui#483)
  • Sending of PUBLISH messages doesn't scale, need to make contracts in advance fill up smaller portions of the contract. Farmers would establish contract with the bridge as part of a startup. (See Publish/offer message propagation timeouts core#693)
  • Larger shards fail to upload to farmers (can be solved by partial data pushes for resuming) (???)
  • Larger shards don't get many offers, because of the max size of bucket in kfs at 1/256 of total available space in kfs
  • More nodes are needed to increase capacity and to make routing more efficient (Erasure encoding in increase the capacity of the network by being more efficient with mirroring. Efforts here should still be helpful though.)
    • Farmers are reluctant to join network because they want more reliable prediction for the return of investment (???)
• The original farmer that receives the data may disappear right after the data was uploaded (DONE Implement reed solomon encoding libstorj#274)
• Some transfers may be slow during initial upload (improve selection of offers to more reliable and faster nodes) (DONE Configure minimum speed threshold for transfers libstorj#312)
• Sometimes there are not enough mirrors created for a shard on upload, and there are isn't any process that will increase the mirror number Process that monitors established mirror counts per shard #383 (DONE not an issue w/ Implement reed solomon encoding libstorj#274)

Metadata

• Contracts will not be renewed after 3 months (DONE Contract Renewal #372)
• People can make a lot of requests to list bucket entries, create buckets, and other mongodb related queries that can prevent others from making such requests (DONE Rate limiter testing #410)
• Database connection can be closed, and it won't restart/reconnect Bridge does not properly reconnect to mongodb after no mongos proxy available error #327 (solved w/ health check endpoint)
• Health checks to use for load balancing (/health endpoint) Bridge should have a health check endpoint #329 (DONE Health check endpoint #411)
• Diversity of mirrors, some mirrors are being generated to the same nodeID, about ~10% of the mirrors are duplicates (Mirrors established to the same nodeID #391 Duplicate mirror traffic #348 Invalid mirror request from and to the same nodeID #429 Farmer online but only 1 mirror created #433) (DONE Fix issue with creating mirrors to the same nodeid #434)
• When a file is deleted the farmers are not contacted that the contract is invalid and can be deleted (??? - Improvement here may help with increasing the capacity of the network as data will be freed and available for use)
• Stability issues with storj-monitor that need to be addressed
  • Keep track of the number of off-line times Include offlineCount on contact #366 (???)
  • Keep track of the offline time of the monitor itself Monitor process state management #361 (???)
  • Need to replicate when complex/landlord/renter becomes aware that a farmer is offline too (???)
  • Decrease failures when replicating for destinations exhausted by getting new storage offers
• Landlord scaling between bridges, efficiency gains at the landlord and rabbitmq (better utilize features of rabbitmq) (???)
• Farmers are not being audited for malicious activity https://github.com/Storj/bridge/tree/feature/scheduleaudits, being online but not actually storing the data. (Will this be an issue if there is incentive to be online because of bandwidth payouts? May only be needed for files not accessed.) (???)

Downloads

• Timeouts for retrieval pointers
  • Less than five mirrors: degrades over time, ends up being zero, and a pointer can not be retrieved (DONE Implement reed solomon encoding libstorj#274)
  • No mirrors are established because the original farmers doesn't mirror or goes offline (DONE Implement reed solomon encoding libstorj#274)
• Shards are lost and need to be recovered via erasure encoding and added back into the network, and best attempts to mirror before the shard is lost.
• Tokens for farmers are not always used, we query for six but will only use one. This could be more efficient. Keep cache of retrieval pointers #375 (???)

Note: This will be edited to keep track of the current status.

[littleskunk]

More nodes are needed to increase capacity and to make routing more efficient

I don't agree on that. We don't need more farmer. The problem is that the bridge will only accept the first X OFFER and drop all other OFFER. The bridge prefers farmer with a low ping. Even if we increase the number of farmer the bridge will still prefer a tiny low ping group.

Lets use a simple example to explain it. 100 farmer close to the bridge (low ping) are sending OFFER. 500 farmer far away from the pridge (high ping) are sending OFFER as well but they are always to late and will never get a free slot on the mirror queue. This will work as long as the 100 farmer close to the bridge have free space to create all 5 mirrors. Once they are full the bridge will get a problem. 100 queue mirrors but not one of them has the free space to create a new mirror. At the end increased chance for lost shards.

At the same time the 100 low ping farmer have to manage high up and download traffic. At the end slow up and download speed while the resources of all other farmer are unused.

See #344

[44203]

More nodes are needed to increase capacity and to make routing more efficient

I don't agree on that. We don't need more farmer.

This is a general statement, not necessarily a bridge-specific problem. You might not agree, but I'd assert that the statement is correct. At the current network scale, without user rate-limiting, a single user can DoS the network by uploading thousands of small files in parallel. This is because network chatter is a function of demand for shard storage and not size of any given shard. The demand must be met with more participation (not capacity) to keep the network available for others. More nodes means that less of the network has to relay the contract publication, because the neighborhoods are denser.

In short, we always need more farmers - a bigger network increases both reliability and security.

[braydonf]

I ran into an issue today in a local small testing network with 4 farmers each with 100GB allocated, however had issues with uploading a 4GB file. It would seem that 4 farmers should be able to handle it, with all the mirrors it would still only be 24GB, more digging here may be needed.

[littleskunk]

@braydonf

It would seem that 4 farmers should be able to handle it

Each farmer will send one OFFER. That way you can get 1 shard + 3 mirrors. For all 5 mirrors you would need at least 6 farmer.

[44203]

@braydonf i think that's probably related to the default maxOfferConcurrency. with only 4 farmers, I'd imagine the rate at which the contracts go out could exceed this value and end up with farmers refusing to send an offer.

[braydonf]

Bumped up the maxOfferConcurrency to 16, decreased the mirror count to 0 in bridge, increased the farmers from 4 to 16, increased the available space for each from 5GB to 256GB and the issue went away, and offers were made again. It ended up being the last change to the allocated space that made the difference.

[littleskunk]

I uploaded a 64Byte file today. Only 1 mirror was created.

libstorj-1.0.0-rc2

info	
title	"Storj Bridge"
version	"5.12.1"
description	"Access the Storj network using a simple REST API."
x-protocol-version	"1.1.0"
x-core-version	"6.4.2"

storj --debug list-mirrors cb1d4eed725887b892cdf3d4 b16f24adb682540b8caafa49
Unlock passphrase:
Established
-----------
Shard: 0
Hash: 0c639ea9abfab009fb7f042e6c7d8c4bddc50108
        storj://realshels.hopto.org:4232/0c8941dc15c438f636648ebad26514b08113b5ac

Available
---------
Shard: 0

[braydonf]

Closing and breaking out into specific issues.