Skip to content
This repository has been archived by the owner. It is now read-only.

Set a default max shard size at 4GiB #755

Merged
merged 4 commits into from Jan 10, 2018

Conversation

@braydonf
Copy link
Contributor

commented Jan 3, 2018

This is the largest that is supported on the client side, furthermore there are
larger issues that need to be addressed to be able to support shard sizes larger
than this to reduce the number of errors.

cc: @littleskunk @tempestb @aleitner

Goes along with pull requests:

Related issues:

braydonf added some commits Jan 3, 2018

Set a default max shard size at 4GiB
This is the largest that is supported on the client side, furthermore there are
larger issues that need to be addressed to be able to support shard sizes larger
than this to reduce the number of errors.
Fix runSpaceCheck
There was an issue with this method consuming lots of memory and CPU when it was run
over large data sets. This new method uses operating system features to determine if there
is space available on the disk. This is a quick fix to address major concerns until
we have better accounting of storage and bandwidth usage.

braydonf added some commits Jan 3, 2018

@littleskunk

This comment has been minimized.

Copy link
Member

commented Jan 3, 2018

I would expect out of memory exceptions caused by usedSpace calculation. We had this problem in the past: storj/storjshare-daemon#247 (comment)

I will test this with my ~4TB farmer.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

commented Jan 3, 2018

Yes, that's why I switched it from using du to diskusage.check

@littleskunk

This comment has been minimized.

Copy link
Member

commented Jan 3, 2018

I installed your branch but running free space check is missing on my logfile. Looks like still not called.

root@storj:~# npm list -g storj-lib
/usr/local/lib
└─┬ storjshare-daemon@5.3.0
  └── storj-lib@8.5.0  (github:braydonf/core#e7a3f2b8f2ffe1fa35cee49b7b055e7f00822345)
@braydonf

This comment has been minimized.

Copy link
Contributor Author

commented Jan 3, 2018

You may need to add farmer.runSpaceCheck() in scripts/farmer.js in storjshare-daemon.

@littleskunk

This comment has been minimized.

Copy link
Member

commented Jan 3, 2018

Looks better now. The check is running. I don't see any memory problems.

Please take a look at: #738

The SpaceCheck ignores my allocated size and is only looking on the total free space of my drive. That will not fix this issue.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

commented Jan 3, 2018

spaceAvailable should be set, we can test to confirm. The issue with ignoring the allocated size will need to be addressed in another PR (as that check is currently expensive to make).

@littleskunk

This comment has been minimized.

Copy link
Member

commented Jan 3, 2018

Yea that is correct. At the moment we only see the total used space of that drive and can't see how many of that are storj shards. That was the part with the out of memory exception we don't want to get again.

Ok #738 and storj/storjshare-daemon#307 can't be fix that easy.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

commented Jan 3, 2018

Yeah, they require some larger overhauling.

However immediately concerns should be addressed with storj/bridge#547 for storj/storjshare-daemon#307

@aleitner

This comment has been minimized.

Copy link
Contributor

commented Jan 3, 2018

I merged models. Now bridge needs to update the package json to use that

@braydonf

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2018

Models isn't part of this, but I think we're good on this one to merge.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2018

And then we can make a release at v8.6.0

@braydonf braydonf merged commit 9bdba6f into storj:master Jan 10, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
3 participants
You can’t perform that action at this time.