Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filestore cleanup is too slow #38

Closed
TheDiscordian opened this issue Mar 30, 2021 · 8 comments
Closed

Filestore cleanup is too slow #38

TheDiscordian opened this issue Mar 30, 2021 · 8 comments
Labels
performance Strictly performance related
Milestone

Comments

@TheDiscordian
Copy link
Owner

TheDiscordian commented Mar 30, 2021

Maybe we can speed it up by only removing offset 0 references, see if that fixes filestore errors just as well, and we can let the gc clean up the rest later.

@TheDiscordian TheDiscordian added the performance Strictly performance related label Mar 30, 2021
@TheDiscordian TheDiscordian added this to the v1.0.0 milestone Mar 30, 2021
@TheDiscordian

This comment has been minimized.

@TheDiscordian

This comment has been minimized.

@TheDiscordian

This comment has been minimized.

@TheDiscordian
Copy link
Owner Author

TheDiscordian commented Mar 31, 2021

We can't remove just offset 0, we must remove all the blocks, but we can just check offset 0, before we build the list to remove (1 bad block, then they're all bad blocks EDIT: This is incorrect, some blocks might be shared by other objects).

@TheDiscordian
Copy link
Owner Author

TheDiscordian commented Jun 2, 2021

filestore/verify can be asked to verify just a specific object. This will hopefully dump us back the blocks that are truly unfindable, and then we can safely remove those :D.

@TheDiscordian
Copy link
Owner Author

TheDiscordian commented Jul 16, 2021

Unfortunately it doesn't take an object, but actually a block, this is a shame. We can use refs to get all the CIDs, can't omit the irrelevant ones, but that's fine for now.

@TheDiscordian TheDiscordian pinned this issue Jul 23, 2021
@RomeSilvanus
Copy link

RomeSilvanus commented Jul 23, 2021

image
Please haste, codefriend

@TheDiscordian TheDiscordian modified the milestones: v1.0.0, v0.8.0 Jul 30, 2021
TheDiscordian added a commit that referenced this issue Oct 8, 2021
…formance increase. Fix for #38, now if a file fails to make it into MFS, it's nuked from the blockstore, then retried. Initial filestore scan made optional (default disabled) and now likely only occurs on failed pin updates (will need testing)
@TheDiscordian
Copy link
Owner Author

TheDiscordian commented Oct 8, 2021

Fixed in 78d6721

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Strictly performance related
Projects
None yet
Development

No branches or pull requests

2 participants