Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement status filtering in PinTracker/Status #1360

Closed
hsanjuan opened this issue May 20, 2021 · 0 comments · Fixed by #1377
Closed

Implement status filtering in PinTracker/Status #1360

hsanjuan opened this issue May 20, 2021 · 0 comments · Fixed by #1377
Assignees
@hsanjuan
Labels
effort/days Estimated to take multiple days, but less than a week exp/expert Having worked on the specific codebase is important P2 Medium: Good to have, but can wait until someone steps up status/ready Ready to be worked
Milestone
Release v0.14.0

Comments

@hsanjuan
Copy link
Collaborator

Currently, the REST API allows pin status filtering using TrackerStatus flags like:

https://github.com/ipfs/ipfs-cluster/blob/bfe179e94343321484f423d882b8770a6ecd4354/api/types.go#L84-L85

This happens on the API after the StatusAll RPC call to the pintracker has happened.

The StatusAll implementation lists the local pinset, calls pin ls --type recursive on IPFS to tell if things in the local pinset are pinned or not, and finally incorporates ongoing operations to the response.

For very big pinsets, the pin ls call is very expensive and, in particular, would not be necessary if we are filtering for things like queued or pinning, which affect ongoing operations only so the information could be obtained directly from the operation tracker.

Thus, doing the filtering directly on the pintracker would allow for big improvements for things like "show me all queued items" or "show me all items that are pinning".
@hsanjuan hsanjuan added status/ready Ready to be worked P2 Medium: Good to have, but can wait until someone steps up exp/expert Having worked on the specific codebase is important effort/days Estimated to take multiple days, but less than a week labels May 20, 2021
hsanjuan added a commit that referenced this issue Jul 6, 2021
@hsanjuan
Fix #1360: Efficient pinset status with filters
343c943 
This commit modifies the pintracker StatusAll call to take a status filter.

This allows to skip a PinLs call to ipfs when checking status for items that
are queued, pinning, unpinning or in error. Those status come directly from
the operation tracker. This should result in a significant performance
increase for those calls, particularly in nodes with several hundred thousand
pins and more, where the call to IPFS is very expensive.

A new TrackerStatusUnexpectedlyUnpinned status has been introduce to
differentiate between pin errors (tracked by the operation tracker) and "lost"
items (which before were pin errors too). This new status is handled by the
Recover() operation as before.
@hsanjuan hsanjuan mentioned this issue Jul 6, 2021
Merged
@hsanjuan hsanjuan added this to the Release v0.13.4 milestone Jul 6, 2021
@hsanjuan hsanjuan self-assigned this Jul 6, 2021
hsanjuan added a commit that referenced this issue Jul 6, 2021
@hsanjuan
Fix #1360: Efficient pinset status with filters
1393eeb 
This commit modifies the pintracker StatusAll call to take a status filter.

This allows to skip a PinLs call to ipfs when checking status for items that
are queued, pinning, unpinning or in error. Those status come directly from
the operation tracker. This should result in a significant performance
increase for those calls, particularly in nodes with several hundred thousand
pins and more, where the call to IPFS is very expensive.

A new TrackerStatusUnexpectedlyUnpinned status has been introduce to
differentiate between pin errors (tracked by the operation tracker) and "lost"
items (which before were pin errors too). This new status is handled by the
Recover() operation as before.
hsanjuan added a commit that referenced this issue Jul 6, 2021
@hsanjuan
Fix #1360: Efficient pinset status with filters
edfcfa3 
This commit modifies the pintracker StatusAll call to take a status filter.

This allows to skip a PinLs call to ipfs when checking status for items that
are queued, pinning, unpinning or in error. Those status come directly from
the operation tracker. This should result in a significant performance
increase for those calls, particularly in nodes with several hundred thousand
pins and more, where the call to IPFS is very expensive.

A new TrackerStatusUnexpectedlyUnpinned status has been introduce to
differentiate between pin errors (tracked by the operation tracker) and "lost"
items (which before were pin errors too). This new status is handled by the
Recover() operation as before.
hsanjuan added a commit that referenced this issue Jul 6, 2021
@hsanjuan
Merge pull request #1377 from ipfs/fix/1360-efficient-pin-status
54c3608 
Fix #1360: Efficient pinset status with filters
hsanjuan added a commit that referenced this issue Jul 7, 2021
@hsanjuan
pintracker: avoid listing the state unless necessary
27569bd 
This is a follow up to #1360 which further optimizes StatusAll calls by
avoiding listing and filtering the cluster state when requesting status for
operations that should be direclty in the operation tracker because they are
ongoing (queued, pinning, unpinning, error).
@hsanjuan hsanjuan mentioned this issue Jul 7, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hsanjuan hsanjuan

Labels
effort/days Estimated to take multiple days, but less than a week exp/expert Having worked on the specific codebase is important P2 Medium: Good to have, but can wait until someone steps up status/ready Ready to be worked
Projects
None yet
Milestone
Release v0.14.0
Development

Successfully merging a pull request may close this issue.

Fix #1360: Efficient pinset status with filters ipfs-cluster/ipfs-cluster
1 participant
@hsanjuan