Pinning content #1509

jmozah · 2019-06-23T08:53:48Z

Useful Pinning Links:

ethersphere/user-stories#10
https://swarmresear.ch/t/pinned-content-in-swarm/18
#1274

Pin during upload
- swarm up --pin )
- Add a pin header in the http API
- If header present, set the pin Counter to 1 for the file & the manifest in the retrievalDataIndex
- Store the top level manifest/file hash in the pinIndex
Pin after upload
- swarm pin <top level manifest/file hash>
- Increase the pin counter for all the chunks in the merkle tree
Take care of different types of upload
- Tar upload
- Raw upload
- Multipart upload
Unpin pinned content
- swarm unpin <top level manifest/file hash>
- decrement the pinCounter by 1 for the entire merkle tree
List pinned files
- display all the top level hashes that are pinned
- Also display their pin counters and size
During GC, don't collect pinned chunks

acud

@jmozah a few notes about this PR:

the delta is huge (61 files so far). we'd have to break this down to smaller units
there's no test coverage whatsoever for the new functionality
the necessity of pinCounter (which is always set to 1 in the api) is probably arguable
pinCounter definitely should not be uint8. Imagine that within a certain pin you limit yourself to 255 instances of the same chunk, which is very low (just imagine all of the .DS_Store files and other possible examples of just 32 byte repetitions)
i'm not sure i understand why we would need to store information about whether to traverse a manifest as part of the localstore index. it seems like to much of high-level information to store somewhere
i think that a good next step would be to discuss the correct test vectors for this functionality and proceed from there

api/api.go

storage/localstore/localstore.go

…at/pin

…o feat/pin

…at/pin

api/client/client_test.go

state/dbstore.go

storage/localstore/gc.go

storage/pin/pin.go

janos · 2019-08-02T11:17:53Z

storage/pin/pin.go

+//     1) Whether the file is a RAW file or not
+//     2) Size of the pinned file or collection
+//     3) the number of times that particular file or collection is pinned.
+func (p *API) ListPinFiles() (map[string]FileInfo, error) {


If this function is used only in tests, maybe not to export it?

No.. This is exposed outside too. This is the only way through which the node owner will know what files are pinned , their size and their type.

OK. In general, exposing such listing functionality should be done with some form of pagination, to prevent very large responses. Maybe that can be considered later if needed.

Is there any other command in swarm done with pagination? Just to see and understand. I agree that we should do a pagination... this will be especially useful for dapps which are pinning and pinning service operators like us. But as you said.. we should take it later.. I want to push this core functionality first.

I do not think that there is in classical way. But manifests implement a tree that can be considered as a way not to represent all available data at once.

ok. I get it.

janos

👏 @jmozah

zelig

well done. this PR defo holds the record for most comments ever :)

storage/localstore/gc.go

jmozah · 2019-08-05T09:31:39Z

@acud @zelig @janos I have finished all your comment and multiple re-comments. I hope i have taken care of all the issues you have raised. Thank you for the review. Please approve and merge this PR if you are okay.

janos

I thought that I have approved it, but I've only submitted a comment. Sorry, my last comment was intended to be an approval.

acud · 2019-08-07T11:38:32Z

@jmozah thanks for your infinite patience and endless iterations on this. this is commendable and admirable. and once more thank you for the great work on this PR. it has made great distance and I think that the end-product is great.
bravo 🙏

jmozah · 2019-08-07T12:24:32Z

@acud Thanks for your persistence and nit picking :-). The end result looks great because of you.

* 'master' of github.com:ethersphere/swarm: (54 commits) api, chunk, cmd, shed, storage: add support for pinning content (ethersphere#1509) docs/swarm-guide: cleanup (ethersphere#1620) travis: split jobs into different stages (ethersphere#1615) simulation: retry if we hit a collision on tcp/udp ports (ethersphere#1616) api, chunk: rename Tag.New to Tag.Create (ethersphere#1614) pss: instrumentation and refactor (ethersphere#1580) api, cmd, network: add --disable-auto-connect flag (ethersphere#1576) changelog: fix typo (ethersphere#1605) version: update to v0.4.4 unstable (ethersphere#1603) swarm: release v0.4.3 (ethersphere#1602) network/retrieve: add bzz-retrieve protocol (ethersphere#1589) PoC: Network simulation framework (ethersphere#1555) network: structured output for kademlia table (ethersphere#1586) client: add bzz client, update smoke tests (ethersphere#1582) swarm-smoke: fix check max prox hosts for pull/push sync modes (ethersphere#1578) cmd/swarm: allow using a network interface by name for nat purposes (ethersphere#1557) pss: disable TestForwardBasic (ethersphere#1544) api, network: count chunk deliveries per peer (ethersphere#1534) network/newstream: new stream! protocol base implementation (ethersphere#1500) swarm: fix bzz_info.port when using dynamic port allocation (ethersphere#1537) ...

jmozah added 13 commits June 23, 2019 13:58

storage,shed,api,cmd: Pin with upload

e9d6d6c

Moved pinning to a seperate indexs instead of retrievalDataIndex

992a65a

GC now ignores pinned chunks

c0e3dd7

Added pin support for manifest and access commands

96f1555

Fix tests errors

9b61c0e

Fix lint errors

a60d601

Fix lint errors

52bf5c6

Fix address copy bug

179179e

Fix client api testcase

6c1bb08

Fix the 'could not add file in pinFilesIndex' yesycase failing

6ef0469

Unpin support, Display merkle tree from a root chunk

3ac561b

Fix lint errors

14b4acf

Fix more lint issues

a8f2f56

acud suggested changes Jul 1, 2019

View reviewed changes

acud assigned jmozah and acud Jul 1, 2019

acud requested review from zelig and janos July 3, 2019 06:40

jmozah added 11 commits July 10, 2019 13:52

Fix lint error

1cb08b3

Added testcases. Fixed bugs

6451240

Fix test case for multi manifest scenario

63dfa67

Add multipart upload support and testcase for that

bd22661

Added new mode for pinning and fixed test cases

1379bc1

Testcase refactoring to accomodate seperate pin mode

a99b873

Fix import cycles in testcase

c06c937

Added gdoc

b5519a5

Final touches

ed6609b

GC testcase fixes

ca3be88

Merge branch 'master' of https://github.com/ethersphere/swarm into fe…

aa18362

…at/pin

jmozah changed the title ~~[WIP] Pinning content~~ Pinning content Jul 16, 2019

jmozah requested a review from zelig August 1, 2019 20:44

jmozah added 3 commits August 2, 2019 02:34

spelling mistake

c450bd5

Merge branch 'master' of https://github.com/fairDataSociety/swarm int…

729c8f9

…o feat/pin

Merge branch 'master' of https://github.com/ethersphere/swarm into fe…

f90ab40

…at/pin

zelig previously approved these changes Aug 1, 2019

View reviewed changes

api/client/client_test.go Outdated Show resolved Hide resolved

Add comments removed while refactoring

3ff9e96

jmozah dismissed zelig’s stale review via 3ff9e96 August 2, 2019 06:16

janos reviewed Aug 2, 2019

View reviewed changes

Fixed final comments from Janos

90ebad7

janos previously approved these changes Aug 2, 2019

View reviewed changes

zelig previously approved these changes Aug 3, 2019

View reviewed changes

acud reviewed Aug 5, 2019

View reviewed changes

storage/localstore/gc.go Show resolved Hide resolved

Fixed few comment review from Elad

1ca63c6

jmozah dismissed stale reviews from zelig and janos via 1ca63c6 August 5, 2019 07:34

Changing pin header name and moving to http package

8762cb3

zelig previously approved these changes Aug 5, 2019

View reviewed changes

Changed the API input from string to []byte

1c775d4

jmozah dismissed zelig’s stale review via 1c775d4 August 5, 2019 08:24

Fixed errors while converting it to []bytes

410df80

janos approved these changes Aug 6, 2019

View reviewed changes

acud approved these changes Aug 7, 2019

View reviewed changes

acud merged commit b635061 into ethersphere:master Aug 7, 2019

skylenet added this to the 0.4.4 milestone Aug 9, 2019

zelig mentioned this pull request Aug 14, 2019

add documentation for pin API #1666

Closed

jmozah mentioned this pull request Jun 9, 2020

Local pinning fix ethersphere/bee#255

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pinning content #1509

Pinning content #1509

jmozah commented Jun 23, 2019 •

edited

Loading

acud left a comment

janos Aug 2, 2019

jmozah Aug 2, 2019

janos Aug 2, 2019

jmozah Aug 2, 2019 •

edited

Loading

janos Aug 2, 2019

jmozah Aug 2, 2019

janos left a comment

zelig left a comment

jmozah commented Aug 5, 2019

janos left a comment

acud commented Aug 7, 2019

jmozah commented Aug 7, 2019

Pinning content #1509

Pinning content #1509

Conversation

jmozah commented Jun 23, 2019 • edited Loading

Useful Pinning Links:

acud left a comment

Choose a reason for hiding this comment

janos Aug 2, 2019

Choose a reason for hiding this comment

jmozah Aug 2, 2019

Choose a reason for hiding this comment

janos Aug 2, 2019

Choose a reason for hiding this comment

jmozah Aug 2, 2019 • edited Loading

Choose a reason for hiding this comment

janos Aug 2, 2019

Choose a reason for hiding this comment

jmozah Aug 2, 2019

Choose a reason for hiding this comment

janos left a comment

Choose a reason for hiding this comment

zelig left a comment

Choose a reason for hiding this comment

jmozah commented Aug 5, 2019

janos left a comment

Choose a reason for hiding this comment

acud commented Aug 7, 2019

jmozah commented Aug 7, 2019

jmozah commented Jun 23, 2019 •

edited

Loading

jmozah Aug 2, 2019 •

edited

Loading