feat: pin

[AdamStone]

• core
  • add
  • rm
  • ls
  • isPinned
  • isPinnedWithType
  • directKeys / recursiveKeys / internalKeys
  • flush (persist keys to datastore)
  • load (restore keys from datastore)
• cli
  • add
  • rm
  • ls
• http
  • add
  • rm
  • ls

Updated core interface to align with interface-ipfs-core, still need to update cli and http.

[daviddias]

@AdamStone, js-ipfs has gone through a good amount of updates, including bug fixes and new features. Would you like to revisit pinning again? rebase master on your branch and continue the PR

[AdamStone]

Sure, I guess I should have been rebasing rather than merging to begin with. I think the core implementation is basically done, so I'll update the description. The failing test should pass once this issue is fixed.

[dignifiedquire]

Are we sure about the licence for this module, can't find info on npm nor the repo.

[AdamStone]

Ah, good point. I'll swap in something with MIT license.

[dignifiedquire]

thanks :)

[daviddias]

This is looking good @AdamStone :)

Would you also like to lead the ipfs-inactive/interface-js-ipfs-core#34, so that we have the same battery of tests for js-ipfs and js-ipfs-api for the pinning API.

Also, go-ipfs stores the root hash of the pinset inside level-db (datastore). Currently, js-ipfs doesn't use leveldb (datastore) because it is not implemented in ipfs-repo, would you like to tackle that as well -> ipfs/js-ipfs-repo#13

[AdamStone]

@diasdavid Sorry for the delayed response, I see you've already started on ipfs/interface-ipfs-core#34 but I can continue with that if you like. Am I understanding correctly that it basically just involves migrating the core pinning tests from js-ipfs to interface-ipfs-core?

For the datastore, I did notice the discrepancy between js-ipfs and go-ipfs when working out how to store the pin set and ended up just putting it in the block store because I didn't see a good alternative. I just read through ipfs/js-ipfs-repo#13 but found it pretty confusing at first pass. I think I'd better do a little more research before committing to take that on.

[daviddias]

@AdamStone sounds good. Let me know if you would like to go through it on IRC or a hangout, you can also join today's js-ipfs call https://github.com/ipfs/pm

[daviddias]

@AdamStone good news, we've finished the Pin interface in interface-ipfs-core, so now in order to validate this Pin implementation, you just need to make sure it passes those tests

• https://github.com/ipfs/interface-ipfs-core/tree/master/API/pin
• https://github.com/ipfs/interface-ipfs-core/blob/master/src/pin.js

[AdamStone]

Sounds good, I'll take a look tonight.

[AdamStone]

@diasdavid Can you clarify, when pin.add succeeds and returns "an array of objects that represent the files that were pinned", should this include indirectly-pinned objects when the add is recursive?

[daviddias]

Since there is the type "recursive pin", we only need to know that the head was recursively pin. So no, we don't need to return them all.

[AdamStone]

@diasdavid In that case, are there any circumstances where this should return more than just one object, since add expects a single hash? Or does that mean it should return the full pin set, not just the new addition?

[daviddias]

The best is to see how go-ipfs does it through the CLI and match the expectations as closely as possible.

[AdamStone]

@diasdavid The go-ifps cli seems pretty flexible. You can pass one or more b58 hashes or ipfs paths like /ipfs/b58hash/link. For both add and rm it prints only the hashes that were changed by that operation, along with the type. I had tried to make the js pin CLI work with an ipfs path and behave like the go CLI, but I didn't notice you could pass more than one item. So I've adjusted the pin core methods to be more consistent with the CLI and to accept one hash/path or an array of them. Not sure if we want to test for those scenarios as well.

I found I needed to make a few modifications to the tests, so I opened a PR in interface-ipfs-core.

[loong]

Any updates on pinning?

[AdamStone]

@diasdavid After forking the latest js-ipfs-api, its tests seem to bypass my link to interface-ipfs-core. That is to say, when I do npm test it's not running against my changes there, even though when I check node_modules I see the interface-ipfs-core folder does link to my local repo. Actually even if I delete interface-ipfs-core entirely from node_modules, it doesn't seem to matter, it still runs the same tests. Another case of the test architecture changing underneath me? Any ideas?

[daviddias]

All these years using npm have taught me that when in doubt, start from the top.

> rm -r all the node_modules
> rm all the package-lock.json if you have any (in case of using npm 5)
> cd interface-ipfs-core
> npm install && npm link
> cd js-ipfs-api
> npm install && npm link interface-ipfs-core
> npm test # tests should pass
> cd js-ipfs
> npm install && npm link interface-ipfs-core && npm link ipfs-api
> npm test # tests should pass

Let me know if this works for you.

[AdamStone]

Still not sure why, but even with those steps I just couldn't get the command-line linking to get all the links right. Eventually I just went through and copied the links manually into the node_modules folders, and that seems to work. Although the js-ipfs-api tests are still running a bunch of other tests in addition to what I mark with describe.only. Still need to make some of the other changes, then I'll push an update.

[AdamStone]

Migrating the go test repo from v5 to v6 breaks http tests where something expects to find v5. I couldn't tell from this stack trace where that expectation is coming from.

[daviddias]

It was announced yesterday on all hands that the IPFS Repo had been updated. It was an update also for me.

That said, the file structure didn't change. You just need to update the test to look for v6 :)

[AdamStone]

@diasdavid I rebased against master just now, but I still see the go test repo as version 5. I also haven't been able to find where in the tests or configuration it's deciding that it expects to find a particular version. So I'm not seeing what it would mean exactly to update the test to look for v6?

[AdamStone]

I rebased what I have so far against current master here and in these related PRs for interface-ipfs-core and js-ifps-api. With these repos linked locally and with the changes in these PRs, all the pin tests are locally passing for me. However, there are placeholder workarounds in the code for two main blocking issues:

1. The pin datastore key is /local/pins/js because using the key from go-ifps '/local/pins causes the tests to try to load pins from go-ipfs-repo, which doesn't work. Still unclear if the repo just needs to be migrated, since trying to do so causes version mismatch errors.
2. The datastore is closed during the CLI core tests when trying to save and load pins due to some timing problem - the tests only pass if I manually open the datastore within the pin code.

@diasdavid I hate to leave this PR hanging again after getting this far, I was hoping this would be the final push to get all this wrapped up while I had the time to spend on it, but sorry to say I'm basically out of time now and I don't expect I'll be able to take this any further. Hopefully someone else can find it a useful starting point.