Implement ipfs refs #2004

dirkmc · 2019-04-22T10:49:48Z

Implement the ipfs refs command:

USAGE
  ipfs refs <ipfs-path>... - List links (references) from an object.

SYNOPSIS
  ipfs refs [--format=<format>] [--edges | -e] [--unique | -u] [--recursive | -r] [--max-depth=<max-depth>] [--] <ipfs-path>...

implement unique flag
implement http-api

Depends on ipfs-inactive/js-ipfs-http-client#978

src/cli/commands/refs.js

src/core/components/files-regular/refs-pull-stream.js

dirkmc · 2019-04-24T12:32:40Z

After talking to @achingbrain I will take a look at making a change to js-ipfs-unixfs-exporter to include the parent CID so that a call to refs can be streamed (instead of collecting all the results, working out each link's parents and only then outputting a result)

alanshaw · 2019-04-24T13:56:40Z

Is refs only for unixfs nodes or should it work with all IPLD nodes? I was originally thinking you’d be talking directly to IPLD for this...

dirkmc · 2019-04-25T00:40:53Z

hmm that's a good question. I'll take a look at exactly what refs does and whether it makes sense to use unixfs exporter

dirkmc · 2019-04-25T04:21:10Z

Looking at the code it seems like js-ipfs-unixfs-exporter does traverse any kind of IPLD object, but only in order to find unix format ipld objects. I will want to do something very similar, but more generic, ie not just for unix format objects.

dirkmc · 2019-04-28T15:13:45Z

It turns out it's actually simple to traverse the refs DAG in js-ipfs with pull-traverse instead of relying on js-ipfs-unix-exporter, which is specifically aimed at unix format files

alanshaw

🚀

alanshaw · 2019-05-01T15:07:27Z

src/cli/commands/refs-local.js

+const { print } = require('../utils')
+
+module.exports = {
+  command: 'refs-local',


We should be able to use this as ipfs refs local i.e. without the hyphen

@alanshaw it seems like it will be tricky to make the parser work with both
ipfs refs <path>
ipfs refs local

I can probably hack it to make it work, but before I do that do you think it would make more sense to the user to give the commands different names instead?

Lets keep compat with go-ipfs - it's not too hacky to detect when path === 'local' and do the right thing in the refs handler.

That part is easy, it's making the command line parser / help text auto-formatter understand it that will be a little trickier, I'll see if I can make it work

Good point, please take a look but if it's going to be a big time sink or over complicated then lets forget it.

I added a command aliaser at the beginning of parsing (unfortunately it's not possible to do with yargs middleware): https://github.com/ipfs/js-ipfs/pull/2004/files#diff-71bd1bde77e38a68f76195b8f85ce19c

src/cli/commands/refs-local.js

src/cli/commands/refs.js

src/http/api/resources/files-regular.js

src/http/api/routes/files-regular.js

test/utils/ipfs-exec.js

src/core/components/files-regular/refs-pull-stream.js

src/core/components/object.js

alanshaw · 2019-05-13T12:23:24Z

@dirkmc I've released ipfs-http-client@31 and updated the dependency version here. What's left to do on this PR? I'd like to get it merged ASAP 😁

dirkmc · 2019-05-13T12:26:29Z

@alanshaw I think it's good to go 🚀

dirkmc · 2019-05-13T12:28:41Z

Oh hmm let me fix that commit message... it's failing commitlint

src/cli/commands/refs-local.js

alanshaw · 2019-05-13T14:39:00Z

Can we get some 💚 CI please @dirkmc?

…s-ipfs into feat/support-file-dom-api * 'feat/support-file-dom-api' of github.com:ipfs/js-ipfs: feat: implement ipfs refs and refs local (#2004)

* 'feat/support-file-dom-api' of github.com:ipfs/js-ipfs: feat: implement ipfs refs and refs local (#2004)

resolves #2012 Depends on - [x] #2004 - [x] ipfs-inactive/js-ipfs-http-client#992 - [x] ipfs-inactive/interface-js-ipfs-core#462 - [x] achingbrain/mortice#1 TODO: - [x] Core (mark and sweep) - [x] CLI - [x] http interface - [x] interface-js-ipfs-core tests ipfs-inactive/interface-js-ipfs-core#462 - [x] nodejs-specific tests - [x] Locking - [x] Tests for locking

* feat: implement ipfs refs * feat: refs support in http api * feat: use ipld instead of unix-fs-exporter for refs * test: add basic refs test * feat: refs local * feat: add refs.localPullStream && refs.localReadableStream * chore: change Ref -> ref * feat: make object.links work with CBOR * feat: handle multiple refs. Better param handling * fix: print refs errors to stderr * chore: add comment to explain cli param parsing * refactor: use streaming for refs local * chore: update interface-ipfs-core package * refactor: cleaner refs param handling * fix: alias 'refs local' to 'refs-local' * refactor: move links retrieval from object to refs * chore: add missing packages * refactor: use streaming for cli refs and refs local * fix: add refs and refs local to command count * fix: refs in browser * fix: restore param parsing behaviour * chore: update interface-ipfs-core * chore: update http-client * chore: update interface-ipfs-core * fix: skip failing config.set test for now * fix: skip failing config.set test in http-api * chore: update interface-ipfs-core and ipfs-http-client * chore: fix ipfs-http-client version

ghost assigned dirkmc Apr 22, 2019

ghost added the status/in-progress In progress label Apr 22, 2019

dirkmc commented Apr 22, 2019

View reviewed changes

src/cli/commands/refs.js Outdated Show resolved Hide resolved

dirkmc force-pushed the feat/refs branch 2 times, most recently from 24ccb51 to aaf2d3b Compare April 22, 2019 12:10

dirkmc marked this pull request as ready for review April 24, 2019 06:06

dirkmc requested a review from alanshaw April 24, 2019 06:07

dirkmc commented Apr 24, 2019

View reviewed changes

src/core/components/files-regular/refs-pull-stream.js Show resolved Hide resolved

alanshaw suggested changes May 1, 2019

View reviewed changes

dirkmc requested a review from alanshaw May 3, 2019 16:05

dirkmc mentioned this pull request May 8, 2019

Implement Garbage Collection #2022

Merged

11 tasks

alanshaw reviewed May 9, 2019

View reviewed changes

src/core/components/files-regular/refs-pull-stream.js Outdated Show resolved Hide resolved

alanshaw reviewed May 9, 2019

View reviewed changes

src/core/components/object.js Outdated Show resolved Hide resolved

alanshaw mentioned this pull request May 9, 2019

⚡️ v0.36.0 RELEASE 🚀 #2024

Closed

44 tasks

dirkmc force-pushed the feat/refs branch from b69d378 to ba840f1 Compare May 9, 2019 20:01

momack2 added this to In Progress in ipfs/js-ipfs May 10, 2019

momack2 added this to In Progress in ipfs/js-waffle May 10, 2019

dirkmc force-pushed the feat/refs branch from bb4356b to 29112bb Compare May 13, 2019 12:32

alanshaw suggested changes May 13, 2019

View reviewed changes

src/cli/commands/refs-local.js Outdated Show resolved Hide resolved

alanshaw approved these changes May 13, 2019

View reviewed changes

dirkmc force-pushed the feat/refs branch from bc9b9d6 to d0d577c Compare May 13, 2019 15:03

dirkmc added 14 commits May 15, 2019 09:44

fix: print refs errors to stderr

1129e4c

chore: add comment to explain cli param parsing

348b1e2

refactor: use streaming for refs local

1e9aff9

chore: update interface-ipfs-core package

1d81be8

refactor: cleaner refs param handling

bf9fdff

fix: alias 'refs local' to 'refs-local'

e8b76b4

refactor: move links retrieval from object to refs

762aff0

chore: add missing packages

3416564

refactor: use streaming for cli refs and refs local

f991388

fix: add refs and refs local to command count

5e5cd59

fix: refs in browser

07e4267

fix: restore param parsing behaviour

b55cfde

chore: update interface-ipfs-core

e663075

chore: update http-client

c731ddf

dirkmc force-pushed the feat/refs branch from ce7c41f to c731ddf Compare May 15, 2019 15:44

chore: update interface-ipfs-core

6ac01e5

alanshaw mentioned this pull request May 16, 2019

Fix: use fixtures for refs tests ipfs-inactive/interface-js-ipfs-core#471

Merged

dirkmc added 4 commits May 16, 2019 07:07

fix: skip failing config.set test for now

6b002ac

fix: skip failing config.set test in http-api

0fc81e5

chore: update interface-ipfs-core and ipfs-http-client

a6bc527

chore: fix ipfs-http-client version

384c5de

alanshaw merged commit 6dc9075 into master May 16, 2019

whyrusleeping removed the status/in-progress In progress label May 16, 2019

hugomrdias added a commit that referenced this pull request May 16, 2019

chore: merge

a7ed29d

* 'feat/support-file-dom-api' of github.com:ipfs/js-ipfs: feat: implement ipfs refs and refs local (#2004)

alanshaw deleted the feat/refs branch May 19, 2019 12:06

dirkmc mentioned this pull request Aug 23, 2019

refs and resolve don't work with sharded directories #2389

Closed

snyk-bot mentioned this pull request Sep 27, 2021

[Snyk] Security upgrade ipfs from 0.25.4 to 0.36.0 rtpro/js-ipfs#47

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement ipfs refs #2004

Implement ipfs refs #2004

dirkmc commented Apr 22, 2019 •

edited

dirkmc commented Apr 24, 2019

alanshaw commented Apr 24, 2019

dirkmc commented Apr 25, 2019

dirkmc commented Apr 25, 2019

dirkmc commented Apr 28, 2019

alanshaw left a comment

alanshaw May 1, 2019

dirkmc May 2, 2019

alanshaw May 8, 2019

dirkmc May 8, 2019

alanshaw May 8, 2019 •

edited

dirkmc May 9, 2019

alanshaw commented May 13, 2019 •

edited

dirkmc commented May 13, 2019

dirkmc commented May 13, 2019

alanshaw commented May 13, 2019

Implement ipfs refs #2004

Implement ipfs refs #2004

Conversation

dirkmc commented Apr 22, 2019 • edited

dirkmc commented Apr 24, 2019

alanshaw commented Apr 24, 2019

dirkmc commented Apr 25, 2019

dirkmc commented Apr 25, 2019

dirkmc commented Apr 28, 2019

alanshaw left a comment

Choose a reason for hiding this comment

alanshaw May 1, 2019

Choose a reason for hiding this comment

dirkmc May 2, 2019

Choose a reason for hiding this comment

alanshaw May 8, 2019

Choose a reason for hiding this comment

dirkmc May 8, 2019

Choose a reason for hiding this comment

alanshaw May 8, 2019 • edited

Choose a reason for hiding this comment

dirkmc May 9, 2019

Choose a reason for hiding this comment

alanshaw commented May 13, 2019 • edited

dirkmc commented May 13, 2019

dirkmc commented May 13, 2019

alanshaw commented May 13, 2019

dirkmc commented Apr 22, 2019 •

edited

alanshaw May 8, 2019 •

edited

alanshaw commented May 13, 2019 •

edited