fix: ipfs get with raw blocks #3683

olizilla · 2021-05-10T13:46:28Z

tweak ipfs-cli get to handle raw
tweak ipfs-core mapFile to handle raw

License: MIT
Signed-off-by: Oli Evans oli@tableflip.io

- tweak ipfs-cli get to handle `raw` - tweak ipfs-core mapFile to handle `raw` fixes #3682 License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

achingbrain · 2021-05-10T14:24:39Z

packages/ipfs-core/src/utils.js

@@ -103,28 +103,32 @@ const mapFile = (file, options = {}) => {
    name: file.name,
    depth: file.path.split('/').length,
    size: 0,
-    type: 'file'
+    type: file.type


Types are broken - output has a type of IPFSEntry - IPFSEntry.type can only be 'dir' or 'file'. file.type has more possible values than that so the typecheck fails.

Just to check, should the value of output.type be file when file.type === 'raw' ?

ok so i guess I need to push the "can't deal with objects or identity" check down into mapFiles.

objects or identity

AFAIK these parts of UnixFS aren't used by anything.

should the value of output.type be file when file.type === 'raw'

Yes, unless go-ipfs does something different 🙂

License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

- mapFile doesn't handle all mapping object or identity types to an IPFSEntry, so throw if it finds one. This makes the cli behaviour match go-ipfs for ipfs get <cbor cid> License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

olizilla · 2021-05-10T16:02:46Z

@achingbrain i think i have sated the type machine

- add interface-core test for get with rawLeaves: true - remove cli test for "raw" as core normalises the type to "file" so it was not testing a valid conditon License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

olizilla · 2021-05-10T20:17:33Z

Nope, more noodling was required. The cli test was failing was testing a condition that the core normalises out, so that is removed. Meanwhile the interface-core tests were missing a test for getting a block that was added with rawLeaves: true so that is added. I can confirm that that test would fail on default branch and passes (locally at least) on this branch with the changes to the mapFiles fn.

achingbrain · 2021-05-11T06:18:01Z

Linting is failling?

ipfs-core: /home/travis/build/ipfs/js-ipfs/packages/ipfs-core/src/utils.js:130:1: Trailing spaces not allowed. [Error/no-trailing-spaces]

achingbrain · 2021-05-11T06:19:30Z

Also the new test doesn't seem to work over http:

ipfs:   1) interface-ipfs-core over ipfs-http-client tests against js-ipfs
ipfs:        .get
ipfs:          should get a file added as CIDv1 with rawLeaves:
ipfs:      Error: Timeout of 120000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/travis/build/ipfs/js-ipfs/packages/ipfs/test/interface-http-js.js)
ipfs:       at listOnTimeout (internal/timers.js:554:17)
ipfs:       at processTimers (internal/timers.js:497:7)

olizilla · 2021-05-11T08:26:36Z

yes i've been leaning on ci to lint and test as it's not working locally. am trying a npm run reset && npm i now. I'll take a look at the http test failure.

❯ npm run lint                                                                                                                                                        09:23:32

> js-ipfs@1.0.0 lint
> lerna run lint

lerna notice cli v3.22.1
lerna info versioning independent
lerna info Executing command in 16 packages: "npm run lint"
interface-ipfs-core: > interface-ipfs-core@0.144.2 lint
interface-ipfs-core: > aegir lint
ipfs-core-types: > ipfs-core-types@0.3.1 lint
ipfs-core-types: > aegir lint
ipfs-core-types: [09:23:42] Lint files [started]
interface-ipfs-core: [09:23:42] Lint files [started]
interface-ipfs-core: [09:23:44] Lint files [failed]
interface-ipfs-core: [09:23:44] → Plugin "no-only-tests" was conflicted between "package.json » eslint-config-ipfs#overrides[0] » ./js" and "../../package.json » eslint-config-ipfs#overrides[0] » ./js".
interface-ipfs-core: Plugin "no-only-tests" was conflicted between "package.json » eslint-config-ipfs#overrides[0] » ./js" and "../../package.json » eslint-config-ipfs#overrides[0] » ./js".
lerna ERR! npm run lint exited 1 in 'interface-ipfs-core'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.

achingbrain · 2021-05-11T08:38:45Z

it's not working locally

interface-ipfs-core: [09:23:44] → Plugin "no-only-tests" was conflicted between "package.json » eslint-config-ipfs#overrides[0] » ./js" and "../../package.json » eslint-config-ipfs#overrides[0] » ./js".

Do you have a rogue package.json at ../../package.json that has a dep on a different version of eslint-plugin-no-only-tests somehow, or some other weird eslint config?

The ./js it refers to I think is this one, but it looks like it's resolving some other eslint config in your path or env somewhere that's causing the conflict.

olizilla · 2021-05-11T09:00:19Z

running reset and install has fixed it. I'll get those checks green yet.

License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

olizilla · 2021-05-11T10:08:42Z

The only place I can see an issue could occur for fetching a rawLeaves node via the js-ipfs http api, that doesn't occur on the ipfs-core get path would be here

js-ipfs/packages/ipfs-http-server/src/api/resources/files-regular.js

Lines 146 to 148 in 42f166b

    
           if (file.type === 'file' && file.content != null) { 
        
             yield { header: { ...header, size: file.size }, body: toBuffer(file.content) } 
        
           } else {

Is there an easy way to run the interface core get tests against my local js-ipfs code and see the server side console log output? I've running npm run test:interface:http-go from the root which is working well, but I need to see what's going on.

achingbrain · 2021-05-11T14:40:16Z

I need to see what's going on.

Run with the DEBUG env var, e.g.:

$ DEBUG=ipfs* npm run test:interface:http-js -- -- -- --grep 'should get a file added as CIDv1 with rawLeaves'

It's probably more legible if you run it from the packages/ipfs dir as you'll get colour-coded output without lerna getting in the way and stripping it:

$ cd packages/ipfs
$ DEBUG=ipfs* npm run test:interface:http-js -- --grep 'should get a file added as CIDv1 with rawLeaves'

License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

olizilla · 2021-05-11T16:17:50Z

the file.size wasn't being set for raw nodes. Setting it fixes the test.

olizilla · 2021-05-11T16:18:41Z

packages/ipfs-core/src/utils.js

  /** @type {import('ipfs-core-types/src/root').IPFSEntry} */
  const output = {
    cid: file.cid,
    path: file.path,
    name: file.name,
    depth: file.path.split('/').length,
-    size: 0,
+    size: file.size,


is it legit to pass through the raw size for a raw node?

Yes, it'll be the size of the block

achingbrain · 2021-05-11T17:33:11Z

CI failure is dns related, I've restarted the job & will merge when it's green.

achingbrain · 2021-05-11T18:23:49Z

Shipped in ipfs@0.55.1

fix: ipfs get with raw blocks

befb8e6

- tweak ipfs-cli get to handle `raw` - tweak ipfs-core mapFile to handle `raw` fixes #3682 License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

achingbrain reviewed May 10, 2021

View reviewed changes

olizilla added 2 commits May 10, 2021 16:13

chore: fix type warnings

90f353b

License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

chore: satisfy the type constraints

a2c86d9

- mapFile doesn't handle all mapping object or identity types to an IPFSEntry, so throw if it finds one. This makes the cli behaviour match go-ipfs for ipfs get <cbor cid> License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

chore: add test for get with rawLeaves

5875102

- add interface-core test for get with rawLeaves: true - remove cli test for "raw" as core normalises the type to "file" so it was not testing a valid conditon License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

chore: delint

42f166b

License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

fix: provide file.size for raw blocks

d445b91

License: MIT Signed-off-by: Oli Evans <oli@tableflip.io>

olizilla commented May 11, 2021

View reviewed changes

achingbrain approved these changes May 11, 2021

View reviewed changes

achingbrain merged commit 28235b0 into master May 11, 2021

achingbrain deleted the get-raw branch May 11, 2021 18:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: ipfs get with raw blocks #3683

fix: ipfs get with raw blocks #3683

olizilla commented May 10, 2021

achingbrain May 10, 2021 •

edited

olizilla May 10, 2021

olizilla May 10, 2021

achingbrain May 10, 2021

olizilla commented May 10, 2021

olizilla commented May 10, 2021 •

edited

achingbrain commented May 11, 2021 •

edited

achingbrain commented May 11, 2021

olizilla commented May 11, 2021

achingbrain commented May 11, 2021

olizilla commented May 11, 2021

olizilla commented May 11, 2021

achingbrain commented May 11, 2021

olizilla commented May 11, 2021

olizilla May 11, 2021

achingbrain May 11, 2021

achingbrain commented May 11, 2021

achingbrain commented May 11, 2021

fix: ipfs get with raw blocks #3683

fix: ipfs get with raw blocks #3683

Conversation

olizilla commented May 10, 2021

achingbrain May 10, 2021 • edited

Choose a reason for hiding this comment

olizilla May 10, 2021

Choose a reason for hiding this comment

olizilla May 10, 2021

Choose a reason for hiding this comment

achingbrain May 10, 2021

Choose a reason for hiding this comment

olizilla commented May 10, 2021

olizilla commented May 10, 2021 • edited

achingbrain commented May 11, 2021 • edited

achingbrain commented May 11, 2021

olizilla commented May 11, 2021

achingbrain commented May 11, 2021

olizilla commented May 11, 2021

olizilla commented May 11, 2021

achingbrain commented May 11, 2021

olizilla commented May 11, 2021

olizilla May 11, 2021

Choose a reason for hiding this comment

achingbrain May 11, 2021

Choose a reason for hiding this comment

achingbrain commented May 11, 2021

achingbrain commented May 11, 2021

achingbrain May 10, 2021 •

edited

olizilla commented May 10, 2021 •

edited

achingbrain commented May 11, 2021 •

edited