ipfs-http-client: Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed, when I make a call

[olehmell]

Hi guys,

We created a simple wrapper class SubsocialIpfsApi around IPFS API. From this wrapper we make calls to get/add data from/to IPFS. We use such IPFS methods as “cat” and “add”.

When we make a call directly to IPFS from SubsocialIpfsApi class, everything works perfectly. But when make a call from another top-level wrapper SubsocialApi (it encapsulates both IPFS and Substrate API wrappers) we experiense an issue: the data displayed in the console inside the loop is returned, but the loop ends with an error:

const res = this.ipfs.api.cat(cids[0])
for await (const buf of res) {
    const content = JSON.parse(buf.toString())
    console.log(content);
}

Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed

An error is also generated when using the "it-all" package.

const promiseArray = ipfsCids.map(cid => all(this.api.cat(cid) as any));
const jsonContentArray = await Promise.all(promiseArray);

Test 1 - passes, it make calls to IPFS from SubsocialIpfsApi

Test 2 - fails, it make calls to IPFS from SubsocialApi

[achingbrain]

I'm having problems running these tests. This is what I've done:

1. Cloned https://github.com/mell-old/dappforce-subsocial-js.git
2. Checked out the branch mell-fix-ipfs
3. Run yarn in the root of the project
4. Started a js-IPFS daemon (consider using ipfsd-ctl to do this as part of your test setup)
5. Run yarn test

I see:

$ yarn test
yarn run v1.22.4
$ jest --testPathIgnorePatterns e2e
 PASS  packages/api/test/ipfs.test.ts
  ✓ Save a blog to IPFS (56ms)
  ✓ Find blog (25ms)

  console.log packages/api/src/ipfs.ts:24
    Created SubsocialIpfsApi instance

  console.log packages/api/src/ipfs.ts:24
    Created SubsocialIpfsApi instance

  console.log packages/api/src/ipfs.ts:37
    [ 'QmSFh97yz7XcL2rZwGheYEsTFoXHsk5jq6H2cwR94YTRcp' ]

  console.log packages/api/src/ipfs.ts:41
    [
      [
        <Buffer 7b 22 6e 61 6d 65 22 3a 22 54 65 73 74 20 42 6c 6f 67 22 2c 22 64 65 73 63 22 3a 22 42 6c 6f 67 20 64 65 73 63 22 2c 22 69 6d 61 67 65 22 3a 22 22 2c ... 12 more bytes>
      ]
    ]

  console.log packages/api/src/ipfs.ts:43
    [
      [
        <Buffer 7b 22 6e 61 6d 65 22 3a 22 54 65 73 74 20 42 6c 6f 67 22 2c 22 64 65 73 63 22 3a 22 42 6c 6f 67 20 64 65 73 63 22 2c 22 69 6d 61 67 65 22 3a 22 22 2c ... 12 more bytes>
      ]
    ]

  console.log packages/api/src/substrateConnect.ts:60
    Get Substrate API: DfApi.setup()

  console.log packages/api/src/substrateConnect.ts:18
    Connecting to Substrate API at ws://127.0.0.1:9944/

  console.log node_modules/@polkadot/util/logger.js:87
    2020-03-27 17:40:14          API-WS: socket error Event { isTrusted: [Getter] }

  console.error node_modules/@polkadot/util/logger.js:87
    2020-03-27 17:40:14          API-WS: disconnected from ws://127.0.0.1:9944/ code: '1006' reason: ''

 FAIL  packages/api/test/subsocial.test.ts
  ✕ Find blog from Substrate and IPFS (80ms)

  ● Find blog from Substrate and IPFS

    Failed: Event {
      "isTrusted": true,
    }

       7 | import { BlogData } from '@subsocial/types/src/dto'
       8 | 
    >  9 | test('Find blog from Substrate and IPFS', async () => {
         | ^
      10 |   const substrarteApi = await getApi();
      11 |   const blogId = new BN(1);
      12 |   const api = new SubsocialApi(substrarteApi, ipfsConnect);

      at Env.it (node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:89:24)
      at Object.<anonymous> (packages/api/test/subsocial.test.ts:9:1)

Test Suites: 1 failed, 1 passed, 2 total
Tests:       1 failed, 2 passed, 3 total
Snapshots:   0 total
Time:        3.819s
Ran all test suites.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Digging around, the promise here is getting rejected with an Event (e.g. not an Error). The event seems to come from inside polkadot which is not a framework I'm familiar with.

If you could let me know how to run these tests or better yet, provide an example of the error without all the application code around it, that would be amazing.

[achingbrain]

I'm guessing I need to start something else that will listen on ws://127.0.0.1:9944?

[siman]

I'm guessing I need to start something else that will listen on ws://127.0.0.1:9944?

Hey @achingbrain you are right. There should be a Substrate blockchain node running on that address. Sorry for the inconvenience with the tests. We will try to create a Docker image for Substrate node so it can be easily started with tests.

[siman]

consider using ipfsd-ctl to do this as part of your test setup

@achingbrain could you elaborate on this please?

[achingbrain]

ipfsd-ctl is a convenient way to spin up and tear down IPFS nodes - go, js, in-process - works in the browser, in node, etc.

You can use it to start an IPFS node, something like:

const { createFactory } = require('ipfsd-ctl')
const factory = createFactory({
  type: 'proc' // or 'js' or 'go'
})

describe('some feature', () => {
  let ipfs

  before(async () => {
    ipfs = factory.spawn()
  })

  it('tests something', async () => {
    // ... assert something here
    assert(await ipfs.id())
  })

  after(async () => {
    factory.clean()
  })
})

[olehmell]

@achingbrain
We updated branch and created docker for Substrate.
Instruction:

1. Clone https://github.com/mell-old/dappforce-subsocial-js.git
2. Check out the branch mell-fix-ipfs
3. Run yarn in the root of the project
4. Run docker that will start Substrate node on port 9944:

docker run --rm -d --name subsocial-node --network="host" -v ~/subsocial_data:/data dappforce/subsocial-node:df-v2 ./subsocial-node --dev

5. Start a js-IPFS or go-IPFS daemon on 5001 port
6. Run yarn test

[achingbrain]

The tests are failing to connect to the Substrate API for me with:

  console.log node_modules/@polkadot/api/node_modules/@polkadot/util/logger.js:87
    2020-05-29 16:59:40          API-WS: socket error Event { isTrusted: [Getter] }

  console.error node_modules/@polkadot/api/node_modules/@polkadot/util/logger.js:87
    2020-05-29 16:59:40          API-WS: disconnected from ws://127.0.0.1:9944/ code: '1006' reason: ''

I've got the docker image running:

$ docker ps
CONTAINER ID        IMAGE                                            COMMAND                  CREATED             STATUS              PORTS               NAMES
c43c5bf8b742        dappforce/subsocial-node:df-v2                   "./subsocial-node --…"   51 seconds ago      Up 50 seconds                           subsocial-node

IPFS is running:

$ lsof -i :5001
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ipfs    98798 alex   42u  IPv4 0xda838f22bc58c89f      0t0  TCP localhost:commplex-link (LISTEN)

Nothing is listening on port 9944:

$ lsof -i :9944
* no output *
$ nmap localhost -p 9944
Starting Nmap 7.70 ( https://nmap.org ) at 2020-05-29 17:12 BST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00022s latency).
Other addresses for localhost (not scanned): ::1 fe80::1

PORT     STATE  SERVICE
9944/tcp closed unknown

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

I can ignore the --network="host" option and map the ports manually:

$ docker run --rm -d --name subsocial-node -p 30333:30333 -p 9933:9933 -p 9944:9944 -v ~/subsocial_data:/data dappforce/subsocial-node:df-v2 ./subsocial-node --dev
04e17a17fc6849fd13c2280ba24540f817337a69b1e11b79c7cd3294be3e1ce9
MacBook-Pro-2:~ alex$ docker ps
CONTAINER ID        IMAGE                                            COMMAND                  CREATED             STATUS              PORTS                                                                      NAMES
04e17a17fc68        dappforce/subsocial-node:df-v2                   "./subsocial-node --…"   2 seconds ago       Up 1 second         0.0.0.0:9933->9933/tcp, 0.0.0.0:9944->9944/tcp, 0.0.0.0:30333->30333/tcp   subsocial-node

Now something is listening:

$ nmap localhost -p 9944
Starting Nmap 7.70 ( https://nmap.org ) at 2020-05-29 17:14 BST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00035s latency).
Other addresses for localhost (not scanned): ::1 fe80::1

PORT     STATE SERVICE
9944/tcp open  unknown

..but the tests still fail with the same error:

  console.log node_modules/@polkadot/api/node_modules/@polkadot/util/logger.js:87
    2020-05-29 17:14:28          API-WS: socket error Event { isTrusted: [Getter] }

  console.error node_modules/@polkadot/api/node_modules/@polkadot/util/logger.js:87
    2020-05-29 17:14:28          API-WS: disconnected from ws://127.0.0.1:9944/ code: '1006' reason: ''

Any ideas?

[achingbrain]

Closing due to lack of response.

If you're still having a problem with this, please reopen.