Pubsub subscription data encoding

[achingbrain]

Version information:

go-ipfs version: 0.8.0
Repo version: 11
System version: amd64/darwin
Golang version: go1.15.8

Description:

The HTTP API has a few endpoints that return newline-delimited JSON which is used to stream responses to the client.

One such API is pubsub/sub. The problem here is that the data is sent as an ascii-encoded string with no escaping so can include the newline character code which ipfs-http-client then interprets (not unreasonably) as ending the current line.

The object API accepts a data-encoding argument that lets you encode binary data in a form of the users' choosing. The pubsub API (and anything that outputs binary data) should accept a data-encoding argument.

Refs: ipfs/js-ipfs#3567

[aschmahmann]

@achingbrain can you clarify a bit more what's going on? When we emit JSON binary data it's automatically encoded as base64 so there shouldn't be any issues here with newlines.

e.g.

Subscribing node:

curl -X POST "http://localhost:5001/api/v0/pubsub/sub?arg=foo&enc=json"
{"from":"ACQIARIgGbM2BsBIgAmyRUBJGsbSZSJ1kbecCrdFWVJNaOY5GJI=","data":"eAp5","seqno":"Fma/NnTLl2Y=","topicIDs":["foo"]}

Publishing node:
ipfs pubsub pub "foo" "x`ny" // Note: ` is the escape character in PowerShell

Base64 decoding eAp5 gives x\ny

[achingbrain]

If I use echo with -e to interpret the backlash, I get two pubsub messages at the other end - :

$ echo -ne 'he\nllo' | ipfs pubsub pub example

{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"aGU=","seqno":"FmcMdzDnDEM=","topicIDs":["example"]}
{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"bGxv","seqno":"FmcMdzDnDEQ=","topicIDs":["example"]}

The first message contains he and the second llo, the newline character is lost.

If I run wireshark while I'm doing this, I can see only one HTTP request is made to the HTTP API.

I think in the background this is what is happening in ipfs/js-ipfs#3567

If I run this simplified version of the repro case in that issue, I can see that only one HTTP request is made to the API, yet two pubsub messages are received by the subscriber:

const { createFactory } = require('ipfsd-ctl')
const ipfsBin = require('go-ipfs').path()
const ipfsHttpModule = require('ipfs-http-client')

const factory = createFactory({
  type: 'go',
  test: true,
  ipfsBin,
  ipfsHttpModule,
  disposable: true,
  args: ['--enable-pubsub-experiment']
})

async function main () {
  // Spawn and connect nodes
  const daemonA = await factory.spawn({ profile: 'test' })
  const daemonB = await factory.spawn({ profile: 'test' })
  const { addresses: addressesA } = await daemonA.api.id()
  const { addresses: addressesB } = await daemonB.api.id()
  const ipfsA = daemonA.api
  const ipfsB = daemonB.api
  await ipfsA.swarm.connect(addressesB[0])
  await ipfsB.swarm.connect(addressesA[0])

  // Create an empty buffer, write a Uint8 0x0a at position 2
  const buffer = Buffer.alloc(4)
  buffer.writeUInt8(0x0a, 2) // 0x0a \n

  ipfsB.pubsub.subscribe("example", ({ data }) => {
    // two messages are received
    console.info('received:')
    console.info(Buffer.from(data).toString('base64'))
  })

  // Publish the buffer
  console.info('sending:')
  console.info(buffer.toString('base64'))
  await ipfsA.pubsub.publish("example", buffer)
}

main().catch(err => {
  console.error(err)
  process.exit(1)
})

$ node index.js
sending:
AAAKAA==
received:
AAA=
received:
AA==

[aschmahmann]

I'm confused why this isn't an issue with the HTTP client instead of with go-ipfs. Both of the following exhibit the single message of x newline y (or eAp5 in base64):

curl -X POST "http://localhost:5001/api/v0/pubsub/pub?arg=foo&arg=x%0Ay"
ipfs pubsub pub "foo" "x`ny" // Note: ` is the escape character in PowerShell

As for your CLI issues:

$ echo -ne 'he\nllo' | ipfs pubsub pub example

You may be running into issues with your shell breaking up the data into multiple commands (e.g. powershell will break up inputs by newline characters). What shell are you using?

[achingbrain]

What shell are you using?

Nothing fancy, just bash:

$ echo $SHELL
/bin/bash

You may be running into issues with your shell breaking up the data into multiple commands

That occurred to me too but wireshark shows there's only one HTTP request being sent so I don't think that's the case.

The data itself is different, if I pipe the echo output to a hex dumper with and without the -e flag you can see the difference (the -n flag strips the trailing newline that would have been added):

$ echo -n 'x\ny' | xxd
00000000: 785c 6e79                                x\ny
$ echo -en 'x\ny' | xxd
00000000: 780a 79                                  x.y

If I use backticks instead of a pipe, I get the double message:

$ ipfs pubsub pub example `echo -ne 'x\ny'`

{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"eA==","seqno":"FmcMdzDnDF8=","topicIDs":["example"]}
{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"eQ==","seqno":"FmcMdzDnDGA=","topicIDs":["example"]}

Wireshark shows me one request, but with multiple arg args:

I can get the same double message with:

$ ipfs pubsub pub example x y

If I wrap the backticks in quotes I get one message, which is the same as your example above:

$ ipfs pubsub pub example "`echo -ne 'x\ny'`"

{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"eAp5","seqno":"FmcMdzDnDGY=","topicIDs":["example"]}

So I guess that's working as expected. Maybe. If newlines can delimit CLI arguments.

---

Back to the HTTP API though.

The pubsub pub endpoint seems to take the data arg as both a query string parameter and a multipart part.

If I send it as a querystring parameter, as you do above, everything works:

$ curl -X POST http://localhost:5001/api/v0/pubsub/pub?arg=example\&arg=x%0ay

{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"eAp5","seqno":"FmcMdzDnDG0=","topicIDs":["example"]}

If I send it as a multipart/form-data request, I get the double message:

echo -en "x\ny" | curl -X POST http://localhost:5001/api/v0/pubsub/pub?arg=example -F data=@-

{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"eA==","seqno":"FmcMdzDnDG4=","topicIDs":["example"]}
{"from":"ACQIARIgWZ+O1WZdgbeglCYBeHuFbWNpZ7GnqTit80jbzL3Md38=","data":"eQ==","seqno":"FmcMdzDnDG8=","topicIDs":["example"]}

You can see the data being sent in a single request here:

Appending this data to the URL limits the maximum message size so sending as form data is preferable but it doesn't appear to work if there's a newline character in there.

From what I understand there's a 4MB effective limit on pubsub message sizes but it's impossible for some clients to reach that size over the HTTP API if we send the data as part of the query string (also bear in mind that non-printable bytes get escaped into three characters - %NN - further limiting the max sendable message size).

There are also fun times to be had with some non-printable bytes and text conversion:

new TextEncoder().encode(new TextDecoder().decode(Uint8Array.from([254])))
> Uint8Array(3) [239, 191, 189]
new TextEncoder().encode(new TextDecoder().decode(Uint8Array.from([253])))
> Uint8Array(3) [239, 191, 189]
new TextEncoder().encode(new TextDecoder().decode(Uint8Array.from([252])))
> Uint8Array(3) [239, 191, 189]
// etc

Really, this would all be much simpler if we could just specify data input encoding for passing data on the query string.

It would be better to be able to pass data as a multipart request without it getting split into multiple messages. Then we can just send raw bytes and not have to worry about encoding.

[achingbrain]

Just came across another part of the HTTP API where this will be a problem - sending byte arrays as strings for DHT put/gets will have the same encoding problems as pubsub pub data as they go on the query string.

[aschmahmann]

Maybe short term let's log an error in go-ipfs if you send data that contains newlines and stop splitting on newline in multipart.

[BigLep]

@aschmahmann : please also link this to project proposals that would fix the root cause.

[aschmahmann]

@BigLep I can't find if it was split out into a separate proposal, but the proposal for a simpler HTTP Writeable Gateway style API (which would support things like adding data to IPFS) ended up tailing towards a higher prioirity for support of a websockets+grpc style API protocol/web3-dev-team#1 (comment). (e.g. like ipfs/js-ipfs#3403).

@achingbrain or @lidel might know of other pointers/issues to link to as well.

[lidel]

Summary of decision made during Stewards Discussion Call today:

• pubsub is EXPERIMENTAL in go-ipfs, so we decided to make a BREAKING CHANGE and fix /api/v0/pubsub/* commands

Broad strokes fix:

1. add ipfs multibase encode|decode utility commands (similar to ipfs cid) – necessary, so people can encode topic names and encode data with regular ipfs client
   • make sure it works with stin and flat files as input (+output for decode)
   • 👉 this is
2. require topic name and data to be encoded in multibase
   • request: topic name in the first arg
   • response: data and topic name in fields inside JSON
3. ensure pubsub pub publishes only one message: first arg is topic name (in multibase), everything after that should be interpreted as a binary message

@achingbrain @aschmahmann @BigLep I've added this to maintenance board.

---

Update 2021-06:

• (1) is being added in feat: add 'ipfs multibase' commands #8180
• (2) and (3) is being fixed in fix: multibase in pubsub http rpc #8183

[BigLep]

I'm assuming this needs to stay open until #8183 is closed.

[BigLep]

@lidel will write the remaining steps once he reengages.