Connection dies with larger writes

[jensenhertz]

I noticed a curious behavior. If you write much more than ~204800 bytes, the connection dies instantly. Is this expected behavior?

I don't have much in-depth experience with WebRTC, so excuse me if this is well-known thing or something ;)

Here's my test code...

const Swarm = require('../')

;(async () => {
  const hostname = 'your hyperswarm server instance'  
  const swarm = Swarm({
    bootstrap: [hostname],
  })
  
  const topic = Buffer.from('aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f', 'hex')
  
  swarm.once('connection', (connection, info) => {
    console.log('new connection!', info)
    connection.once('end', () => {
      console.log('Connection client1 -> client2 ended')
    })
    connection.once('data', (d) => {
      console.log('The client1 got data', d)
    })
    const b = Buffer.alloc(200*1024)
    console.log(`Sending ${b.byteLength} bytes`)
    connection.write(b)
  })
  
  
  swarm.on('disconnection', (socket, details) => {
    console.log(details.peer.host, 'disconnected!')
    console.log('now we have', swarm.peers.length, 'peers!')
  })
  
  swarm.join(topic)
  
  swarm.webrtc.signal.once('connected', () => {
    console.log('client webrtc connected')
  })
  
  await swarm.flush(()=>{})
})()

[tinchoz49]

Hi @jensenhertz, the data channel of webrtc has a limited payload defined by the browser. You will need to use a split chunk solution to avoid this limitation.

https://tensorworks.com.au/blog/webrtc-stream-limits-investigation/#14-maximum-data-channel-message-size

[jensenhertz]

Hey @tinchoz49, I thought it might be something like that. It's going to be curious exercise to get hypercore replication to work with this.

[tinchoz49]

Remember that hypercore also has a limitation of 8 mb by block.

What we did in https://sher.app is to persist hypercore blocks of 10 kb (maximum) by doing this we avoid the webrtc limitation since the hypercore.stream sends chunks of 10 kb

[tinchoz49]

We also have a splitter module that split the packets before to send to webrtc and it does a resemble from the other side (peer)

[tinchoz49]

I should publish that module :S I will try to do it next week

[jensenhertz]

@tinchoz49 I was just thinking that I need to write some sort of splitter, so if you have that already and you're willing to publish it, that would be perfect! 🙏

[tinchoz49]

https://github.com/geut/splitter

[jensenhertz]

Thank you, that worked!