Can't seem to get mutable Put with salt to work

[bensyverson]

I'm having some issues getting mutable Put to work with a salt—it works fine without. Here's some example code to add a mutable item to the DHT… (usage: node put.js Hello world!)

var DHT = require('bittorrent-dht')
var KP = require('bittorrent-dht-store-keypair');
var kp = KP();
var dht = new DHT({ verify: KP.verify });

dht.on('ready', function () {
    var key = dht.put(kp.store(process.argv.slice(2).join(' '), {salt: "test"}), function (err, res, n) {
	  console.error('error=' + err  + " / hash=" + res.toString('hex') + " / copies=" + n);
	});
	console.log("Trying to put " + key.toString('hex'));
});

…And the script to retrieve the key… (usage node get.js 81b692589463a1745493dbfa2dd403d1c9d095b6)

var DHT = require('bittorrent-dht')
var KP = require('bittorrent-dht-store-keypair');
var kp = KP();
var dht = new DHT({ verify: KP.verify });

function fetch(){
	dht.get(process.argv[2], function(err, res){
		console.log("err=" + err);
		console.log("res=" + JSON.stringify(res));
		if (!res) {
			fetch();
		}
	});
};
fetch();

If I remove the {salt: "test"} options argument in put.js, everything works fine. But for my use case, I really need the salt. I may be misunderstanding some part of the API—if so, let me know where I've gone wrong!

[pldubouilh]

I tried to replicate you issue with some code from the tests, here's what I came up with :

var DHT = require('bittorrent-dht')
var ed = require('ed25519-supercop')

var keypair = ed.createKeyPair(ed.createSeed())

var dht1 = new DHT({ verify: ed.verify })
var dht2 = new DHT({ verify: ed.verify })

var sign = function (keypair) {
  return function (buf) {
    return ed.sign(buf, keypair.publicKey, keypair.secretKey)
  }
}

var pending = 2
dht1.listen(function () {
  dht2.addNode({ host: '127.0.0.1', port: dht1.address().port })
  dht2.once('node', ready)
})

dht2.listen(function () {
  dht1.addNode({ host: '127.0.0.1', port: dht2.address().port })
  dht1.once('node', ready)
  ready()
})

function ready () {
  if (--pending !== 0) return

  console.log('start')

  var fopts = {
    k: keypair.publicKey,
    v: Buffer('aaaaaaaaaa'),
    seq: 0,
    salt: Buffer('seasalt'),
    sign: sign(keypair),
  }

  dht1.put(fopts, function (err, hash) {
    if(err) console.warn(err)
    else console.log('dht1 set', fopts.v.toString('utf8'), 'at', hash.toString('hex'))

    dht2.get(hash, function (err, res) {
      if(err) console.warn(err)
      else console.log('dht2 received', res ? res.v.toString('utf8') : null )
    })
  })
}

Here's what I figured so far :

1. Local (boostrap: false) > working fine
2. Live with both dhts added to one-another > working fine
3. Live with both dhts not knowing about one another (no dht.addNode) > not working

And as you noticed, it's all working fine if the salt parameter is removed

[bensyverson]

Yeah, it's strange. We're getting replies from remote DHTs that they've stored the value. Perhaps those peers are quickly discarding mutable values with salts?

I've added a bunch of debug messages to monitor what's happening, but the core of it seems to be that krpc.closest() can never find the key.

[the8472]

Afaik utorrent did not support salted puts for a while, but iirc it returned an error response to the put RPC in that case which should lead the writing node to try storing it on other nodes. Libtorrent and my own implementation do support it.

but the core of it seems to be that krpc.closest() can never find the key.

Since BEP44 support is not as widespread as basic BEP5 it might have to cast a wider net, at least looking for a sufficient amount of responses with tokens, since that would indicate that the nodes actually understood the query instead of just returning nodes lists.

[thomasyuan]

Local (boostrap: false) > working fine
Live with both dhts added to one-another > working fine
Live with both dhts not knowing about one another (no dht.addNode) > not working

It's not strange. 1 and 2 work because there is at least one node is bittorrent-dht, which calculate the hash based on salt + key, but the BEP44 said the hash should be calculated based on key + salt, that's why 3 doesn't work.

#157 suppose to fix this issue.

[feross]

This is fixed in 7.5.2, thanks to @thomasyuan 👏

[bensyverson]

Maybe I'm being dense, but this still isn't working with bootstrap == true. I can get it to work when I manually add known nodes, but how can I discover them? @thomasyuan

[thomas-yuan]

It's weird. I tested yesterday at home, with the code @pldubouilh posted here. It failed.
But today, I tested again at the office, It worked fine!

BTW, you might need another patch I posted at #158. (I think the PR is not enough, see my comments).

Here is the log when it worked.

start
visited 18 nodes
dht1 set aaaaaaaaaa at b4de78d9fb67496e83a0b7eb052a1b948165b8de
value  { v: <Buffer 61 61 61 61 61 61 61 61 61 61>,
  k: <Buffer d6 17 b1 fa 22 31 73 9d fd 15 85 a2 33 16 fb d4 30 58 83 2b 9c 2e 25 1d 91 bb 6a d1 63 c2 8c 92>,
  salt: <Buffer 73 65 61 73 61 6c 74>,
  sig: <Buffer 36 b7 49 f7 2f 87 9f aa 0a 78 e5 85 7f 41 4b 6f a0 fb c6 0a 5c 82 17 63 2e 70 48 f1 8a 2e d8 3d b0 e9 a1 b6 64 ac dd 64 03 15 65 a2 47 29 87 cf a0 20 ... >,
  seq: 0,
  id: <Buffer 7e 23 89 d0 77 a1 3c c0 2e 4e 07 a0 c7 7b 39 5a 1a ac cd 59> } r.seq 0
value  { v: <Buffer 61 61 61 61 61 61 61 61 61 61>,
  k: <Buffer d6 17 b1 fa 22 31 73 9d fd 15 85 a2 33 16 fb d4 30 58 83 2b 9c 2e 25 1d 91 bb 6a d1 63 c2 8c 92>,
  salt: <Buffer 73 65 61 73 61 6c 74>,
  sig: <Buffer 36 b7 49 f7 2f 87 9f aa 0a 78 e5 85 7f 41 4b 6f a0 fb c6 0a 5c 82 17 63 2e 70 48 f1 8a 2e d8 3d b0 e9 a1 b6 64 ac dd 64 03 15 65 a2 47 29 87 cf a0 20 ... >,
  seq: 0,
  id: <Buffer 7e 23 89 d0 77 a1 3c c0 2e 4e 07 a0 c7 7b 39 5a 1a ac cd 59> } r.seq 0
value  { v: <Buffer 61 61 61 61 61 61 61 61 61 61>,
  k: <Buffer d6 17 b1 fa 22 31 73 9d fd 15 85 a2 33 16 fb d4 30 58 83 2b 9c 2e 25 1d 91 bb 6a d1 63 c2 8c 92>,
  salt: <Buffer 73 65 61 73 61 6c 74>,
  sig: <Buffer 36 b7 49 f7 2f 87 9f aa 0a 78 e5 85 7f 41 4b 6f a0 fb c6 0a 5c 82 17 63 2e 70 48 f1 8a 2e d8 3d b0 e9 a1 b6 64 ac dd 64 03 15 65 a2 47 29 87 cf a0 20 ... >,
  seq: 0,
  id: <Buffer 7e 23 89 d0 77 a1 3c c0 2e 4e 07 a0 c7 7b 39 5a 1a ac cd 59> } r.seq 0
visited 37 nodes
dht2 received aaaaaaaaaa

[bensyverson]

FWIW, this is still not working for me. I'm not able to retrieve anything stored with a salted mutable Put.