some local hash is not listed by ipfs refs local

[MichaelMure]

$ ipfs get QmUYndb1SkY49khYAg9Zn2yp9Z44cRFtqMnVAZP5qes8ce
$ ipfs refs local | wc -l
181
$ ipfs refs local | grep -i QmUYndb1SkY49khYAg9Zn2yp9Z44cRFtqMnVAZP5qes8ce
return nothing

It's a file part of the WebUI and is correctly written on my filesystem. Reproduced on different machine, and by @jbenet.

[whyrusleeping]

huh, query must be broken, or something.

[whyrusleeping]

we're apparently getting some bad keys back from the datastore, or something... adding an error log to the if err != nil { case after the multihash cast shows a bunch of odd failures:

ERRO[12:15:52:000] CAST FAIL!6PLpVkDrcB8qxphq4xdtiNLaeqdgp6N8QvwmtYwAwJ9Ft  module=blockstore
ERRO[12:15:52:000] CAST FAIL!6PKFxVD7zaVVmeQkUj6QJ9MNNpkmNCtEa9CBh4TBaB1A6  module=blockstore
ERRO[12:15:52:000] CAST FAIL!6PHrjoAXVGyaaUxg6KUMD3aoqEXBtQFVtUpE4Kb43Ncj6  module=blockstore
ERRO[12:15:52:000] CAST FAIL!6PJ8fb8ZfW31UD5sxfbib9JV5VWudSagADDtYpgVwyecE  module=blockstore
ERRO[12:15:52:000] CAST FAIL!6PKFNEuwhSGrJ7x5i2cMtEtB3kkfa2bJ6qyW9oQVNUhw3  module=blockstore
ERRO[12:15:52:000] CAST FAIL!6PKcZetiLXM76Ego6tVCEjQgg8t6kcxgdT14YqqRpADrH  module=blockstore
ERRO[12:15:52:000] CAST FAIL!6PJCVQMJctynH2QR6HjM6JPLSmNwdmj2ME5pEqRfM9FCi  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PKW3wmwJbMAFCn2T2h8UFTdbQDhg5AUDtw9jcPPi9zWe  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PMSDJ1MVz1u6TXsyMnRZ9ENaBkroYhMWbqTKhZdsAdrg  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PJFC68ZtavpyAqQSgexuWfUJ3Qia4iAQSHAUmdWBp56W  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PKWjNmWuhH5SjEzaNXiXwgT4k8Q1r7YvA3kJX21W254d  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PMDKuJdKuDCsAzENEJpuhT6iZYS8uyzVzJrhTfshquDi  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PLKF7wWoSM3VLz9TFEHQorUSdRu5kwYdHvWzPTzGceem  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PL4sZin5ZN7uXB9e9U9C3LyDk6wUxYHhfgRp1bQEkSiq  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PK4WaqbuyQ2sf4NXTvKVdRoSjYu6oEm2LXuEvjMvDLY6  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PKoivBj8hCmgeq95LuQpadXK3vTuA9DYdCdRyiyt38Ej  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PKrNfDKV99DL3WX4wEc8u6WPBnHj5RFXdjca9raa1ZUf  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PJpJGERQNDCVAg6dN1wWZMQqC4VCgfLVV4TpTAwgPR35  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PK2K3Q7SmkjrZZJ2jrX5mQJ2V78cfaHGwZ4D8nnW9aZY  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PL4PixqobeJmUvb3crZwTauYRr8SH8Qh66TtQBBjJRwG  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PHgTGdi4vLrUpJy46wSbTkekLbF4NW5x19b3eP8Y2ZSE  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PHndE8ej1GrGqyCrSDE3yCvBQVpfDHvqfvwFvphWCuuh  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PL6dGwwHrcobiAkk6k6VdXTtWqTgNcdAAcgoWNn2FjNQ  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PHgHm91D4KhvLjg8prZibW9g2Zru1t8N42dKyWRyVZYy  module=blockstore
ERRO[12:15:53:000] CAST FAIL!6PJA9GrP7eaj42J3fP9zodR29zhgZEjYLgtGbUXyzPWwQ  module=blockstore

[whyrusleeping]

It appears that some of the 'keys' coming up from the datatstore query are one character short, for example, a good key: (bytes)

b := []byte{18, 32, 235, 50, 173, 52, 23, 133, 116, 214, 32, 172, 62, 11, 121, 252, 229, 212, 249, 93, 27, 199, 218, 12, 180, 95, 146, 103, 203, 172, 127, 209, 63, 161}

And a bad key (from the datastore):

a := []byte{18, 32, 210, 15, 82, 56, 1, 13, 206, 69, 227, 152, 104, 48, 96, 162, 187, 1, 48, 51, 64, 139, 20, 179, 147, 247, 116, 23, 27, 243, 234, 159, 39}

The bad key is 33 bytes long while the good key is 34.

[whyrusleeping]

There is a bug in the encode/decode code here: https://github.com/jbenet/go-datastore/blob/master/flatfs/flatfs.go#L50

some of the filenames are shorter than the others, a byte is being left off somewhere in the process.

[jbenet]

@whyrusleeping good work.

• could it be the 1: in encode: https://github.com/jbenet/go-datastore/blob/master/flatfs/flatfs.go#L51
• there's no equivalent in decode: https://github.com/jbenet/go-datastore/blob/master/flatfs/flatfs.go#L67

[jbenet]

before merging a fix for this, we should try to make a test that repros it.

[whyrusleeping]

found it: any key that ends in the byte 47, aka / will be stripped off by this: https://github.com/jbenet/go-datastore/blob/master/key.go#L50

[jbenet]

@whyrusleeping yep. makes sense. we need our own clean (+test). silly not to have considered it before.

[whyrusleeping]

this is needed before dev0.4.0 to ensure a correct migration

[MichaelMure]

Would something like this do the trick ?
MichaelMure@ddfbb5d
(yes, it would need to be backported to go-datastore)

The other way I thought would be to somehow have escape character to avoid the clean of /, but it start to be invasive.

[jbenet]

@whyrusleeping i think you had a plan for this, right?

[whyrusleeping]

This has been resolved, it was the flatfs path.Clean bug