Document on the IPFS Repo Spec, the things we store on Datastore

[daviddias]

No description provided.

[SidHarder]

@diasdavid :There are four keys in the go-ipfs datastore leveldb database:

key= /F5UXA3TTF4JCB5OTOYTDCBGWYFARRBGGIPJR4KKXVXBGOHK4C5MB47SOZHQM3LH6
key= /F5YGWLYSED25G5RGGECNNQKBDCCMMQ6TDYUVPLOCM4OVYF2YDZ7E5SPAZWWP4
key= /local/filesroot
key= /local/pins

The top two keys are unique to the repo and change each time init is run. The bottom two are static. The values for these keys seem to be encrypted. As an example, the data for the first key is:

Y��9_)a���˹2�R�m�Ŗke�9����

Can you help me figure out how to decode the values?

[dignifiedquire]

@SidHarder why did you close this issue?

[jbenet]

The last two are not encrypted, i believe these are hashes to look into the blocks. The first two may be routing records. also likely an object hash.

[Kubuxu]

We in 0.4.3 (I think) shifted from binary keys which were getting corrupted by our sanitation operations to base32 encoded keys.

[daviddias]

@Kubuxu I believe you just changed that in blockstore, not datastore, right?

[Kubuxu]

no, datastore too as it was also problem with IPNS records where keys are hashes of public key.

[SidHarder]

OK, I am making some progress:

The value for the key '/local/filesroot' is the hash of the root folder. The value is binary. The go function is core/core.go:563.

The value of this key should match the result of the command: 'ipfs files stat --hash /', (however, I have not been able to make this work yet)
Also, the value is a binary cid?

[SidHarder]

@dignifiedquire I'm sorry I didn't mean to close it.

[Kubuxu]

Yes, the value is binary CID, v1 probably in new version and v0 in an older ones.

[SidHarder]

The "/local/pins" key:

go-ipfs reference: pin/pin.go:450

[SidHarder]

Im trying to verify the /local/filesroot key, when I use the js-ipfs-cid class as below the result I get is: 2Dkx8D3WqjChhe3ZGUvQ2Jkj5LT428yvFGzv3KD6c1Pz which doesn't match the result of "ipfs files stat --hash /" which is QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn. Not sure what I'm doing wrong.

var levelup = require('levelup')
const CID = require('cids')

var ops = [{
valueEncoding : 'binary'
}]

var db = levelup('../../.ipfs/datastore', ops, function (err, db) {

var key = '/local/filesroot'		

db.get(key, function (err, value) {
	if (err) {
		if (err.notFound) {
			console.log(err)
			return
		}
		console.log(err)
		return callback(err)
	}	
				
	var buffer = new Buffer(value, 'binary')
	var cid = new CID(buffer)
	var result = cid.toBaseEncodedString()
	console.log(result)		
})

})

[SidHarder]

@Kubuxu Can you help me understand why when I retrieve the value for the "/local/filesroot" key and convert it to base58 I get this: 2Dkx8D3WqjChhe3ZGUvQ2Jkj5LT428yvFGzv3KD6c1Pz instead of QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn which is the result of "ipfs files stat --hash /".

Also, I am retrieving the key/value pair and converting it using Node.

[Kubuxu]

I just dumped this key from go-ipfs:
and it is: 122059948439065f29619ef41280cbb932be52c56d99c5966b65e0111239f098bbef which is QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn

[Kubuxu]

Also the key you sent has quite a weird pattern at the end hex FDFDFDFD at the end, suggesting some corruption.

[SidHarder]

@Kubuxu It must be how level up is retrieving the value. I see if I can figure it out. Thanks for your help.

[SidHarder]

OK the code below will extract the /local/filesroot key and base58 encode it. The extracted value matches the result of ipfs files stat --hash. The "valueEncoding: binary" is really important.

const level = require('level')
const bs58 = require('bs58')
const db = level('../../.ipfs/datastore', { valueEncoding: 'binary' })

db.get('/local/filesroot', function (err, value) {
if(err) {
console.log(err)
}
else{
console.log(bs58.encode(value))
}
})

[daviddias]

awesome progress @SidHarder :) Wanna start making the PR that adds that levelDB store and tests?

[SidHarder]

@diasdavid I'll get started.

[SidHarder]

@diasdavid Does it make sense to have a datastore.js module in js-ipfs-repo/src/stores? This module could be used during the init process to add the necessary keys and values.

[daviddias]

@SidHarder absolutely. Just have in mind that it needs to be LevelDB for nodejs and indexedDB for when in the browser

[daviddias]

We still need this list, @SidHarder could you PR to the README the current keys that ipfs uses to store things in the repo?

[SidHarder]

@diasdavid the keys that Go IPFS stores in the Datastore are:

key= /F5UXA3TTF4JCB5OTOYTDCBGWYFARRBGGIPJR4KKXVXBGOHK4C5MB47SOZHQM3LH6
key= /F5YGWLYSED25G5RGGECNNQKBDCCMMQ6TDYUVPLOCM4OVYF2YDZ7E5SPAZWWP4
key= /local/filesroot
key= /local/pins

I'm sorry but I don't understand what you mean by PR to the README.

[Kubuxu]

The first two records are:

 echo F5UXA3TTF4JCB5OTOYTDCBGWYFARRBGGIPJR4KKXVXBGOHK4C5MB47SOZHQM3LH6 | bases base32 bin
/ipns/� ��v&1���A���C��)W��g�\�X�~N��ͬ�%
 echo F5YGWLYSED25G5RGGECNNQKBDCCMMQ6TDYUVPLOCM4OVYF2YDZ7E5SPAZWWP4 | bases base32 bin
/pk/� ��v&1���A���C��)W��g�\�X�~N��%      

Using https://github.com/whyrusleeping/bases

[daviddias]

@SidHarder Would you like to wrap this task and add the findings to the Readme?

[daviddias]

//cc @pgte