add index of internal blocks

[xrdavies]

• add index of internal blocks

It's based on #409

Create index for all internal blocks in RAM with hash and address of internal blocks to avoid too many page faults when doing rbtree searches, since internal blocks are stored in mmap memory which is actually on disk . It will take about 5G extra RAM when use tmp files, but speed up the loading and syncing process, performance is similar with -z RAM option.

With 8G RAM and SSD, it will take about 7 hours to load storage, syncing is still in testing right now.

[sgaragagghu]

there is no need to add complexity everywhere, you can easily just hide index in tree insert, remove and search function, for example:

static inline struct block_internal *block_by_hash(const xdag_hashlow_t hash)
{
	struct block_internal_index *index = (struct block_internal_index *)ldus_rbtree_find(root, (struct ldus_rbtree *)hash - 1);
	return ((index) ? index->bi : NULL );
}

static inline int block_insert(struct ldus_rbtree **root, struct block_internal *nodeBlock)
{
	struct block_internal_index *index = calloc(sizeof(struct block_internal_index),1);
	if(index == NULL){
		return 0;
	}
	memcpy(index->hash,nodeBlock->hash, sizeof(xdag_hashlow_t));
	index->bi = nodeBlock;
	ldus_rbtree_insert(root,&index->node);
	return 1;
}

and you can do block_remove the same way like example above

[xrdavies]

there is no need to add complexity everywhere, you can easily just hide index in tree insert, remove and search function, for example:

static inline struct block_internal *block_by_hash(const xdag_hashlow_t hash)
{
	struct block_internal_index *index = (struct block_internal_index *)ldus_rbtree_find(root, (struct ldus_rbtree *)hash - 1);
	return ((index) ? index->bi : NULL );
}

static inline int block_insert(struct ldus_rbtree **root, struct block_internal *nodeBlock)
{
	struct block_internal_index *index = calloc(sizeof(struct block_internal_index),1);
	if(index == NULL){
		return 0;
	}
	memcpy(index->hash,nodeBlock->hash, sizeof(xdag_hashlow_t));
	index->bi = nodeBlock;
	ldus_rbtree_insert(root,&index->node);
	return 1;
}

and you can do block_remove the same way like example above

Yes, good suggestion.

If we name it as stage 1.0 for the previous implementation, this is stage 1.5, not stage 2.0 yet. After this version, I will try to introduce a new cache mechanism to switch cold/hot data in RAM to reduce page faults. As the cache usage is quite low when do syncing, switch cold/hot data would give another boost of performance.

[jonano614]

Can you just use for(int j = 0; j < i; j++)?

[xrdavies]

Sure, any other comments with this PR?

[sgaragagghu]

Waiting for index to be hidden in tree functions to keep the code tidy (and to avoid to add possible bugs in all that tricky changes with index->bi... )

[xrdavies]

@sgaragagghu
Here is thought next step, to cache hot bi in RAM rather than in mmap memory. It's easy to hide index->bi