Skip to content

Address search performance improvement for cc modules

dimxy edited this page Jul 30, 2020 · 9 revisions

updated: key and value changed

Trying to work out a db soluton to solve the performance problem when cc addresses are searched. It depends on use cases. I could see two main use cases:

  1. getting list of cc utxos and their data on a ccaddress (including marker addresses and usual cc addresses)
  2. searching for utxos with planid (creation txid) of a cc instance on an address (like in AddTokenInputs for tokenid) (It could be also a use case of getting spent utxos on a ccaddress. I almost never use this, maybe we could do without it)

We have two options in the codebase: leveldb and berkeleydb. Leveldb is used for indexes and berkeleydb is used for the wallet. Leveldb is very simple and allows to store key value and enumeration of values by the key or by its part. As leveldb stores in a sorted order it provides logarithmic complexity. Berkeley db supports b-tree or hash access. The hash access provides O(1) complexity and it is good for the second use case. But considering that we need also getting all utxos I think leveldb seems sufficient. We might create a special leveldb-based index db of unspents designed for cc modules, basically for those 2 use cases above. The key and value structure of this db:

Key:
  evalcode - maybe not needed as it could be several evalcodes in a vout and no need to duplicate the entry for each one
  ccaddress - address of cc vout
  creationid - id of the initial cc transaction
  txhash - cc transaction txid
  index - cc vout number
note: decided not to have funcid and version in the key (unlikely someone will search by funcid)    

Value:
  satoshi
  height
  spk with OP_DROP data
  opreturn data (only if it exists for this evalcode)
  funcid
  version - opret version

We create several functions:

void SetCCUnspentsV2(uint8_t evalcode, char *address, std::vector<CCIndex> &outputs, int start, int end, int max); // gets all outputs for a cc address, using a partial key
void SetCCUnspentsV2(uint8_t evalcode, char *address, uint256 creationid, std::vector<CCIndex> &outputs, int start, int end, int max); // gets outputs for a cc address and creationid, using a more extended key. This is needed to quickly find all outputs for this creationid on this address (like AddTokenCCInputs function does)

The functions will have params of start and end heights and max number of results. (I think we should not return millions from a marker address and provide a way how to get the results partially).

I think if we have rules defined where the evalcode and creationid are stored in the opreturn or opdrop we could make the unified code updating this db (in main.cpp), without modifying each cc's code.

Also it seems this db could be done only for the new cryptoconditions lib version2 for which we could get the evalcode(s) from tx cc outputs and check if the opreturn matches the evalcode. The cryptoconditions lib v2 also allows to ensure that only valid cc transactions are added to the chain and removes the need to validate the already added referred transactions, so the info in the cc index will be valid

Clone this wiki locally