TDBv0.3: transactions, indexes, durability

[krizhanovsky]

Tempesta DB must be reworked:

• index must be separated from data management and both of them must be accessible from user-space, such that tdbq can directly delete and insert data to support tdbq: full select, insertion and deletion of binary data #279; Issue Web-content management tool #528 (domain and wildcards purging) require index like patricia trees forest.
• TDB must support plugable indexes, e.g. logs and web-cache requires different indexing;
• table format must be reworked such that additional indexes can be created on top of current data. Correspondingly queries (lookups) must be able to choose which index to use;
• data manager must support large contiguous allocations (up to extent size);
• data and table managers must be rewored to support dynamically growing/shrinking and created/deleted tables, so freed space must be properly reused. However, there could OLTP tables with round-robin and/or eviction strategies which still must be fixed-size. This should be configurable for particular table format This is the tradeoff between non-pageable memory for the new real time persistent database and flexibility (see comments in alloc.c
• durability: TDB must writeback updates in background threads in transactional manner, presumably using BtrFS mechanisms;
• disk-aware load distribution for machines with many different storage hardware. IO must be distributed by disck IOPS, space utilization, vhost, It makes sense to provide an interface for plugable IO schedulers
• records must be introduced for structured data storage;
• last hit timestamps and hit counter are probably should be stored in HTrie nodes to support TDBv0.2: Cache background revalidation and eviction #515;
• NUMA code should be moved from Web-cache to TDB, so IP filter code must be made NUMA-aware;
• trigger callbacks for DML must be implemented (DELETE in sense of Reducing origin server requests #500 for automatic cache entries refreshing). The callbacks can be treated as update events (like MongoDB or MySQL binary log events).
• TDB must support rollback of failed transaction - if a particular update inside a transaction fails, then all previous updates must be rolled back. Required for consistent configuration updates (Full/per-vhost dynamic (re)configuration: gRPC API #67). It seems the online reconfiguration is the only user for transactions and it's doubtful that we really need real transactions just for online reconfiguration, so TBD.
• optional cache warmup (also see Huge pages allocation issue and the crash on cache sizes >=2GB #1515)

Also with persistent memory probably there is no need to copy skb data to web cache in tfw_cache_copy_resp(), see Extending Linux network stack for persistent memory.

Linked with #66 (tdbfs). Virtual containers can be supported either on TDB level or tdbfs - TBD.

The task is crucial for Web-server mode, so keep in mind and update issue #471 during the work on the task.