RWStore
The RWStore provides re-allocatable file storage for BigData. It is intended to mirror the malloc, realloc and free memory management functions of the standard C library but for persistent storage. The RWStore scales to 4TB of data. The MemoryManager does similar things for main memory.
The RWStore has two different modes for handling recycling. One is based on a "session protection" mechanism. The other is based on deferred frees.
The session protection mechanism will immediately recycle historical data during any commit in which there are no open transactions. This is very efficient, but a database with a heavy concurrent query workload and no "quiet" periods will never recycle anything as there will never be a commit without a concurrent transaction which is still open.
** Note: The session protection mechanism is no longer supported. **
The deferred frees mechanism will cause the addresses of the records to be deleted to be logged on a chain of delete blocks for the journal. During the commit protocol, the delete blocks associated with commit points older than the earliest active transactions which are also greater than the minReleaseAge will be processed and the associated records recycled.
The minReleaseAge property (com.bigdata.service.AbstractTransactionService.minReleaseAge) controls which mechanism is used to recycle records. The default value is ZERO (0). When this value is zero, recycling only occurs when there are NO active transactions (session protection). The deferred frees mechanism is enabled by specifying a positive value. For example, a value of ONE (1) for the minReleaseAge will cause recycling of commit points older than the greater of one millisecond ago and the earliest open transaction.
The minReleaseAge property is read from the properties file for the Journal on start up. It is possible to change the value through a journal restart.
You do need to make a decision for your application between these two recycling mechanisms.
- If you have a heavy concurrent query workload without any appreciable "quiet" periods then you should set the MIN_RELEASE_AGE property to the #of milliseconds of history that you want to retain. If you do not want to retain history, then set it to ONE (1). That will enable the deferred frees recycling mechanism and ensure that storage is recycled even under a heavy concurrent query workload.
- If you want to retain a fixed amount of history, then set the MIN_RELEASE_AGE to that many milliseconds.
- If you do not want to retain access to historical commit points AND your application has "quiet" periods in which there are no concurrent readers, then you can set the MIN_RELEASE_AGE to ZERO (0). This will enable session protection.
To achieve this it provides an allocation method similar to some malloc implementations, with pre-allocations of arrays of fixed size slots.
-
FixedAllocators - The allocators themselves require persistence and these are managed with a meta-allocation strategy
-
MetaBits - The store can be configured with varying allocation sizes, but there will probably always be a need to be able to store data larger than the largest fixed allocation. Any allocation requirement larger than the largest defined allocator size requires a Blob implementation.
-
Blobs - How the store grows, both in file size and in the extension of the allocator and meta allocator structures, and how it integrates with the commit strategy requires its own explanation
-
Store Growth - The role and functioning of shadow allocation contexts for handling concurrent writers against the RWStore.
-
Allocation Contexts - How the RWStore recycles allocation slots, both within a native allocation context and across commit points.
-
Recycler Protocol
