Home
Welcome to the LunarBase Engine wiki!
LunarBase engine targets a real time analysis and free-style database engine, managing 2 billions records in one table, where each record has a size limitation up to 32k bytes. Therefore 64 TB data can be stored and queried via one table of LunarBase.
The key challenge for a server end software is the memory fragments generated after a period of time running. Fragments slow down the system, unless user restart the OS, the resource will be eaten up soon or later. LunarBase has a memory management unit, named LunarMMU, embedded in LunarMax, to solve this pain point by none-pause garbage collection technique, with which we are able to manage hundreds of GB memory off the JVM heap in practice.
On the JVM side, garbage collection technique brings fast engineering but causes pause in big throughput applications. When we handle billions of object via network/memory, GC may takes even more time than dealing with the data itself. Using sun.misc.Unsafe is an option to precisely control the memory, serializing/de-serializing objects to the off-heap memory. But oracle has no guarantee that the Unsafe interfaces will be in the future JVM releases, since it is not following the JAVA philosophy.
LunarMMU offers a better solution in memory control, letting JAVA programmers to delete/free the memory that they newed/allocated, just like what C/C++ programmers do. Therefor the efficiency of data process has an insurance.
Life cycles of objects, containing the data you care, now is under the control of programmer again, based on which, LunarBase breaks the restriction of JVM and pushes the boundary of performance to what the bare mental has.
Not only the memory that generates garbage, search engines, file systems, or any database storage system also create fragments. Since as of now, underlying hardware is block based device, the fragmentation is inevitable. Search engines like lucene or sphinx perform defragmentation after a certain period of writing, re-organizing the disk blocks by moving small pieces of data into a contiguous space.
In Lunarbase, disk defragmentation is amortized during the service running. When the backend thread detects the fragments reaching a threshold, it starts up to move the relevant data to a internal queue, waiting to be merged into a new contiguous block space, and marks these fragments as available for new small piece of data writing.
Administrators are enabled to configure a big direct memory caching hot data for quick access. Never need to deploy another third party cache solution. It is not only cost effective, but also much more important for data consistency and validation. These are pains-in-coding if applications have no choice but independent outer cache solutions. Consult why internal big cache for an elaboration on this subject.
Since memory is a critical resource for server applications, Lunarbase allows you to configure extra disk(better be a SSD) as swap space, avoiding resource racing with OS or other services. Using SSD as cheaper memory is the instruction on how to configure Lunarbase to do so.
LunarBase has no pre-defined schema for your data. You may extend your data model at any time to meet your business updates. The records in LunarBase is in this format: "{name=jackson, payment=500, age=36}" , while any property can be added or removed in the future use. Chapter insertion has more discussion on this subject.
Besides the basic CRUD(Create, Read, Update, Delete) operations that a database shall provide, operations like JOIN in traditional RDBMSs, are supported via materialized view by LunarMView. Materialized view separates read and write, then greatly accelerates query speed, and gets eventually consistency for any data updates. For very busy sites with data modeled by tons of 1-to-1, 1-to-many, many-to-many relations, LunarMView is the suitable solution in managing this.
Materialized view is not the only solution for data consistency. When embedding LunarBase to a distributed system, consistency may be achieved via distributed message system, 2-phase commitment or its enhancement Percolator transaction model(from google). All are candidates for a distributed transaction solution.
For records stored, user queries any column by specifying its value he seek. Queries are "payment=300", "payment=300 AND 10<= age <=25", or something like these. These functionalities are supported by LunarMax real time engine. Consult chapter "real time computation" for its mechanism and usage.
Vertical scalability, real time computation, big memory and none-pause garbage collection, join relational tables, transactional log, these features together make LunarBase your best choice for your forever-online business.
Following the right side navigation bar, you will learn how to use Lunarbase, how it works, what the benifits are. Or you may jump to the 10 minutes start here to get a quick experience.
Knowledge base for LunarBase-0.8 in pdf can be found here, from which you will have the blueprint of the engine architecture.