Home
Welcome to the blade wiki!
This is the wiki for programmers. User Wiki (Engine editor) is here: https://github.com/crazii/blade/wiki/Editor-Wiki
The Blade engine is inspired by this article:
Designing the Framework of a Parallel Game Engine (Intel)
This article may help your understanding the architecture of the engine, although parts of it are not designed as the article says.
Table of Contents
- Memory management overview
- Foundation overview
- Framework overview
- Tasks overview
- Data binding overview
- Graphics subsystem overview
- Try Your own plugin
In blade, IPool is abstract interface and its implementations are not always actual pools. Blade have multiple strategies of pools, based on object's lifetime. The main purpose is too maximize allocation speed & minimize memory fragmentation.
The commonly used "pooling" tech for fixed sized objects. Use
BLADE_FACTORY_CREATE(IPool, BTString("FixedSize")); //or PoolFactory::getSingleton().createInstance(BTString("FixedSize"));to create fixed sized pools.
The pool with fast speed allocation. Equivalent to the stack-based frame allocator in popular game engines. Used for mainly 2 type of life-time:
- temporary objects: freed right away in scope.
- static objects: never be freed (until app exit).
The implementation is pretty simple & fast: allocated blocks are never re-used (that what 'incremental' means).
Use
BLADE_FACTORY_CREATE(IPool, BTString("Incremental")) // or PoolFactory::getSingleton().createInstance(BTString("Incremental"));to create a custom incremental pool.
Usually it is not needed to create a custom one, but access the global ones is enough. Use Memory::getTemporaryPool() to get the global temporary incremental pool, and use Memory::getStaticPool() to get the global static incremental pool.
This type of pool is implemented by using DL-Malloc. It's used for more complicated cases, i.e. large object that are not suitable for fixed sized pooling. Use
BLADE_FACTORY_CREATE(IPool, BTString("Resource")); //or PoolFactory::getSingleton().createInstance(BTString("Resource"));or
BLADE_FACTORY_CREATE(IPool, BTString("Misc")); //or PoolFactory::getSingleton().createInstance(BTString("Misc"));to create a custom one. Or use Memory::getResourcePool() to access the global one. Each resource intense module using it's own created resource pool is recommended.
Holds registry for custom-created pool objects. Currently not in use.
Base memory classes with custom operator new/delete are used to simplify codes. To use the required strategy of pooling, simply inherit from a base object.
note: Careful when using those base classes: they are not template classes but unique classes (Actually Blade previously uses Allocatable<T>) and may cause undesired size for objects. i.e. under some circumstances EBO(Empty Object Optimization) will fail due to C++ spec.
class Quaternion : public Allocatable{};
class DualQuaternion : public Allocatable { Quaternion q1; Quaternion q2;};
static_assert(sizeof(Dualternion) == 32, "unexpected size");
//the above assertion will fail, because DualQuaternion contains 3 Allcatable subobjects,
// 2 of them are overlapped,
//and C++ spec guarantee the subjects "have different address", that makes extra memory to avoid address overlapping.
//Usually it's not a problem unless compact data is needed to gain cache performance,
//or those data may be in uniform/constant buffer or passed to rendering API as shader constants,
//or make sure an explicit size when used to serialization.a base class that utilize the Fixed Size pools. Different sized objects are allocated in different fixed sized pools.
a base class that utilize the global temporary pool (Incremental).
a base class that utilize the global static pool (Incremental).
a base class that utilize the global resource pool(Resource/Misc).
Blade has std::allocator compatible utility to customize std containers. Like the base classes above, different types of containers are defined to easily use the pooling strategy.
Vector/List/Map/Set, etc. Use fixed-sized pooling to allocate elements.
TempVector/TempList/TempMap/TempSet etc. Use the global temporary pool to allocate elements. Usually used in local/function scope.
StaticVector/StaticList/StaticMap/StaticSet etc. Use the global static pool to allocate elements. Usually used inside a global/static object, i.e. application global database/table/config.
Note: vector may have reallocation. Careful on using StaticVector, if you use it without pre-allocation(reserve/resize), reallocation may happen, and previously allocated memory may not actually freed (based on Incremental pool's behavior) and will be wasted.
So use reserve/resize for StaticVector is almost mandatory. if the size is hard to decide, use a TempVector to create a fully initialized one, then copy it to a StaticVector.
First of all, loop containers are temporary containers. They will auto clear at the beginning/end of each app-main-loop iteration.
Frame containers are almost the same except they will clear on each rendering, since multiple rendering(frames) may happen during a single loop, especially in editor with multiple rendering-window/view-port.
BLADE_RES_ALLOC/BLADE_RES_ALIGN_ALLOC: use the global resource pool to allocate memory. BLADE_TMP_ALLOC/BLADE_TMP_ALIGN_ALLOC: use the global temporary pool to allocate memory. BLADE_STATIC_ALLOC/BLADE_STATIC_ALIGN_ALLOC: use the global static pool to allocate memory.
Memory allocated with BLADE_TMP_ALLOC/BLADE_STATIC_ALLOC should be freed manually. Theoretically they can be auto freed (like a stack-based frame allocator in popular game engines). But Blade choose to explicit free memories in respect of C style allocations/de-allocations. Besides it will be more safe if you change BLADE_TMP_ALLOC to other functions that do need explicit free (you don't need to remember to add free function when changing it).
Use of BLADE_TMP_ALLOC is not recommended, use TempVector<T> instead. BLADE_TMP_ALLOC need to free by BLADE_TMP_FREE, while TempVector<> will auto perform cleaning up by its nature.
Technical notice on foundation libraries.
In Windows (previously Android too), all modules are built into DLL/so, so that each module have initialization routine on loading, to perform static data initialization. Thus the init order is controlled by DLL loading order. the StaticDataInit.cc defines objects to control static object init order in foundation.
Some init function are used for static linking(i.e. currently Android, iOS in the future),
- intializeFoundation();
- initializeFramework();
- initializeAppFramework();
- initializeGameLibrary();
Those function are not necessary for DLL/so builds (yet if removed, their content need to move to global scope), but to get max compatibility, they are always used.
When static linking is used, objects in foundation library are not guaranteed to init before other module, so some compiler tricks are used, i.e. attribute((init_priority(101)))
static Lock objects can be used without order restriction, just use StaticLock to avoid init order dependency. it'll perform default-init. They will be set to 0 (unlock state). example:
extern Lock thelock;
class A
{
A() { thelock.lock(); ... }
};
static A a; //use lock object defined in other compilation unit:
//there's no guarantee that object 'a' is initialized after object 'thelock',
//when 'thelock' is initialized after 'a', 'thelock' will set its states to initial states(unlocked)
//and that will overwrite the lock states. Anyways, it's UB when use thelock before its ctor.
//to avoid this, use StaticLock instead. StaticLock's ctor will do nothing like it is default-initialized.In computer programming, a handle is generally a reference to a resource. It can be a pointer(like Windows' HANDLE), a pointer to a pointer, or an integer number to the resource table. In Blade it is a simplified shared_ptr.
Factories in Blade are abstract factory, yet they have no abstract Factory interface or client implementation, it just wrapper the interface and implementation with an internal FactoryUtil::Creator, so that it is simplified and you don't need to implement a factory class.
Note: since factories are templates, they need to be DLL-exported before using it, or there'll be multiple instantiates and that won't work. i.e.
extern template class BLADE_BASE_API Factory<IFileDevice>;to export the instantiated template class of factory. And do remember to define it in your cpp file. i.e.
template class Factory<IFileDevice>;Note: the definition in cpp file is not needed in Windows(MSVC), as it will be DLL-exported, but for other platform/compilers, it is needed. So always do it.
Factories create objects with name. use
T* p = BLADE_FACTORY_CREATE(T, name); //or
T* p = Factory<T>::getSingleton().createInstance(name);to create the instance. BLADE_FACTORY_CREATE is recommended as it will record the file & line to help dump memory leaks.
use
RegisterFactory(ImplType, BaseType);to register your ImplType to the BaseType factory, with the name as "ImplType".
or use
NameRegisterFactory(ImplType, BaseType, name);to register your ImplType with a custom alternative name.
Interface singletons are singleton wrappers that utilize factory creation, so factory of the interface must be DLL-exported too.
However DLL-export of factory can be skipped(not linked in) if InterfaceSingelton<T>::getInterface() is used.
There're maybe more than one implementations for a interface singleton, so strictly speaking, interface singletons are not "singleton"s anymore. You can consider it a "global pointer" that can point to different singleton implementations.
use
InterfaceSingleton<T>::interchange()to change default implementation, or
InterfaceSingelton<T>::getOtherSingleton();to get other implementation. use
InterfaceSingelton<T>::getInterface()to get the interface without linking dependency to the implementation module (the module that DLL-exports the factory).
About devices: see Subsystems & devices.
IFileDevice is an abstraction of media, it maybe native file system, or memory data(MemoryStream), or a package, or network data transfer, etc.
the default implementation, created using BLADE_FACTORY_CREATE(IFileDevice, IFileDevice::DEFAULT_FILE_TYPE), is native file system.
the files in file device can be listed or searched, and then opened to be read or written (if it is not read only) using IStream interface.
Perform resources loading/unloading.
Support resource schemes (like URI) that can be remapped to different media locations (disk, package etc.).
Events can be registered and dispatched.
Event handlers can be added to a registered event.
Environment variables can be registered, changed, read.
Tasks can be added and executed.
App-defined hot keys can be added. When a hotkey (normal key with combination with ALT/CTRL/SHIFT etc) is triggered, the corresponding command will be executed, and the normal key press will be skipped/consumed.
ISubsystem is an abstraction in a macro level. i.e. an application(especially a game) may contains: graphics, sound, physics, AI, network, etc.
List of subsystems
- Window System
Manages OS visual windows. - UI System
manages input devices, and GUI events (in game GUI not implemented yet). - Graphics System
Graphics/rendering related. - Geometry System
manages object transform hierarchy. - Game System
Used for client game logic. - Network System (not implemented)
- Physics System (not implemented)
- Sound System (not implemented)
- AI System (not implemented)
IDevice is an abstraction for platform related APIs, i.e. disk I/O, Rendering, Sound, etc. It is not a map to physical hardware.
List of devices
-
Window device
wrapper for OS dependent visual windows. implementations:- Win32WindowDevice
- QtWindowDevice
Used in editor only. Will auto override/replace default window device in editor mode. - AndroidWindowDevice
- iOSWindowDevice (not implemented)
-
File devcie
Abstraction of media/transfers. implementations:- WindowsFileDevice
Windows native file system. - UnixFileDevice
*nix system native file system (POSIX API). - BPKFileDevice
BPK packages. - NetworkFileDevice (not implemented)
Used to debug app without installing the full data package, but loading them through network from PC instead.
- WindowsFileDevice
-
Input device
Abstraction for user inputs. implementations:- Win32KeyboardDevice
uses Window messages to handle keyboard input. - Win32MouseDevice
uses Windows messages to handle mouse input. - DInput8KeyboardDevice
uses DirectInput8 to handle keyboard input. - DInput8MouseDevice
uses DirectInput8 to handle mouse input. - QtKeyboardDevice
uses Qt to handle keyboard input. used in editor (UI featured with Qt) only. - QtMouseDevice
uses Qt to handle moue input. used in editor only. - AndroidTouchDevice
android touch screen handling. - iOSTouchDevice (not implemented)
- Win32KeyboardDevice
-
Render device
Rendering API abstraction. impelemtations:- D3D9RenderDevice
Direct3D9 rendering. - GLES3RenderDevice
OpengGL ES 3.0 rendering API. - VulkanRenderDevice (not implemented)
- MetalRenderDevice (not implemented)
- D3D9RenderDevice
-
Network device (not implemented)
-
Sound device (not implemented)
-
Physics device (not implemented)
A subsystem have one device or multiple, or none, based on implementation. Device implementations can use a 3rd party SDK, i.e. physics device using havok/newton/physx, a sound device using fmod, etc.
All device implementations are located in project BladeDevice, one exception is DefaultFileDevice(created with IFileDevice::DEFAULT_FILE_TYPE by factory) which is at BladeBase, with the purpose of easily using native I/O functions with minimal dependency, without additional dependency on BladeDevice.
It's just a simple ECS (entity component system), with components named as elements.
A subsystem may have multiple elements or none, based on implementation.
Stage is a generic concept in games.
Scenes are stages from subsystems' perspective. i.e. a stage can have a graphics scene, physics scene, etc. You can consider them as a specialized "stage entity" with "scene components".
Each elements of single entity may have its own resource, and may be loaded on different condition. Subsystems/plugins should register their own resource & serializer(loader) to the resource factory, using RegisterFactory or NameRegisterFacotry. i.e.
NameRegisterFactory(TextureResource, IResource, TextureResource::TEXTURE_RESOURCE_TYPE);
NameRegisterFactory(Texture2DSerializer, ISerializer, TextureResource::TEXTURE_RESOURCE_TYPE);
IResourceManager::getSingleton().registerFileExtension( TextureResource::TEXTURE_RESOURCE_TYPE, BTString("dds") );
IResourceManager::getSingleton().registerFileExtension( TextureResource::TEXTURE_RESOURCE_TYPE, BTString("png") );will register a texture resource and a serializer to the factory, and assign 2 extension to the texture resource type. The Resource Manager will create/load resources using the resource factory.
Blade uses tasks to execute specific work. A subsystem may have one or more specific type of task. i.e. graphics system have a graphics task which do the rendering job.
All task are executed in parallel, in random native threads on each loop. i.e. if one task is executed in thread A in loop 1, then it maybe in thread B in loop 2.
The task type has task affinity which allow one task to bound to a fixed thread (not recommended to use it). Tasks with the same type are queued in the same thread and executed in linear order.
After all tasks are executed, there'll be a sync point and then all tasks are updated in parallel.
The global variable GLOBAL_TASK_STATE records the current state of task execution.
- TS_MAIN_SYNC
On each loop start, all tasks are pending, only main thread is working. - TS_ASYNC_RUN
Tasks are running. - TS_ASYNC_UPDATE
Tasks are updating, parallel states sync here.
Illustration chart: tasks on each main loop
Parallel states are multiple buffered data holding by elements. i.e. graphics elements have position to render, and physics elements have simulated position too, there's a priority to decide which should use. After each tasks run, elements of different subsystem may have different data, and tasks update will sync those data.
Each element has one parallel state set. Parallel state set is a collection of parallel states with name map. The mapped name is a semantic shared among all elements. i.e. CommonState::POSITION is the position of an object, by using the same name, different states are linked & synced. Custom names can be defined and used for custom purpose.
Parallel states are grouped and so that later sync is more easy. Parallel states (of different elements) with the same name are grouped together, in one same group. Enity holds a set of parallel state groups, one group for each name.
Parallel state queues are used for sync parallel states, after tasks running, parallel states may change, if they are changed, they will put themselves into a queue, for later sync (during task update). Generally one subsystem have one queue, and parallel states of all elements within the subsystem are put into it and synced in linear order.
illustration chart: parallel state set, group, and queue
TODO:
Graphics system are mainly divided to two sections:
- 3d space
- rendering pipeline
The 2 sections are connected by Camera object, which is a object in space and is used for rendering. Note: Objects in spaces doesn't use scene graph (scene nodes), sub divided spaces may be node based hierarchy. There is Geometry System which uses node hierarchy: Nodes are used for dynamic geometries.
ISpace is an abstraction for subspace dividing & bounding hierarchy. It is used for culling & 3D queries, i.e. picking.
ISpaceCoordinator has 2 uses:
- Managing & Connecting sub spaces (i.e. portal)
- Trigger re-positioning: move spaces & objects for large worlds (to solve floating point coordinates precision issue).
Currently blade has one implementation of spaces: Quadtree space, Octree spaces may be added later.
3d objects that are located in spaces. i.e. lights, cameras, models, terrains etc.
Space contents contains one or more IRenderable that can be fill into render pipeline. A content can be a renderable itself: Graphics system only defines the interface, implement it whatever you want.
Space mask is used for loading optimization. When a scene is initially created offline, static objects will be put into the right tight bounding sub-spaces by bounding box intersection. Once that is done, bounding box intersection is not needed by later loading operation, unless it is moved in editor. Thus an unique id of the tightly enclosing sub space is recorded to the object as space mask, so that it will be used to fast locate the right sub-space (sub bounding in hierarchy).
TODO:
What you need to do (take BladeModelPlugin.cc, ModelPlugin::install() as example):
-
an element definition to be added to entity (for a graphics plugin, space content & renderables are needed too)
class ModelElement : public GraphicsElement { ... };
and register it to element factory.
NameRegisterFactory(ModelElement, GraphicsElement, ModelConsts::MODEL_ELEMENT_TYPE); -
register resources & serializers need by elements
NameRegisterFactory(ModelResource, IResource, ModelConsts::MODEL_RESOURCE_TYPE ); NameRegisterFactory(ModelSerializer, ISerializer, ModelConsts::MODEL_SERIALIZER_BINARY);
-
register resource extension (.blm)
IResourceManager::getSingleton().registerFileExtension( ModelConsts::MODEL_RESOURCE_TYPE, BTString("blm") );
-
add plugin support list & dependency:
//////////////////////////////////////////////////////////////////////////
void ModelPlugin::getSupportList(TStringParam& supportList) const
{
supportList.push_back( BTString("GraphicsModel") );
}
//////////////////////////////////////////////////////////////////////////
void ModelPlugin::getDependency(TStringParam& dependencyList) const
{
bool bDependency = IEnvironmentManager::getSingleton().getVariable( ConstDef::EnvString::WORKING_MODE) != BTString("tool");
if( bDependency )
dependencyList.push_back(BTString("GraphicsService"));
}And the plugin is almost done.
To use your plugin, add your plugin to Bin/Config_Editor/plugins.cfg.
In the app code (example in Demo.cc, class AnimDemoState):
- create model element that earlier registered by plugin
HELEMENT element = graphicsScene->createGraphicsElement(BTString(BLANG_MODEL_ELEMENT)); - add to an entity
const TString ELEMENT_NAME = BTString("Model"); entity->addElement(ELEMENT_NAME, element);
- set element resource (.blm)
EntityResourceDesc desc; desc.addElementResource(ELEMENT_NAME, BTString("bloodelf.blm")); - load entity & element resources
stage->loadEntitySync(entity, &desc);
P.S. the stage have a paging system that can auto streaming element resources for you. Different types of element can be configured in different streaming atom area size(granularity), and range.