The SceneTrack Process
SceneTrack is a temporal and scene based recorder. It organises information into objects and components. The states and changes of states of those objects and components are neatly recorded in equal slices of time called Frames.
An Object is an instance of a Class, which is a description of a Thing that has components; Such as a Transform. A component is a single property of that Object such as localPosition.
A component can be any of these types
- Unsigned and Signed 8-bit integer
- Unsigned and Signed 16-bit integer
- Unsigned and Signed 32-bit integer
- Unsigned and Signed 64-bit integer
- Floats and Doubles
- Strings
With the exception of strings; Values can be in scalar form, or vector form (upto 4 elements). Values can be as an array or as a single value.
Classes are created through stCreateObjectType which you get a class Handle in return, likewise a component is created through stAddObjectTypeComponent. There are multiple versions of these functions to suit the type of components and objects as you like.
A Class can have upto 32 components, and an object is an instance of a class. There can be up to 16,777,215 objects in a SceneTrack file. Depending on memory and disk spaces, Scenetrack can have upto 2,147,483,647 frames.
Objects and Components are created, deleted and changed through Command Submission. These can be done any time during a frame, and in as many frames as you like. There are a number of commands to do this, but they boil down to three kinds; stCreateObject, stDestroyObject and stSetValue_xyz.
A Command Key, presents the meta information about a change of value, new objects, delete object or any object orienated command issued to SceneTrack during frame submissions.
A Command Key is a 64-bit structure which is passed around as a 64-bit integer, but is made up of the following components.
| Byte | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 |
|---|---|---|---|---|---|---|---|---|
| Bit | 76543210 |
76543210 |
76543210 |
76543210 |
76543210 |
76543210 |
76543210 |
76543210 |
| Segment | FFFFFFFF |
OOOOOOOO |
OOOOOOOO |
OOOOOOOO |
CCCCCCCC |
TTTTTTTT |
FFFFFFFF |
SSSSSSSS |
Key:
- F = Frame modulus
frameNum % 256 - O = Object Id (24-bit)
- C = Component Id (8-bit) or
255when command is New or Delete - T = Command Type (see
stCommandType) i.e. New Object, Delete Object, SetValue_uint8, ... - F = Flags
- S = Custom Size (Num of Bytes * 2)
The Command Data is a set of memory allocated via a memory pool. The size is inferred by the command type, or custom size (when not 0). Data Size is always rounded up to the nearest 4 bytes. For each command submission each command data is laid out in memory linearly. The maximum size is 1024 bytes per Command Data. Data always points to the first argument.
Examples:
| .. | 0 | 3 | 4 .. |
|---|---|---|---|
| .. | 0x42 | Alignment Padding | .. |
| .. | 0 | 2 | 4 .. |
|---|---|---|---|
| .. | 0xCAFE | 0xBEEF | .. |
The Command Buffers are where the commands are written to during a frame submission. They are usually fixed sized in length, and have a maximum command submission size of 65535 commands, although the naximum memory of the computer and data submitted during that frame is also a factor.
Commands can be submitted in any order, but are sorted by a radix sort to make sure that the order of them make sense. This 'sense' is the numerical value of stCommandType, as well as other factors in the stCommandKey.
Each Command submitted has four components.
- A Key - stCommandKey, representing the meta information about the submitted Command.
- Value Index - A numerical id, representing the field in a the buffer data array.
- Data Pointer- A pointer to the memory.
- Memory - The actual command parameter information itself.
Keys, value Indexes and data pointers are allocated in first-come-first served format. Memory is allocated via dynamic pools, although the memory allocated will be in one-chunk, it's neighbouring command may have memory allocated from another pool.
Once a frame has been submitted stSubmit. The commands are sorted using a radix sort. Commands that belong to the same object will be grouped together (sub-organised by Component), with New Object coming first in that group, and a Delete coming last.
The Buffer then is swapped with another free buffer, so it can be serialised in to the SceneTrack format.
On another frame submission, the data then is written into "Delta Frames". A Delta frame, is a frames worth of changes to a Scene. Looking at a Deleta Frame in the middle of a file is worthless, because the proceeding frames gives the Deleta Frame some context, as it doesn't store the state of the frame, it stores the change of state of each frame.
Delta Frames are organised into hot and cold data. Hot data are frame numbers, frame counts, what objects have changed in the frame. Colder data is what object information has changed during the frame.
When saving, SceneTrack will deal with hot and cold data at the same time, when loading and iterating. Hot data is persistant in memory, but cold data is only read from the file when needed. This method of seperating data, saves a tremendous amount of memory and processing time.
SceneTrack saves and loads DeltaFrames through the stFileWriter and stFileReader interfaces. These in turn have interfaces for the file formats that SceneTrack supports; Binary and ASCII (Experimental). The Binary file format saves the SceneTrack data in a key-value store, organised by chunk headers. Each chunk header has a FourCC code and the size of chunk. This allows fast iterating over the data, to quickly navigate over colder data.
Iterators stIterator are used by the Exporters, to iterate over the data. It will iterate through each component, object and then frame in terms of hierarchy. Because everything is sorted; Object components are organised together allowing the exporter to quickly convert the data into it's own format in a linear manner.
The FBX, Midi and Video exporters all share the same exporter and iterator interfaces. This is a light C++ wrapper code around the SceneTrack iterator interface. It complements it with a Schema system to match Object Component names to Exporter Objects, and provides helper functions to convert values into the correct formats and types.