|
Democrite |
Democrite is an open-source framework for building robust, scalable and distributed 'multi-agent like' system based on Microsoft Orleans.
Caution
The development is still in beta phase
Democrite offers an automated orchestration and configuration system, enabling dynamic creation, editing, and modification of grain interactions.
It incorporates built-in Features to manage virtual grains and facilitate communication between them. This simplifies the creation of various Sequences of virtual grains that can be transformed an input to output. Democrite utilizes Signals to transmit information and Triggers to initiate different Sequences, thereby chaining processes through a graph. Most Features are describe in a serializable structure.
Democrite functions as a layer on top of Microsoft Orleans, ensuring that all features of Orleans are retained.
- Scalability using silo as virtually one server
- Robustes through virtual actor model
- Simplicity by grain design
All configurations possible with Orleans are still available, but Democrite offers a simplified, fluent configuration model.
- Nodes : Backend part where all the VGrain live to solve requests.
- Client : Proxy used to send query to cluter's node.
- Cluster : Group of client and nodes to distribute work allong multiple devices.
- Storages : Define where and how the data are persisted.
- Virtual Grains (vgrain) : Democrite extention of orleans grain.
- Virtual Grains Id : Democrite dynamic usage of id and templating.
- Sequences : Linear transformer that chain VGrain to treat a input data and provide an output from it.
- Signals : Information send through the cluster that could be used as stimulus. (Trigger, Services ...)
- Triggers : Connectors designed to initiate reactions based on a stimulus. (Signal, Door, Cron, Stream Queue, ...)
- Doors : Connectors designed to initiate reactions based on stimulus that follow specific conditions.
- External Code VGrain : VGrain write in other technologie than .net, like Python, C++, Javascript, ...
- Blackboard : Technologie of shared memory with intelligent controller. Use to organize and process a specific task, goal.
- Redirections : Enable switch of implementations for a particular contract during runtime.
- Dynamic Definitions : Allow definition to be generated and inject at runtime.
Tip
In green this is the lastest feature integrated
A node refers to a server within a cluster.
It is recommended to establish a client (server API) along with multiple nodes that process requests.
This approach enables scaling of the processing component while maintaining a simplified facade.
Democrite node has an individual setup.
var node = DemocriteNode.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.ClusterFromConfig()
.ConfigureLogging(c => c.AddConsole());
...
});
await using (node)
{
await node.StartUntilEndAsync();
}Tip
Next Incomming: IHostBuilder integration to allow configuration on existing application.
A client is a separate program that utilizes the cluster's capabilities.
It requires configuration to locate the cluster and send requests.
It is advisable to set up the client and the cluster's nodes on different machines.
Democrite client could be setup individually or through existing application setup.
Individually
var node = DemocriteClient.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.ClusterFromConfig()
.ConfigureLogging(c => c.AddConsole());
...
});
await using (node)
{
await node.StartUntilEndAsync();
}Integrated
// Add democrite client
builder.Host.UseDemocriteClient(cfg => { ... });Like Orleans, Democrite is build to work in cluster. Client and nodes communicates to each other to solve request.
This solution provide a strong resilience because if one node falldown automatically his work is relocate to other nodes.
To build a cluster you need a way for each new node to discovert the other and share information like status or workload.
Different strategy exist, Orleans choose the database meeting point.
When you work in local you just need to configure using (NoCluster) :
var node = DemocriteNode.Create(cfg =>
{
cfg.WizardConfig()
.NoCluster() // -> Setup the system to only bind to 127.0.0.1 and allow only local clientBy default, in opposed of Orleans way, the local configuration allow multiple local nodes to form a cluster.
I you want to prevent multiple nodes you have to add the following configuration:
var node = DemocriteNode.Create(cfg =>
{
cfg.WizardConfig()
.NoCluster() // -> Setup the system to only bind to 127.0.0.1 and allow only local client
.AddEndpointOptions(new ClusterNodeEndPointOptions(loopback: true, siloPort:50000))Nuget package : Democrite.Framework.Node.Mongo
You can use mongo db as a Meeting point.
To do so you just have configure :
var node = DemocriteNode.Create(cfg =>
{
cfg.WizardConfig()
// Setup mongo db as cluster meeting point
.UseMongoCluster(m => m.ConnectionString("127.0.0.1:27017"))Same go for the client part
builder.Host.UseDemocriteClient(b =>
{
b.WizardConfig()
.UseMongoCluster(m => m.ConnectionString("127.0.0.1:27017"))Important
Attention by default a democrite node doesn't allow client to connect to it. (Except in NoCluster configuration use to dev)
it's a security to prevent any node to be consumed by client.
To enabled the node to be open to client you have two choose:
- Add manually a gateway port : .AddEndpointOptions(new ClusterNodeEndPointOptions(gatewayPort: 4242)) // It will conflict if you have multiple node in local
- Let system choose a free port as gateway : .AddEndpointOptions(new ClusterNodeEndPointOptions(autoGatewayPort: true))
See the full usage in sample ExternalStorages/MongoDB
A Democrite cluster need some information to be stored:
- Cluster information (cf. Cluster)
- VGrain State
- Reminder clock information
- Definitions (Sequences, triggers, signal, ...)
- Custom Data
When you write a VGrain you can inherite from VGrainBase{TVGrainInterface} or VGrainBase{TSTATE, TVGrainInterface}.
The second one provide a State management for you. It means the state will be load/create when the VGrain is activate and store when it is desactivate.
Tip
In case of crash, the state storage could not be garanty, to prevent this you can call in you code the method "PushStateAsync"
The issue with that is the time consumtion
To store the state you need to provide through the constructor a IPersistentState{TState}.
The good practice is to get this instance from dependency injection with tag.
public CounterVGrain(ILogger<ICounterVGrain> logger,
[PersistentState("Counter")] IPersistentState<CounterState> persistentState) In this example we request a storage place to store a CounterState object.
Using the attribute PersistentStateAttribute we customize "How" it will be stored.
First parameter is the Storage name (for example the table or collection) and you can specify a second parameter as the Storage configuration Key.
Tip
You can request by constructor as many storage as you want.
It's an easy way to managed the data storage
Nuget package : Democrite.Framework.Node.Mongo
You can use the mongo extension to provide storage for all the components:
-
Cluster information -> cf. Cluster
-
Definitions (Sequences, triggers, signal, ...)
// Define mongo db as definition source
.AddMongoDefinitionProvider(o => o.ConnectionString("127.0.0.1:27017"))- State/Custom storage (VGrain State, Reminder clock information, Custom Data)
// Setup mongo db as default storage
.SetupNodeMemories(m =>
{
// Enum StorageTypeEnum provide all sub type
// You can customize the database name, the connection string ...
// By default it will reuse the LAST connection string configured
m.UseMongoStorage(StorageTypeEnum.All);
});See the full usage in sample ExternalStorages/MongoDB
In accordance with Orleans terminology, a grain is a virtual actor that can appear in any compatible silo, with its state being restored if necessary.
In Orleans, to invoke a grain, one must request a proxy instance from the IGrainFactory. This proxy seamlessly manages the communication between the caller and the called.
This is why a grain consists of an interface and an implementation, allowing the proxy to inherit the interface.
With Democrite, there is no need to explicitly call the grain yourself, it will do it for you based on the configuration.
This is the reason we refer to them as Virtual Grains (VGrain), to denote a behavior that prevent direct call consumption.
A Sequence is a series of virtual grains executed sequentially, where the output of one can be used as input for the next VGrain.
The aim is to set up this sequence description just once and save it in a database. To run the sequence, only its unique identifier (Uid) is required.
Important
Currently, only local declarations are supported. Please refer to the Next section for information on future capabilities to load these configurations from a storage system, such as databases.
To configure and test sequences you need to create and register it in the DemocriteNode configuration.
Build definition
var collectorSequence = Sequence.Build()
// Ask a web URI in input
.RequiredInput<Uri>()
// Fetch html page and return it
.Use<IHtmlCollectorVGrain>().Call((a, url, ctx) => a.FetchPageAsync(url, ctx)).Return
// Configure inspector on specific pair inspect and extract current value
.Use<IPriceInspectorVGrain>().Configure(currencyPair)
.Call((a, page, ctx) => a.SearchValueAsync(page, ctx)).Return
// Store the value received into a dedicated statefull grain
.Use<ICurrencyPairVGrain>().Configure(currencyPair)
.Call((a, data, ctx) => a.StoreAsync(data, ctx)).Return
.Build();Register definition
var node = DemocriteNode.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.NoCluster()
.ConfigureLogging(c => c.AddConsole())
.AddInMemoryMongoDefinitionProvider(m =>
{
// Local in node memory setup
.SetupSequences(collectorSequence);
})A Sequences can be executed manually, but it can also be triggered automatically.
There are differents kind of triggers :
- Time Periodicity, use a cron expression to define the periodicity
- Signals, trigge when configured signal is also fire
Similar to sequences, trigger definitions can currently be created and stored locally, with plans for future storage in external sources like databases.
Important
Currently, only local declarations are handled. The upcoming goal, detailed in the Next section, is to enable loading these configurations from a storage source.
A trigger can supply an input to initiate the sequence.
Important
Currently, only static data collection is supported. Please see the Next section for information on the future goal of loading these configurations from an external provider.
Time Periodicity
// Every minutes between 9h and 18h UTC between monday and friday
var triggerDefinition = Trigger.Cron("* 9-18 * * mon-fri")
// Define what will be trigged (Sequence or signals)
.AddTarget(collectorSequence)
// You could have many target or many types
//.AddTarget(collectorSequence2)
//.AddTarget(collectorSequence3)
// Define how to get input information that will be send to targets
.SetInputSource(input => input.StaticCollection(collectionsources)
.PullMode(PullModeEnum.Circling)
.Build())
.Build();Signals
// listen inputSignal and trigger when this one is fire
var signalTriggerDefinition = Trigger.Signal(inputSignal)
// Define what will be trigged
.AddTargetSequence(collectorSequence)
.SetInputSource(input => input.StaticCollection(collectionsources)
.PullMode(PullModeEnum.Circling)
.Build())
.Build();Register definition
var node = DemocriteNode.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.NoCluster()
.ConfigureLogging(c => c.AddConsole())
.AddInMemoryMongoDefinitionProvider(m =>
{
// Local in node memory setup
m.SetupTriggers(signalTriggerDefinition);
})Use an EBS (Entreprise Bus Service) as storage an diffuseur of job to to.
Orlean can nativaly used different type of ESB.
With Democrite we create connector through trigger to push and pull.
Send to stream :
// PUSH
var trigger = Trigger.Cron("*/25 * * * * *", "TGR: Push Every 25 sec")
.AddTargetStream(streamDef) // <----- Add stream as target
.SetOutput(s => s.StaticCollection(Enumerable.Range(0, 50))
.PullMode(PullModeEnum.Broadcast))
.Build();Consume from stream :
// PUSH
var fromStreamTrigger = Trigger.Stream(streamDef) // <----- Trigger that fire until theire is a message in the stream queue
.AddTargetSequence(consumeSeq.Uid)
// Limit the number of concurrent execution, Prevent consuming all messages without resources in the cluster to process them (CPU, RAM, ...)
.MaxConcurrentProcess(2)
.Build();The signals feature consists of two components:
- Signal
- Door
A signal functions similarly to an event, but with a "fire and forget" approach.
By default, the signal includes:
- Definition name & Uid
- The VGrain information that fire
- The datetime when it is fire
- The possible previous signal that cause this one to fire
However, it can carry a small amount of information.
It is recommended to keep this information as minimal as possible to avoid memory issues.
It could be something as simple as an ID referencing data in storage.
Define a signal:
var signalA = Signal.Create("signalA");A Door can listen to multiple signals and, based on specific conditions, can emit its own signal.
Currently, a boolean logic door is available, but you can easily create and configure your own gate logic.
Define a Logic boolean door:
var door = Door.Create("CheckPairAboveAverage")
.Listen(valueEurUsdStoredAboveAverage, valueEurChfStoredAboveAverage)
// Basic
// Fire If (A & B) are fired in a 10 sec window
// By default the door unlock as soon as the condition is valid and
// signal activation are only use one.
.UseLogicalAggregator(LogicEnum.And, TimeSpan.FromSeconds(10))
// Advanced
//.UseLogicalAggregator(b =>
//{
// return b.Interval(TimeSpan.FromSeconds(0.5))
// .AssignVariableName("A", valueEurUsdStoredAboveAverage)
// .AssignVariableName("B", valueEurChfStoredAboveAverage)
// .AssignVariableName("C", manualForceDoorFireing)
// /* Fire (if A and B are signal in an interval of 0.5 second except if i was already fire in less than 0.5 seconds)
// Or
// C
// */
// .Formula("(A & B & !this) | C");
//})
.Build();- LogicalAggregator : Apply a boolean condition based on signal activation (1 if activate on period of time, otherwise 0)
- RelayFilterDoor: Apply a condition a the signal structure itself, use a filter to trigger specific sequence
Register definition
var node = DemocriteNode.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.NoCluster()
.ConfigureLogging(c => c.AddConsole())
.AddInMemoryMongoDefinitionProvider(m =>
{
// Local in node memory setup
m.SetupSignals(t => t.Register(signalA))
.SetupDoors(t => t.Register(door));
})In the Orleans framework, a grain definition can have multiple virtual instances.
However, only one instance is active at any given time, associated with a unique identifier known as a GrainId.
In Orleans, it is the user's responsibility to supply the correct GrainId of the grain they wish to call.
In Democrite, virtual grains are instantiated and called by a generic orchestrator.
By default, a new Guid is used each time, which is ideal for stateless grain.
Virutal Grain interface could be tag by attribute VGrainIdFormatAttribute to indicate how to build the GrainId.
The template ID system provides the ability to dynamically create a GrainId using data input or execution context as the source of information.
You can see a good example in the sample Forex.
This one use stateless virtual grain (vgrain) to download html page and parse it
but use a statefull virtual grain (vgrain) to store the value extracted.
This virtual grain (vgrain) employs a string value, such as a forex pair (eur-usd, eur-chf, etc.), from the execution context to form its GrainId, resulting in the creation of a single reusable instance for each pair
This allow :
- A client to directly call this vgrain to extract the values.
- To create only one grain by pair that is single-thread handled by orleans (no need to think of concurrent access)
- Store information in dedicate a serializable model in class and not to focus on the storage mode (databases, files, ...)
Tip
You can access those grain usign the classic orleans IGrainFactory way.
BUT it is better to use IDemocriteExecutionHandler who will use correctly and automatically the correct GrainId.
With Democrite we can use other satellite program or script to perform jobs.
To do so you need simple steps:
- Create Code artifact definitions
- Using through a vgrain
- Use generic vgrain in the library IGenericArtifactExecutableVGrain
- Create your own vgrain by inherite from ArtifactExecutableBaseVGrain<> instead of VGrainBase
A python package exist to handle the communication protocol with democrite README.md
You can found a full sample Here
A blackboard is a temporary shared space with specific controllers to solve a problem.
The goal is to group specific analyze result and have controllers deciding the next step to acheived to a solution.
You can found a full sample Here
Democrite has devised a dynamic solution for determining the implementation to be utilized for a contract.
There are several reasons why multiple implementations might be necessary.
Now, with the introduction of a feature called "Redirection" you can dynamically alter the implementation used for a specific contract.
It is usefull for:
- Testing: Ensure that implementation won't break existing sequences
- Competition: You can them make implementation in competition to select the best related to context.
- ...
This redirection could be setup a different level:
- Call Scope: During a call using IDemocriteExucutorHanlder. Limit the redirection only to a specific call, you can even specialize to a specific stage.
- Global Scope: Available on all the cluster
You can found a full sample Here
Democrite serves as an orchestrator for vgrains, utilizing serializable definitions as work orders.
These definitions are primarily formulated and stored in an external system such as databases.
The objective of the "Dynamic Definitions" feature is to generate definitions (sequences, triggers, etc.) at runtime, with a lifecycle linked to storage.
If no specific storage is provided, the definition will exist only for the duration of the cluster's lifetime.
This capability enables dynamic testing of new sequences, creation of temporary triggers, and activation of temporary debugging features, among other functionalities.
You can found a full sample Here
Important
For now democrite is in beta version. Don't forget to use the pre-release flag in visual studio
- Democrite.Framework.Node: Reference this one by your node project.
- Democrite.Framework.Client: Reference this one by your client project.
- Democrite.Framework.Builder: Reference this one by your project that build definitions.
If you split the agent implementation and definition in separate projet you could only reference the nuget package Democrite.Framework.Core
Framework Feature
- Democrite.Framework.Node.Cron: Reference this one by your node project to enable the cron mechanism.
- Democrite.Framework.Node.Signals: Reference this one by your node project to enable the signals mechanism.
- Democrite.Framework.Node.StreamQueue: Reference this one by your node project to enable the stream mechanism.
- Democrite.Framework.Node.Blackboard: Reference this one by your node project to enable the stream mechanism.
Extensions
- Democrite.Framework.Extensions.Mongo: Reference this one by your node project to enable the mongo db Storage.
Bags
- Democrite.Framework.Bag.DebugTools: Reference this one by your node project to enable debug sequences or VGrain (Like Display, ...).
- Democrite.Framework.Bag.Toolbox: Reference this one by your node project to enable basic tools sequences or VGrain (like delay, ...).
To create a node you just have to follow the example bellow.
Caution
Orleans scan by default all the project dll. Due to .net assembly load behavior if you deport your agent implementation in another projet is may not be loaded if you don't directly use any of the type defined. Reference the project is not enough. In the Next section you will see an objectif to reference assembly to load for now you have to use the SetupVGrains method in the wizard configurator.
In Program.cs:
var node = DemocriteNode.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.ClusterFromConfig()
.Configure(b => b.ConfigureLogging(logging => logging.AddConsole()));
...
});
await using (node)
{
await node.StartUntilEndAsync();
}To create a client you just have to follow the example bellow.
Caution
All nodes and clients need a meeting point to know the others and form a cluster, orleans choose the database strategy. By default only one node and one client could be present on the same machine wihtout any db setup. But You could use the orleans fluent method to configure your cluster and client. You can now use different database solution for all storage type look for #Cluster section
In Program.cs:
var node = DemocriteClient.Create((ctx, configBuilder) => configBuilder.AddJsonFile("appsettings.json", false),
cfg =>
{
cfg.WizardConfig()
.ClusterFromConfig()
.Configure(b => b.ConfigureLogging(logging => logging.AddConsole()));
...
});
await using (node)
{
await node.StartUntilEndAsync();
}To execute a sequence or call a specific grain you have to use the service IDemocriteExecutionHandler.
This handler follow democrite rules in grain id generation.
- Normalize your data model and create small Virtual Grain with small sponsability.
- Follow the SRP (Single Repsonsability Principle) as describe in the S.O.L.I.D pattern
- If you attach information to signal use small one like simple id.
- Prefer democrite configuration if possible to prevent any side effect non managed
In the section Sample/Forex
Use case
Fetch reguraly a forex pair value using public web site, store the values and be able to consume them through an api.
Features Use
- Democrite sequence definition
- Democrite Cron Trigger definition
- Democrite automatic Virtual Grain Id
- Orleans persistant state
- Minimal .net API
- Democrite client IDemocriteHandler usage
Use case
Create a node cluster with a client connected through external storage. In storage we will also store definitions of a trigger that will every minute increment a counter tag with a name. Through the client you can look for the counter value through a swagger api
Features Use
- Democrite sequence definition
- Democrite Cron Trigger definition
- Democrite automatic Virtual Grain Id
- Storage:
- Cluster information (client/node)
- Definitions
- Reminder informations
- Orleans persistant state
- Minimal .net API
- Democrite client IDemocriteHandler usage
Mongo
In the section samples/ExternalStorages/MongoDB
In the section Sample/RelayFilterDoor
Use case
Relay filter door can apply a condition on signal received. For example if the signal transport a value of type int, let pass.
- Trigger dedicated sequence if function of the data carry
- Trigger dedicated storage based to the type of data carry
Features Use
- Democrite RelayFilterDoor definition
In the section Sample/PythonVGrains
Use case
Use Python scripts inside democrite environment like VGrain standard
Caution
In version 0.2.2 only local python script deployed with the democrite node as 'Content' is supported
You goal is to be able to support different package provider,
in different format late on. (From api, zip, shared folder, resource embedded ...)
- Use RNN (Neural Network) library in python to solve data
- Re-use you librairies
Features Use
- Democrite Artifact Definition, Packaging
- External code executor (allow democrite to used external program as solver)
In the section Sample/Blackboard
Use case
Use a Blackboard as central point to group information related to a specific case and compute them when needed.
We demonstrate with a simple calculator. We can store in the blackboard different type of numeriacl value and sum them when a limit is reach or manually.
- Process Different modality (Text, Video, Image) in a same place and use dedicated controller to perform advance analyze, remove useless data, archive some others ...
- Train an IA by storing good data set and rejet others, detect automatically when quality response is reach
Features Use
-
Blackboard Storage
- Rules
- Type check
- Multi-Storage
-
Blackboard Controller
- Storage : Responsable of the data integrity
- Event : Responsable to compute when condition are fullfill.
In the section Sample/Streams
Use case
Use a Stream as data input for democrite activity through a specific trigger.
- Buffer a lot of work in a StreamQueue
- Communicate with external system in and out
Features Use
- Stream source trigger
In the section Sample/Redirections
Use case
- Wanted to change the behavior of some implementation whitout impacting existing.
- Be able to put in competition multiple implementation an see the best result
Features Use
- Grain Implemation redirection
- Local during a call
- Global for all the cluster
In the section Sample/DynamicDefinition
Use case
- A blackboard need some data processing, it can generate it's own sequence and signal definition at runtime
- You want to test a VGrain in some condition you can create you sequence using it
- You pilote IA want to test different algorithm variation
Features Use
- Dynamic definition
- Injection
- Activation/Deactivation
- Suppression
v 0.4.1-prerelease:
Release Node
- Migrate to .net 8
- Force grain redirection for a precise sequence execution
- Call sequence from sequence
- Create sequence definition in runtime
v 0.3.0-prerelease:
Release Node
- Blackboard extensions
- Create bags as container of generic (toolbox, debug)
- Repository : global & simple storage system not linked to a grain state
- Process through a foreach a sub properties collection in a sequence
- Trigger use to push or pull data from a stream
- Fire signal from sequence
v 0.2.2-prerelease:
Release Node
- Fully integrate Python VGrain in Democrite environment
v 0.2.1-prerelease:
Release Note
- Easy cluster external storage to store virtual grain state, reminder, membership ...
- IHostBuilder integration to allow configuration on existing application.
- Load Definition, sequence, signals, triggers, ... from an external source like databases using the design pattern strategy through IProviderSource
| Democrite Version | Minimal .net version | Minimal orlean version |
|---|---|---|
| Lastest | .net 8.0 | 8.0.0 |
| 0.3.0 | .net 7.0 | 7.2.4 |
| 0.2.1 | .net 7.0 | 7.0.3 |