Data Management Capability (DMC) Project

Sub-pages: camtek_kaijaytu | Scripts | Common | DMCCore | Bridge | HardwareLogic | Design Examples

This repository contains the architecture, source code, and documentation for the Data Management Capability (DMC) system, designed for high-performance semiconductor equipment control.

Tech Stack

Languages

• C++17
• C# (.NET)

Communication

• P/Invoke
• gRPC

Design Patterns

• Singleton Pattern — Ensure all clients access the same DMC Server instance.
• Strategy Pattern — Each Data Element implements its own Print behavior; DMC does not need to know internal details.

Build System

• CMake
• MSBuild

Testing

• Unit Test
• Integration Test

Platform

• Windows
• Linux

Dependencies

• C++17 compatible compiler
• .NET SDK
• CMake 3.20+
• gRPC
• Visual Studio 2022
• GCC / Clang

Architecture Diagram

System Layer Architecture

+-----------------------+
|     User Interface    |
+-----------+-----------+
            |
            v
+-----------------------+
|       DMCCore         |
|      (C# Layer)       |
+-----------+-----------+
            |
            v
+-----------------------+
|        Bridge         |
|   (P/Invoke / gRPC)   |
+-----------+-----------+
            |
            v
+-----------------------+
|    HardwareLogic      |
|      (C++ Layer)      |
+-----------------------+

Client-Server Architecture

+----------+   +----------+   +----------+
| Client A |   | Client B |   | Client C |
+----+-----+   +----+-----+   +----+-----+
     |              |              |
     +--------------+--------------+
                    |
                    v
          +--------------------+
          |    DMC Server      |  <-- Singleton Instance
          | (Object Registry)  |
          +--------------------+
          | - Register Element |
          | - Update Element   |
          | - Print Element(s) |
          +--------+-----------+
                   |
        +----------+----------+
        |                     |
+-------+-------+    +-------+-------+
|  Data Store   |    | Print Engine  |
| (Dictionary)  |    | (Polymorphism)|
+---------------+    +---------------+

DMC Overview

The DMC is a capability identified as part of a systems infrastructure. It acts as a Server responsible for handling all kinds of objects (Data Elements) input by Clients, without needing to know the internal details of those objects.

Responsibilities

• Storage of Data Elements with a well-defined data structure.
• Allowing multiple users (Clients) to input and manage their data.
• Providing the ability to print out all objects currently in the system.

DMC Requirements

#	Requirement	Description
1	Register Data Element	Accept and store a new Data Element if it does not already exist in the system.
2	Update Data Element	Modify an existing Data Element if it is already registered.
3	Print Data Element	Output the contents of a Data Element without the server knowing its concrete type.
4	Multi-type Storage	Allow storage and management of different types of Data Elements per user.
5	Open for Extension	No redesign required when adding a new type of Data Element.

Data Element Design

Interface Architecture

Data Element behavior is decomposed into two focused interfaces, combined into IDataElement:

+-------------------+     +-------------------+
| IKeyIdentifiable  |     | IPrintable        |
+-------------------+     +-------------------+
| + GetKey(): string|     | + Print(): void   |
+-------------------+     | + ToDisplayString()|
        \                 +-------------------+
         \               /
          v             v
     +---------------------+
     |    IDataElement      |
     +---------------------+
     | + Type : string      |
     +---------------------+
              ^
              |
     +--------+---------+
     | GenericDataElement|
     +------------------+
     | + Properties      |
     | + KeyProperty     |
     +------------------+

Interface	Responsibility
IKeyIdentifiable	Each element decides its own unique key via GetKey(). The server does not impose an ID.
IPrintable	Each element decides how to display itself via Print() and ToDisplayString().
IDataElement	Combines both interfaces + exposes Type.

GenericDataElement

A single class that accepts any type name and arbitrary key-value properties. No new class is needed for new types.

register → Type: Animal
           Properties: Species=Dog, Name=Buddy, Age=3
           Key property: Name
           Generated Key: "Animal:Buddy"

Feature	How it works
Type	User-specified string (e.g. Car, Person, Animal, Bus)
Properties	Arbitrary Dictionary<string, string> key-value pairs
Key	Auto-generated as Type:KeyPropertyValue (e.g. Car:Toyota)
Key property	User chooses which property is the unique identifier
Print	Outputs all properties with the key: [Car] Make=Toyota, Year=2024 (Key: Car:Toyota)

Extensibility (Open-Closed Principle)

To add a new Data Element type at runtime:

1. Enter any type name (e.g. Bus, Desktop, Airplane).
2. Enter key-value properties.
3. Specify which property is the unique key.

The DMC Server requires zero modification — it only depends on IDataElement.GetKey() and IPrintable.Print().

DMC Server Design

Data Structure

DMC Server
├── _instance : DMCServer              (Singleton reference)
└── _registry : Dictionary<string, IDataElement>
                 Key = element.GetKey() (e.g. "Car:Toyota")
                 Value = IDataElement instance

• Uses a single Dictionary<string, IDataElement> as a shared data pool.
• All Clients (modules) read and write to the same registry.
• Key is generated by the element itself via GetKey() — not imposed by the server.
• Provides O(1) lookup, insertion, and update.

Why Shared Data Pool (Not Per-Client Isolation)

The DMC serves as a centralized data registry within a single equipment system. Multiple software modules (Clients) need to:

• See the same global state (e.g., module A writes inspection results, module B reads them for scheduling).
• Print out all objects in the system as a unified view.
• Maintain a single source of truth — avoiding conflicting copies of the same data.

Per-client isolation would turn each module into an island, breaking cross-module collaboration.

How DMC Knows If a Data Element Exists

Register(element):
    key = element.GetKey()       // Element decides its own key
    if _registry.ContainsKey(key):
        -> Route to Update
    else:
        -> Add to _registry

The server calls element.GetKey() to obtain the key, then looks it up in the dictionary. The element itself determines what makes it unique (via IKeyIdentifiable).

How DMC Prints Without Knowing the Data Element

The server calls Print() through the IPrintable interface. Each element provides its own implementation (polymorphism).

PrintAll():
    for each element in _registry.Values:
        element.Print()   // Dispatched via IPrintable

The DMC Server never casts to a concrete type — it relies entirely on the IPrintable interface contract.

How All Clients Access the Same DMC Server (Singleton)

class DMCServer:
    private static _instance : DMCServer = null
    private static _lock : object = new object()

    public static Instance:
        get:
            if _instance == null:
                lock(_lock):
                    if _instance == null:
                        _instance = new DMCServer()
            return _instance

• Private constructor prevents external instantiation.
• Thread-safe double-checked locking ensures a single instance.
• All Clients call DMCServer.Instance to get the same server reference.

Design Assumptions

The following assumptions are made and validated for each design step:

#	Assumption	Justification
1	Each Data Element generates a globally unique key via GetKey().	Key is composed of Type:KeyPropertyValue, ensuring uniqueness within the system.
2	The system runs within a single process.	Allows in-memory Singleton pattern; if distributed, would need a service registry instead.
3	Data Elements are not deleted, only registered and updated.	Assessment only specifies Register/Update/Print; Delete is out of scope.
4	Print() outputs to console/log (text-based).	Assessment asks to "print out" without specifying serialization format.
5	All Data Element types share a composed interface (IKeyIdentifiable + IPrintable = IDataElement).	Required for polymorphic key generation, print, and type-agnostic storage.
6	All Clients share a single data pool (no per-client isolation).	Equipment modules need cross-module visibility; assessment requires "print all Object in System".
7	The system is thread-safe for concurrent client access.	Multiple clients may call Register/Update simultaneously; Singleton uses double-checked locking.
8	New Data Element types are added at runtime without code changes.	GenericDataElement accepts any type name and arbitrary properties; no recompilation needed.

Features

Core Capabilities

• Data Element management

  • Register, update, and print Data Elements through a unified interface.
  • Manage object states and lifecycle without knowledge of concrete types.

• Hardware abstraction

  • Separate low-level hardware operations from higher-level application logic through a dedicated C++ layer.

• Cross-language interoperability

  • Enable communication between native C++ modules and managed C# components through P/Invoke and gRPC.

Reliability & Performance

• High-performance hardware control

  • Provide deterministic and low-latency interaction with semiconductor equipment.

• Fault handling

  • Detect, report, and recover from hardware and communication failures.

Development & Maintenance

• Modular architecture

  • Separate business logic, hardware logic, communication, and infrastructure layers.

• Automated testing

  • Support unit tests and integration tests.

• Automated deployment

  • Simplify build, test, and deployment workflows through scripts.

Test Scenarios

Register Data Element

Scenario	Input	Expected Result
Register new element	GenericDataElement("MobilePhone", {Brand="Apple", Model="iPhone"}, "Brand")	Successfully added, Key: MobilePhone:Apple, registry count +1
Register duplicate key	Same type + same key property value	Routed to Update, returns true
Register different type	GenericDataElement("Car", {Make="Toyota", Year="2024"}, "Make")	Successfully added as separate entry, Key: Car:Toyota

Update Data Element

Scenario	Input	Expected Result
Update existing element	Update MobilePhone:Apple Brand to "Samsung"	Property updated successfully, returns true
Update non-existing element	Update key Car:Unknown	Returns false (element not found)

Print Data Element

Scenario	Input	Expected Result
Print single element	Print("MobilePhone:Apple")	Outputs: [MobilePhone] Brand=Apple, Model=iPhone (Key: MobilePhone:Apple)
Print all elements	PrintAll()	Outputs all registered elements in sequence
Print after update	Print("MobilePhone:Apple") after update	Reflects updated values

Singleton Guarantee

Scenario	Operation	Expected Result
Multiple clients access	Client A and B both get Instance	Same object reference (ReferenceEquals == true)
Concurrent registration	Client A and B register simultaneously	Both elements stored, no data loss

Open-Closed Principle

Scenario	Operation	Expected Result
Add new type	Create GenericDataElement("Desktop", {...}, "Brand") and register	DMC handles it without code change
Print new type	PrintAll() after adding Desktop	Desktop element output included

Project Structure Overview

The project is organized into multiple directories to provide clear separation between source code, documentation, and automation tools.

Data-Management-Capability/
├── camtek_kaijaytu/      # Core source code and build environment
├── misc/                 # Documentation and reference materials
└── scripts/              # Build, test, and deployment automation

Directory Details

1. camtek_kaijaytu/ (Core Package)

This directory contains the main source code and development environment.

camtek_kaijaytu/
├── build/
├── docs/
│   └── design/
├── src/
│   ├── Bridge/
│   ├── Common/
│   ├── Config/
│   ├── DMCCore/
│   ├── HardwareLogic/
│   ├── Logger/
│   └── ThirdParty/
└── tests/
    ├── test_main.cpp                    # C++ fault injection tests
    ├── HardwareLogicConcurrencyTests.cs # C# concurrency stress tests
    ├── GrpcIntegrationTests.cs          # gRPC client-server integration tests
    ├── DMC.Tests.csproj                 # .NET test project (unit/concurrency)
    └── DMC.IntegrationTests.csproj      # .NET test project (integration)

Source Code

HardwareLogic/

• Native C++ components responsible for hardware control and real-time interaction.

DMCCore/

• C# modules responsible for equipment object management, event processing, and application logic.
• Contains the DMC Server implementation (Singleton, Registry) and gRPC service.

Bridge/

• Interoperability layer connecting native C++ components and managed C# components using P/Invoke and gRPC.

Common/

• Shared utilities, common data structures, and reusable components.
• Contains the IDataElement interface and base abstractions.

Config/

• Configuration management and runtime settings.

Logger/

• Logging subsystem for diagnostics, debugging, and system tracing.

ThirdParty/

• External libraries and third-party dependencies.

Documentation

docs/

Contains design documents and technical references.

• design/

  • Detailed design documents, examples, and technical specifications.

Testing

tests/

Contains automated testing for the HardwareLogic native library.

• C++ Fault Injection Tests (test_main.cpp)

  • Null pointer handling, invalid buffer sizes, illegal state operations
  • HW_GetLastError thread-safety validation (multi-threaded)

• C# Concurrency Stress Tests (HardwareLogicConcurrencyTests.cs)

  • 100-thread parallel read/write, Init/Shutdown races
  • Read storm, throughput measurement, re-initialization cycles

• gRPC Integration Tests (GrpcIntegrationTests.cs)

  • End-to-end client-server Register, Update, Print, PrintAll, BatchRegister
  • Server streaming, client streaming, error handling

Run all tests: ./scripts/test/test.sh

Build Output

build/

Stores generated binaries and build artifacts.

---

2. misc/ (Documentation Archive)

Contains supporting documents, external specifications, assessment reports, and reference materials.

misc/
├── assessment/
└── specifications/

assessment/

• Evaluation reports, feasibility studies, and design assessments.

specifications/

• Hardware specifications and external reference documents.

---

3. scripts/ (Automation and Deployment)

Contains automation scripts used throughout the development lifecycle.

scripts/
├── build/
├── deploy/
└── test/

Build Scripts

Located in:

scripts/build/

Examples:

• build.sh
• build.bat

Compile the complete solution for Linux and Windows environments.

Test Scripts

Located in:

scripts/test/

Examples:

• test.sh
• test.bat

Execute unit tests and integration tests.

Deployment Scripts

Located in:

scripts/deploy/

Examples:

• deploy.sh
• deploy.bat

Deploy binaries and configuration files to target environments.

Getting Started

Clone the Repository

git clone https://github.com/kaijaytu/Data-Management-Capability-DMC.git

Build on Windows

cd scripts/build
build.bat

Build on Linux

cd scripts/build
chmod +x build.sh
./build.sh

Roadmap

• Initialize repository structure
• Design IDataElement interface and base abstractions
• Implement DMC Server with Singleton pattern
• Implement Register / Update / Print operations
• Implement Data Element Factory (replaced by GenericDataElement)
• Implement logging subsystem
• Implement configuration management
• Build hardware abstraction layer
• Implement event dispatcher
• Integrate gRPC communication (code complete, pending RHEL verification)
• Add fault injection tests (C++ null pointer, buffer overflow, illegal state)
• Add concurrency stress tests (C# 100-thread parallel R/W, race conditions)
• Add integration tests for Client-Server scenarios (gRPC end-to-end)
• Add CI/CD pipeline

Notes

This repository is a personal project created for learning and portfolio purposes.

Some implementation details have been simplified or anonymized.