Neuron Framework Documentation (PROJECT HAS BEEN ABANDONNED)

1. Overview

The Neuron Framework is a modular, real nanoservice coding framework built using Flask and SQLAlchemy. It implements what we call the Singular Data Theory—a design philosophy where a single, generic data table is shared among all projects (neurons) while each neuron augments this generic structure with its domain-specific fields and endpoints. Each neuron is implemented as an independent nanoservice with its own configuration, API endpoints, business logic, and Swagger documentation.

---

2. Key Concepts and Philosophy

Singular Data Theory

• Single Generic Table:
  The framework is built around a single generic table (or a small set of generic tables) that covers all business entities. For example, a model called GenericEntity is used to store data for users, orders, payments, etc.

• Neurons as Augmentations:
  Each business domain is encapsulated in a neuron (a nanoservice). A neuron uses the generic table but augments it with additional fields (via JSON columns and metadata) and exposes its own endpoints for domain-specific operations.

• Separation of Concerns:
  Core functionality—such as database connectivity, authentication, error handling, messaging, and metrics—is centralized in the core/ directory. Neurons (e.g., user_neuron, payment_neuron) focus solely on domain-specific business logic and API routing.

• Modular Deployment:
  Each neuron runs independently (with its own Flask app) and can be started, stopped, and managed individually via CLI commands. They can also be aggregated into a unified API documentation (via Swagger).

---

3. Project Architecture

Below is the updated directory structure of the framework:

neuron-framework/
├── core/
│   ├── __init__.py                
│   ├── adapters.py                # Database abstraction for various data providers (SQL, MongoDB, etc.)
│   ├── auth.py                    # Authentication endpoints and decorators.
│   ├── circuit_breaker.py         # Implements circuit breaker patterns.
│   ├── config.py                  # Global configuration and database connection (SQLAlchemy engine, SessionLocal, etc.)
│   ├── error_handlers.py          # Global error handlers for Flask.
│   ├── metrics.py                 # Initializes Prometheus metrics for monitoring.
│   └── ticket_logging_handler.py  # Logs critical errors to external ticket systems (Jira, GitHub, Trello).
├── generators/
│   ├── __init__.py                
│   └── generate.py                # CLI tools to generate new neurons and boilerplate code.
├── neurons/
│   ├── __init__.py                
│   ├── user_neuron/               # Example neuron for user management.
│   │   ├── __init__.py            
│   │   ├── config.py              # Neuron-specific configuration (overrides, metadata, etc.)
│   │   ├── metadata.json          # Default configuration values for this neuron (e.g., neuron_name, neuron_port).
│   │   ├── neuron_app.py          # The entry point for the neuron (initializes Flask, middleware, blueprints, Swagger UI).
│   │   ├── routes.py              # API endpoints (HTTP routes) for the neuron.
│   │   ├── services.py            # Domain-specific business logic for user operations (create, update, delete, etc.).
│   │   └── static/                
│   │       └── swagger.yaml       # Swagger/OpenAPI specification for this neuron's API.
├── celery_worker.py               # Asynchronous task processing using Celery.
├── aggregate_swagger.py           # Aggregates Swagger files from multiple neurons for unified documentation.
├── cli.py                         # CLI command module for managing the framework (DB operations, neuron process management).
├── Dockerfile                     # Containerization instructions.
├── k8s_deployment.yaml            # Kubernetes deployment configuration.
├── requirements.txt               # Python package dependencies.
└── README.md                      # Project overview and usage instructions.

---

4. Core Modules

4.1. core/config.py

• Purpose:
  Centralizes configuration settings by loading environment variables (using python‑dotenv) and creating connections to data providers.
• Key Features:
  • Builds SQL connection strings from individual variables (DB_USER, DB_HOST, etc.).
  • Creates a SQLAlchemy engine and sets a global SessionLocal for database sessions.
  • Supports multiple DB_TYPE values (SQL, MongoDB, Cassandra, JSON, CSV) dynamically.
• Quality:
  Follows DRY and security best practices by not logging sensitive credentials.

4.2. core/adapters.py

• Purpose:
  Provides an abstraction layer to connect to different data providers and manage database sessions.
• Quality:
  Ensures that the same interface is used across various databases.

4.3. core/error_handlers.py, core/metrics.py, core/auth.py, etc.

• Purpose:
  Provide global error handling, metrics monitoring (via Prometheus), authentication, and logging (including ticketing for critical issues).

---

5. Neuron Modules

Each neuron (nanoservice) is self-contained:

• config.py & metadata.json:
  Define neuron-specific settings such as neuron_name, neuron_port, and the publicly accessible server URL.
• neuron_app.py:
  The entry point that creates the Flask application, configures middleware (CORS, rate limiting, caching, distributed tracing, etc.), registers blueprints, and starts the server.
  It also integrates Swagger UI (serving the Swagger specification from the neuron's static folder).
• routes.py:
  Contains the HTTP endpoints for the neuron. For example, the UserNeuron defines endpoints for creating, updating, retrieving, and deleting user records.
• services.py:
  Contains the domain-specific business logic (e.g., input validation, user creation, update, deletion, notification). This module is imported by routes.py to separate HTTP logic from core business operations.
• static/swagger.yaml:
  Contains the OpenAPI specification for the neuron’s API. Each neuron has its own Swagger file stored in its static folder.

---

6. CLI and Process Management

cli.py

• Purpose:
  Provides command-line commands to manage the framework.
• Key Commands:
  • create_db, list_tables, create_table, delete_table, list_current_tables:
    For managing the SQL database and tables.
  • start_neuron, kill_neuron, start_all_neurons, kill_all_neurons, active_neurons:
    For managing neuron processes (starting and stopping individual neurons or all neurons, and listing active neurons).
• Quality:
  Uses standard libraries (subprocess, signal, sys) for process management and integrates with the framework's configuration.

---

7. Additional Components

• celery_worker.py:
  Handles asynchronous task processing.
• aggregate_swagger.py:
  Aggregates individual neurons’ Swagger files for unified API documentation.
• Generators:
  Tools to scaffold new neurons and related boilerplate code.
• Deployment Files:
  Dockerfile, k8s_deployment.yaml, and requirements.txt facilitate containerization, orchestration, and dependency management.

---

8. Configuration Files

• Global .env:
  Contains global settings such as database connection parameters, messaging system settings, logging configuration, Flask settings, etc.
• Neuron-Level .env.local:
  Each neuron (e.g., UserNeuron) has its own .env.local file to override or supplement global settings (like NEURON_NAME, NEURON_PORT, NEURON_SERVER_URL).

---

9. Usage Overview

Starting a Neuron

• To run a specific neuron (for example, UserNeuron), navigate to the project root and run:

  python -m neurons.user_neuron.neuron_app

• The neuron will load its configuration, start the Flask app on the configured port, register its routes (under /user), and serve the Swagger UI at /docs.

Using the CLI

• The CLI provides commands to create the database, manage tables, and start/kill neuron processes. For example:

  python cli.py create_db
  python cli.py list_tables
  python cli.py start_neuron user_neuron
  python cli.py kill_neuron user_neuron
  python cli.py active_neurons

API Endpoints

• GET /user/: Returns a welcome message.
• POST /user/: Creates a new user.
• GET /user/: Retrieves a user by ID.
• PUT /user/: Updates a user.
• DELETE /user/: Deletes a user.
• GET /user/all: Retrieves all users.
• Swagger UI is available at /docs.

---

10. Design Principles

• Separation of Concerns:
  Core functionality (e.g., database connection, authentication, error handling) is in the core/ folder, while neurons implement domain-specific logic.

• DRY:
  Common code (such as connection logic, error handling, and business logic) is centralized in one place, making the codebase easier to maintain.

• Modularity:
  Each neuron is self-contained and can be deployed, started, or stopped independently.

• Scalability:
  The framework supports adding more neurons with minimal changes and provides CLI commands for process management.

• Extensibility:
  New features (like additional endpoints or messaging systems) can be integrated with minimal code duplication.

---

11. Conclusion

This documentation outlines the complete structure, components, and usage of the Neuron Framework. It explains the design rationale and details each module’s responsibilities. By following this guide, developers (or AI assistants) can quickly understand the framework’s psychology and details, and efficiently work with or extend the project.

neuron-framework