Skip to content

DataShifter v0.1.0 — Pluggable Adapters, Marketplace & Workflow DAG Orchestration

Latest
Latest

Choose a tag to compare

View all tags
@dharmarajrdr dharmarajrdr released this 18 Oct 12:14
· 31 commits to main since this release
v0.1.0
7e97668
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

DataShifter v0.1.0

We are excited to announce DataShifter v0.1.0, featuring our pluggable adapter architecture, marketplace ecosystem, visual workflow DAG builder, live CDC replication, and modern UI design system.

🚀 Highlights & Features

1. Pluggable Adapter Ecosystem & Marketplace

  • Adapter SDK & Isolation: Extensible DataReader and DataWriter interfaces with isolated AdapterClassLoader dynamic JAR loading, checksum validation, and security constraints.
  • Rich Adapter Catalog:
    • Relational Databases: Oracle Reader & Writer, PostgreSQL Reader & Writer.
    • Change Data Capture (CDC): Streaming real-time readers for PostgreSQL WAL and Oracle GoldenGate.
    • Destinations & Sinks: File System (JSON, CSV, Parquet) and Apache Kafka event writers.
  • Marketplace Hub:
    • Browse, inspect schema configurations, and install/uninstall adapters directly from the UI.
    • Reference count locking to safeguard active pipelines against adapter uninstallation.
    • Artifact download support for adapter JAR binaries.
    • Beetel AI integration for intelligent adapter discovery and diagnostic remediation.

2. Workflow Orchestration & Live Monitoring

  • Visual DAG Canvas:
    • Interactive nodes and dependency wiring with real-time topology cycle detection.
    • Side-panel inspectors for node execution parameters and edge rules.
  • Workflow & Pipeline Dashboards:
    • Dual view modes (Grid and Table/List views) with pipeline chain sequence tracking.
    • Contextual Validate All / Start Workflow controls and granular node-level mapping validation diagnostics.
  • Live Pipeline Monitor:
    • Real-time CDC metrics, throughput tracking, lag monitors, and diagnostic error log inspection.
  • Visual Column Mapping:
    • Drag-and-drop mapping canvas with transform functions, type coercion, and execution simulators.

3. Enterprise Administration & UI System

  • Tailwind Theme B Design System:
    • Consistent layout shell with responsive collapsible navigation sidebar.
    • Persistent sidebar collapse state (is_collapse) stored across browser sessions.
    • Accessible modal dialogs with sticky headers/footers.
  • Access Control & Notification Hub:
    • Granular RBAC with collapsible permission group selectors in role editors.
    • Centralized notification preferences, event channels, and audit logs.

4. Engine & Database Stability

  • Oracle Connection Pooling:
    • Implemented shared bounded Hikari connection pools and adapter instance caching in the execution engine, eliminating process exhaustion (ORA-00020, ORA-12516, ORA-17800).
Assets 2
Loading