DocEX Processor Plugin & Marketplace System: Design Proposa

[tommyGPT2S]

DocEX Processor Plugin & Marketplace System: Design Proposal

Plugin System Overview
Goals
Enable users to develop, install, and manage custom processors outside the main DocEX repo.
Allow “official” and “community” processors to coexist.

Provide a discovery and sharing mechanism (the “marketplace”).

Processor Plugin Architecture
A. Processor Discovery
Built-in Processors: Found in docex/processors/
User-installed Plugins: Discovered in one or more of:

~/.docflow/plugins/ (local user dir)

Virtualenv or site-packages (via entry points)

Custom path set via DOEX_PLUGIN_PATH env var or config

ProcessorFactory is updated to scan all these locations at startup.

B. Plugin Package Structure
A processor plugin can be:

Single-file Python script (for simple use cases)

Installable Python package (recommended for sharing)

Example directory:

perl
Copy
Edit
~/.docflow/plugins/
my_csv_validator/
processor.py
requirements.txt
README.md
(optional) tests/
Or as a pip-installable package:

nginx
Copy
Edit
pip install docex-processor-image2text
3. Processor Metadata Specification
Each processor must include a metadata block (YAML or Python dict):

python
Copy
Edit
PROCESSOR_METADATA = {
"name": "image2text",
"version": "1.0.1",
"description": "Extracts text from images via OCR",
"author": "Jane Doe",
"license": "MIT",
"docex_version": ">=0.4.0",
"tags": ["ocr", "image", "community"],
"config_schema": { ... } # Optional JSON schema
}
ProcessorFactory will read/import this metadata for registration, validation, and marketplace listing.

CLI Extensions for Processor Management
bash
Copy
Edit
Discover processors (built-in, installed, community)
docex processor list --all

Install from community registry (by name)
docex processor install image2text

Install from a git repo or local path
docex processor install https://github.com/janedoe/docex-image2text
docex processor install ./my_plugins/image2text

Validate a processor for compliance
docex processor validate image2text

Remove (uninstall) a plugin
docex processor remove image2text

Share/upload to marketplace (PR or API)
docex processor share image2text
5. Community Registry/Marketplace
A. Marketplace Backend
Start Simple: Use a curated community_registry.json file in the DocEX repo or a central GitHub repo (e.g., docex-plugins/registry.json).

Contents: Each entry describes a processor:

json
Copy
Edit
[
{
"name": "image2text",
"repo": "https://github.com/janedoe/docex-image2text",
"description": "Extracts text from images",
"version": "1.0.1",
"author": "Jane Doe",
"license": "MIT",
"tags": ["ocr", "image"]
},
...
]
CLI command: docex processor marketplace list fetches and displays available processors.

B. Publishing
Users submit a processor for listing by:

PR to registry repo (add entry to community_registry.json)

Or, use docex processor share to submit directly via API (future)

Moderators/maintainers review, test, and approve PRs.

Plugin Security & Trust
Plugins are disabled by default unless explicitly installed by user.
docex processor list --trusted/--all shows status.

CLI warns if processor is not from a trusted/verified source.

(Optional) Add signature/hashing/author verification for plugins in future.

Developer Experience
A. Cookiecutter Template
Provide cookiecutter-docex-processor for easy new processor scaffolding (with metadata, test, and README stubs).
docex processor init my_new_processor could bootstrap the template.

B. Example Docs
Add a "Create Your Own Processor" tutorial.

Showcase popular community processors.

C. Local Development
Processors in ~/.docflow/plugins are auto-reloaded at startup for easy development/test.

Sample Workflow
User runs:
bash
Copy
Edit
docex processor init my_csv_validator

Edits code and metadata
docex processor validate my_csv_validator
docex processor install ./my_csv_validator
docex processor test my_csv_validator # Runs sample/test suite
docex processor share my_csv_validator # Opens PR or uploads to registry
Others can run:

bash
Copy
Edit
docex processor marketplace list
docex processor install my_csv_validator
9. ProcessorFactory Implementation Pseudocode
python
Copy
Edit
class ProcessorFactory:
@staticmethod
def discover_processors():

1. Discover built-in processors

2. Scan ~/.docflow/plugins/

3. Scan installed Python packages via entry_points (e.g. group="docex.processors")

4. Optional: Scan DOEX_PLUGIN_PATH

@staticmethod
def load_processor(name):
# Import or load processor class, validate PROCESSOR_METADATA
# Return ready-to-use processor class"

[tommyGPT2S]

this is motivated by @garyhe231