diff --git a/README.md b/README.md index 7c654e8..7247836 100644 --- a/README.md +++ b/README.md @@ -1,146 +1,41 @@ # Hatch -Hatch is the official package manager for the Hatch! ecosystem. It provides powerful tools for managing MCP server packages, environments, and interacting with the Hatch registry. Its primary objective is to service **[Hatchling](https://github.com/CrackingShells/Hatchling)** (interactive CLI-based chat application integrating local Large Language Models with MCP for tool calling), but can of course be included in other projects. +![Hatch Logo](./docs/resources/images/Logo/hatch_wide_dark_bg_transparent.png) -## Project Update Summary +Hatch is the package manager for the Hatch! ecosystem. The documentation in `docs/index.md` is the canonical, up-to-date entry point for users and contributors — this README is a short pointer to those resources. -**May 27, 2025**: Version 0.2.1 release of the Hatch package manager ecosystem! 🎉 +## Quick links -- **Integration with Hatchling** for MCP servers package management while chatting with LLMs -- **Environment isolation system** for managing different sets of MCP server packages -- **Registry integration** for discovering and installing packages -- **Package template generation** for creating new Hatch MCP server packages +The major documentation entry points are: -## Features +- Documentation (canonical): `docs/index.md` +- Getting started (users): `docs/articles/users/GettingStarted.md` +- CLI reference: `docs/articles/users/CLIReference.md` +- Developer docs and architecture: `docs/articles/devs/index.md` -- **Environment Management**: Create isolated environments for different sets of MCP servers -- **Package Installation**: Install packages from the registry or local directories -- **Dependency Resolution**: Automatically resolve and manage package dependencies -- **Template Generation**: Create new Hatch MCP server package templates with a single command -- **Package Validation**: Ensure packages conform to the Hatch schema standards +But, really, just look at the site: -## Installation +## Quick start + +### Install from source ```bash -# From source git clone https://github.com/CrackingShells/Hatch.git cd Hatch pip install -e . ``` -## Usage - -Hatch provides both a command-line interface and a Python API for integration into other tools like Hatchling. - -### Command Line Interface - -The table below summarizes the available commands with their arguments: - -| Command | Description | Arguments | Example | -|---------|-------------|----------|---------| -| `create` | Create a new package template | `name` - Package name
`--dir, -d` - Target directory
`--description, -D` - Package description | `hatch create my-package --description "My awesome MCP server package"` | -| `validate` | Validate a package against schema | `package_dir` - Path to package directory | `hatch validate ./my-package` | -| `env list` | List all available environments | None | `hatch env list` | -| `env create` | Create a new environment | `name` - Environment name
`--description, -D` - Environment description | `hatch env create my-env --description "Environment for biology tools"` | -| `env use` | Set the current active environment | `name` - Environment name | `hatch env use my-env` | -| `env remove` | Remove an environment | `name` - Environment name | `hatch env remove my-env` | -| `env current` | Show the current environment | None | `hatch env current` | -| `package add` | Add a package to an environment | `package_path_or_name` - Path or name of package
`--env, -e` - Environment name
`--version, -v` - Package version | `hatch package add ./my-package --env my-env`
`hatch package add awesome-package --env my-env` | -| `package list` | List packages in an environment | `--env, -e` - Environment name | `hatch package list --env my-env` | -| `package remove` | Remove a package from an environment | `package_name` - Name of package to remove
`--env, -e` - Environment name | `hatch package remove awesome-package --env my-env` | - -### Python API - -```python -from hatch import HatchEnvironmentManager, create_package_template - -# Create a new package template -create_package_template("my-package", target_dir="./packages", description="My awesome package") - -# Manage environments -env_manager = HatchEnvironmentManager() -env_manager.create_environment("my-env", description="My testing environment") -env_manager.add_package("my-package", env_name="my-env") -packages = env_manager.list_packages("my-env") -``` - -## Creating Packages - -Hatch makes it easy to create new MCP server packages: +### Create a package template ```bash hatch create my-package --description "My MCP server package" ``` -This creates a template with the following structure: - -``` -my-package/ -├── __init__.py -├── server.py -├── hatch_metadata.json -└── README.md -``` - -Edit the `server.py` file to define your MCP tools: - -```python -import logging -from hatchling import HatchMCP - -# Initialize MCP server with metadata -hatch_mcp = HatchMCP("my-package", - origin_citation="Your Name, 'Original Software', Year", - mcp_citation="Your Name, 'MCP Implementation', Year") - -@hatch_mcp.tool() -def my_tool(param1: str, param2: int) -> str: - """Description of what your tool does. - - Args: - param1 (str): First parameter description. - param2 (int): Second parameter description. - - Returns: - str: Description of the return value. - """ - hatch_mcp.logger.info(f"Tool called with {param1} and {param2}") - return f"Processed {param1} with value {param2}" - -if __name__ == "__main__": - hatch_mcp.run() -``` - -## Dependencies - -Hatch depends on the following Python packages: - -- jsonschema 4.0.0 or higher -- requests 2.25.0 or higher -- packaging 20.0 or higher -- [Hatch-Validator](https://github.com/CrackingShells/Hatch-Validator) - -## Development - -### Project Structure - -- `hatch/`: Core package source code - - `cli_hatch.py`: CLI implementation - - `environment_manager.py`: Environment management functionality - - `package_loader.py`: Package loading and installation - - `registry_retriever.py`: Registry interaction - - `registry_explorer.py`: Package search and discovery - - `template_generator.py`: Package template generation - -## Related Repositories - -Hatch is part of the larger Hatch! ecosystem which includes: +## Contributing and help -- **[Hatchling](https://github.com/CrackingShells/Hatchling)**: Interactive CLI-based chat application integrating local Large Language Models with MCP for tool calling -- **[Hatch-Schemas](https://github.com/CrackingShells/Hatch-Schemas)**: JSON schemas for package metadata and validation -- **[Hatch-Validator](https://github.com/CrackingShells/Hatch-Validator)**: Validation tools for Hatch packages -- **[Hatch-Registry](https://github.com/CrackingShells/Hatch-Registry)**: Package registry for Hatch packages +- Read developer onboarding and contribution guides in `docs/articles/devs/` +- Report issues or feature requests on the GitHub repository: ## License -This project is licensed under the [GNU Affero General Public License v3](./LICENSE) +This project is licensed under the GNU Affero General Public License v3 — see `LICENSE` for details. diff --git a/VERSION b/VERSION index b300caa..8be17a5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.4.4 \ No newline at end of file +0.5.1.dev2 \ No newline at end of file diff --git a/VERSION.meta b/VERSION.meta index 4967036..4395796 100644 --- a/VERSION.meta +++ b/VERSION.meta @@ -3,8 +3,8 @@ # The companion VERSION file contains the simple format for setuptools MAJOR=0 -MINOR=4 -PATCH=4 -DEV_NUMBER= +MINOR=5 +PATCH=1 +DEV_NUMBER=2 BUILD_NUMBER= -BRANCH=main +BRANCH=dev diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100644 index 0000000..2d0fcb2 --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,2 @@ +theme: minima +repository: CrackingShells/Hatch \ No newline at end of file diff --git a/docs/articles/appendices/LimitsAndKnownIssues.md b/docs/articles/appendices/LimitsAndKnownIssues.md new file mode 100644 index 0000000..516ab05 --- /dev/null +++ b/docs/articles/appendices/LimitsAndKnownIssues.md @@ -0,0 +1,298 @@ +# Limits and Known Issues + +This appendix documents current limitations and known issues in Hatch v0.4.2, organized by impact severity and architectural domain. + +## Critical Limitations (High Impact) + +### Non-Interactive Environment Handling + +**Issue**: The dependency installation orchestrator can block indefinitely in non-TTY environments. + +**Code Location**: `hatch/installers/dependency_installation_orchestrator.py:501` (`_request_user_consent`) + +**Symptoms**: + +- Hangs in CI/CD pipelines when TTY is unavailable +- Docker container execution may hang indefinitely +- Programmatic integration requires foreknowledge of `--auto-approve` parameter + +**Workaround**: Use `--auto-approve` flag for automated scenarios + +```bash +hatch package add my-package --auto-approve +``` + +**Root Cause**: Blocking `input()` call without TTY detection or environment variable fallback mechanisms. + +### System Package Version Constraint Simplification + +**Issue**: Complex version constraints for system packages are reduced to "install latest" with only warning messages. + +**Code Location**: `hatch/installers/system_installer.py:332-365` (`_build_apt_command`) + +**Symptoms**: + +- Version constraint `>=1.2.0` becomes "install latest" +- No validation that installed version satisfies original constraint +- Silent constraint violations in production environments + +**Workaround**: Use exact version constraints (`==1.2.0`) for critical system dependencies + +**Root Cause**: Simplified apt command building that only handles exact version matching. + +### Concurrent Access Race Conditions + +**Issue**: Plain file I/O operations without atomic writes or file locking can lead to corrupted state. + +**Code Locations**: + +- `hatch/environment_manager.py:85-90` +- `hatch/package_loader.py:80-85` + +**Symptoms**: + +- Corrupted `environments.json` when multiple Hatch instances run +- Package cache corruption during concurrent downloads +- Lost environment configuration in multi-user scenarios + +**Workaround**: Avoid running multiple Hatch operations simultaneously + +**Root Cause**: Non-atomic file operations for critical state files. + +## Significant Limitations (Medium Impact) + +### Registry Fetch Fragility + +**Issue**: Registry fetching uses date-based URL construction with limited fallback robustness. + +**Code Location**: `hatch/registry_retriever.py:45-65` + +**Symptoms**: + +- Package discovery breaks when registry publishing is delayed +- Poor error messages during network connectivity issues +- Development workflow disruption during registry maintenance + +**Workaround**: Use local packages (`hatch package add ./local-package`) when registry is unavailable + +**Root Cause**: Registry URL construction assumes daily publishing schedule without robust fallback strategies. + +### Package Integrity Verification Gap + +**Issue**: Downloaded packages are not cryptographically verified for integrity. + +**Code Location**: `hatch/package_loader.py:75-125` (`download_package`) + +**Symptoms**: + +- No detection of package tampering in hostile networks +- Corrupted downloads may be interpreted as valid packages +- No audit trail for package provenance + +**Workaround**: Manually verify package sources and use trusted networks + +**Root Cause**: Missing checksum validation and signature verification during package download. + +### Cross-Platform Python Environment Detection + +**Issue**: Hard-coded path assumptions limit Python environment detection across different platforms and installations. + +**Code Location**: `hatch/python_environment_manager.py:85-120` (`_detect_conda_mamba`) + +**Symptoms**: + +- Inconsistent behavior across different conda installations +- Silent feature degradation when Python environments unavailable +- User confusion about Python integration capabilities + +**Workaround**: Ensure conda/mamba are in system PATH or use explicit paths + +**Root Cause**: Platform-specific path assumptions and limited environment variable checking. + +### Error Recovery and Rollback Gaps + +**Issue**: Limited transactional semantics during multi-dependency installation. + +**Code Location**: `hatch/installers/dependency_installation_orchestrator.py:550-580` (`_execute_install_plan`) + +**Symptoms**: + +- Environments left in inconsistent states after failed installs +- Manual cleanup required for partial installation failures +- Difficult recovery from complex dependency conflicts + +**Workaround**: Create environment snapshots before major installations; remove and recreate environments if corrupted + +**Root Cause**: Sequential installation without comprehensive rollback mechanisms. + +## Moderate Limitations (Development Impact) + +### Limited Observability and Progress Reporting + +**Issue**: Minimal structured logging and progress feedback during operations. Typically intermediate installation steps rely on the individual installer's (i.e. pip, apt, etc.), but orchestrator lacks end-to-end visibility. + +**Code Locations**: Logging scattered across multiple modules + +**Symptoms**: + +- Difficult debugging of installation failures +- Poor user experience during long-running operations +- Limited integration with monitoring systems + +**Workaround**: Increase logging verbosity and monitor log files + +**Root Cause**: Progress callbacks exist but are sparsely implemented across the codebase. + +### Template Generation Assumptions + +**Issue**: Templates assume specific MCP server structure and dependencies. + +**Code Location**: `hatch/template_generator.py:130-140` + +**Symptoms**: + +- Template lock-in for specific MCP server patterns +- Reduced flexibility for alternative MCP frameworks +- Potential incompatibility with future MCP specifications + +**Workaround**: Manually modify generated templates for custom requirements + +**Root Cause**: Hard-coded assumptions about entry points and MCP wrapper dependencies. + +### Dependency Graph Resolution Edge Cases + +**Issue**: Limited handling of circular dependencies and complex version constraints. + +**Code Location**: `hatch/installers/dependency_installation_orchestrator.py:290-320` + +**Symptoms**: + +- Potential infinite loops during dependency resolution +- Unclear error messages for complex dependency conflicts +- Unexpected behavior with deeply nested dependency trees + +**Workaround**: Simplify dependency structures and avoid circular dependencies + +**Root Cause**: Dependency graph builder lacks edge case handling for complex scenarios. + +## Minor Limitations (Quality of Life) + +### Security Context Management + +**Issue**: System package installation assumes `sudo` availability without proper validation. + +**Code Location**: `hatch/installers/system_installer.py:365-380` + +**Symptoms**: + +- Poor error messages when privilege escalation fails +- No pre-validation of system package manager availability + +**Workaround**: Ensure proper sudo configuration and system package manager access + +### Simulation and Dry-Run Gaps + +**Issue**: Inconsistent simulation mode implementation across installers. + +**Code Locations**: Various installer modules + +**Symptoms**: + +- No unified dry-run capability across all dependency types +- Limited preview capabilities for complex installation plans + +**Workaround**: Test installations in isolated environments first + +### Cache Management Strategy + +**Issue**: Basic TTL-based caching without intelligent invalidation or size limits. + +**Code Locations**: + +- `hatch/package_loader.py:40-50` +- `hatch/registry_retriever.py:35-45` + +**Symptoms**: + +- Fixed 24-hour TTL regardless of registry update frequency +- No automatic cache cleanup for disk space management +- Force refresh only available at operation level + +**Workaround**: Manually clear cache directory when needed: + +```bash +rm -rf ~/.hatch/cache/* +``` + +### External Dependency Coupling + +**Issue**: Validator dependency fetched via git URL with network requirements. + +**Code Location**: `pyproject.toml:24` + +**Details**: `hatch_validator @ git+https://github.com/CrackingShells/Hatch-Validator.git@v0.6.3` + +**Symptoms**: + +- Build-time network access required +- Dependency on repository and tag availability + +**Workaround**: Ensure network access during installation or consider local installation methods + +### Documentation and Schema Evolution + +**Issue**: Limited handling of package schema version transitions. + +**Code Locations**: Template generation and package validation flows + +**Symptoms**: + +- Templates generate current schema version only +- No migration tools for package schema updates +- Version compatibility checking incomplete + +**Workaround**: Manually update package metadata when schema versions change + +## Impact Classification + +| Severity | Automation | Reliability | Development | +|----------|------------|-------------|-------------| +| **Critical** | Non-interactive handling | Concurrent access, System constraints | - | +| **Significant** | Registry fragility, Error recovery | Package integrity, Python detection | - | +| **Moderate** | - | - | Observability, Templates, Dependency resolution | +| **Minor** | Simulation gaps | Security context, Cache strategy | External coupling, Schema evolution | + +## Recommended Mitigation Strategies + +### For Production Use + +1. **Always use `--auto-approve`** for automated deployments +2. **Avoid concurrent operations** until race conditions are resolved +3. **Use exact version constraints** for system packages when possible +4. **Implement external monitoring** for installation operations +5. **Regularly backup environment configurations** + +### For Development + +1. **Test in isolated environments** before production deployment +2. **Monitor cache disk usage** and clean manually when needed +3. **Use local packages** when registry is unreliable +4. **Simplify dependency structures** to avoid resolution edge cases + +### For Cross-Platform Deployment + +1. **Ensure conda/mamba in PATH** on all target systems +2. **Test Python environment detection** on each platform +3. **Validate system package managers** before deployment +4. **Document platform-specific requirements** + +## Future Improvements + +The Hatch team is aware of these limitations and they are prioritized for future releases: + +**Phase 1 (Stability)**: Address concurrent access, non-interactive handling, and error recovery +**Phase 2 (Security)**: Implement package integrity verification and security context validation +**Phase 3 (Robustness)**: Improve cross-platform consistency and system package handling +**Phase 4 (Quality)**: Enhance observability, caching, and template flexibility + +For the most current status of these issues, check the project's issue tracker and release notes. diff --git a/docs/articles/appendices/glossary.md b/docs/articles/appendices/glossary.md new file mode 100644 index 0000000..28c62d5 --- /dev/null +++ b/docs/articles/appendices/glossary.md @@ -0,0 +1,165 @@ +# Glossary + +This glossary defines terms and concepts used throughout the Hatch documentation. + +## A + +**Auto-approve** +: CLI flag (`--auto-approve`) that automatically approves dependency installations without user prompts, useful for automation scenarios. + +## C + +**Cache TTL** +: Time-to-live for cached registry data, configurable via `--cache-ttl` flag. Default is 86400 seconds (24 hours). + +**Conda** +: Python package and environment management system used by Hatch for creating isolated Python environments. + +**Current Environment** +: The active Hatch environment that commands operate on by default. Set using `hatch env use `. + +## D + +**Dependency** +: A package or resource required by another package. Hatch supports multiple dependency types: Hatch packages, Python packages, system packages, and Docker images. + +**Dependency Installation Orchestrator** +: Component in `hatch/installers/dependency_installation_orchestrator.py` that coordinates the installation of multiple dependency types. + +**Docker Dependency** +: A Docker image required by a Hatch package, managed by the Docker installer. + +## E + +**Entry Point** +: The main file that serves as the executable entry point for a Hatch package, typically `hatch_mcp_server_entry.py`. + +**Environment** +: An isolated workspace for managing packages and their dependencies. Hatch environments can optionally include Python environments. + +**Environment Manager** +: Core component in `hatch/environment_manager.py` that handles environment lifecycle and management operations. + +## F + +**FastMCP** +: A Python framework for building Model Context Protocol servers, used as the base for Hatch package MCP server implementations. + +**Force Download** +: CLI flag (`--force-download`) that forces package download even if the package is already cached locally. + +## G + +**Global Options** +: CLI options available for all commands, including `--envs-dir`, `--cache-ttl`, and `--cache-dir`. + +## H + +**Hatch** +: Package manager for Model Context Protocol (MCP) servers that provides environment isolation and dependency management. + +**Hatch Compliant** +: Indicates whether a package follows Hatch packaging standards and metadata requirements. + +**Hatch Dependency** +: Another Hatch package required as a dependency, managed by the Hatch installer. + +**Hatch Environment** +: An isolated environment managed by Hatch for organizing packages and dependencies. + +**Hatch MCP Server Wrapper** +: Integration component (`hatch_mcp_server`) that bridges Hatch packages with MCP server functionality. + +**Hatch Package** +: A package that follows Hatch conventions and includes `hatch_metadata.json` with required metadata fields. + +**Hatchling** +: Related project in the Hatch ecosystem that provides additional functionality and tooling. + +## I + +**Installation Context** +: Object that manages state and context information during package installation processes. + +**Installer Base** +: Abstract base class in `hatch/installers/installer_base.py` that defines the interface for all installer types. + +## M + +**Mamba** +: Fast, drop-in replacement for conda package manager. Hatch prefers mamba when available for better performance. + +**MCP** +: Model Context Protocol - a standard for building AI-powered tools and integrations. + +**MCP Server** +: A server implementation that follows the Model Context Protocol standard, typically providing tools and resources for AI applications. + +**Metadata** +: Package information stored in `hatch_metadata.json` that defines package properties, dependencies, and compatibility requirements. + +## P + +**Package** +: A distributable unit of code that provides MCP server functionality. In Hatch, packages follow specific structure and metadata requirements. + +**Package Loader** +: Component in `hatch/package_loader.py` that loads and validates packages from local directories. + +**Package Schema Version** +: Version of the metadata schema used by a package, currently "1.2.0". + +**Python Dependency** +: A Python package installed via pip, managed by the Python installer. + +**Python Environment** +: A conda/mamba environment containing a specific Python version and packages, optionally created within Hatch environments. + +**Python Environment Manager** +: Component in `hatch/python_environment_manager.py` that manages Python environments within Hatch environments wrapping around conda/mamba in order to associate python environments with Hatch environments. + +## R + +**Refresh Registry** +: CLI flag (`--refresh-registry`) that forces refresh of registry data, bypassing cache. + +**Registry** +: Central repository for discovering and downloading Hatch packages. + +**Registry Explorer** +: Component in `hatch/registry_explorer.py` that provides package discovery and search capabilities. + +**Registry Retriever** +: Component in `hatch/registry_retriever.py` that handles package downloads and caching from the registry. + +## S + +**Schema** +: JSON schema definition that validates package metadata structure. Cracking Shells defines and uses schemas for its package ecosystem. It is hosted on a [dedicated repository](https://github.com/CrackingShells/Hatch-Schemas). + +**Semantic Versioning** +: Version numbering scheme using MAJOR.MINOR.PATCH format (e.g., 1.2.0). + +**System Dependency** +: A system package installed via OS package managers like apt, managed by the system installer. + +## T + +**Template Generator** +: Component in `hatch/template_generator.py` that creates new package templates with standard structure and files. + +**Tool** +: A function provided by an MCP server package, defined in the package metadata and exposed through the MCP interface. + +## V + +**Validation** +: Process of checking package structure and metadata against Hatch requirements and schema definitions. + +**Version Constraint** +: Specification of acceptable version ranges for dependencies using operators like `>=`, `==`, `<=`, `!=`. + +## W + +**Workspace** +: Development environment containing multiple related projects and packages, such as the Hatch multi-project workspace. diff --git a/docs/articles/appendices/index.md b/docs/articles/appendices/index.md new file mode 100644 index 0000000..e31bd35 --- /dev/null +++ b/docs/articles/appendices/index.md @@ -0,0 +1,27 @@ +# Appendices + +This section contains supplementary information that supports the main documentation. + +## Available Appendices + +- [**Glossary**](glossary.md) - Definitions of terms and concepts used throughout Hatch documentation +- [**State and Data Models**](state_and_data_models.md) - Detailed information about data structures and state management +- [**Limits and Known Issues**](LimitsAndKnownIssues.md) - Current limitations and known issues with workarounds + +## Purpose + +The appendices provide: + +- **Foundational knowledge** for beginners new to package management or MCP servers +- **Detailed reference information** that would clutter the main articles +- **Troubleshooting resources** for common problems and their solutions +- **Historical context** and background information for design decisions + +## Using the Appendices + +- **New users** should start with the glossary to understand key terms +- **Developers** can reference state and data models for implementation details +- **Troubleshooters** can check limits and known issues for common problems +- **All users** can benefit from cross-references back to main documentation + +The appendices are designed to be self-contained while providing clear links back to relevant sections in the main documentation. diff --git a/docs/articles/appendices/state_and_data_models.md b/docs/articles/appendices/state_and_data_models.md new file mode 100644 index 0000000..1a708f7 --- /dev/null +++ b/docs/articles/appendices/state_and_data_models.md @@ -0,0 +1,188 @@ +# State and Data Models + +This appendix provides detailed information about data structures and state management in Hatch. + +## Package Metadata Schema + +The complete package metadata schema is defined in `Hatch-Schemas/package/v1.2.0/hatch_pkg_metadata_schema.json`. + +### Required Fields + +```json +{ + "package_schema_version": "1.2.0", + "name": "package_name", + "version": "1.0.0", + "entry_point": "hatch_mcp_server_entry.py", + "description": "Package description", + "tags": ["tag1", "tag2"], + "author": { + "name": "Author Name", + "email": "author@example.com" + }, + "license": { + "name": "MIT", + "uri": "https://opensource.org/licenses/MIT" + } +} +``` + +### Optional Fields + +```json +{ + "contributors": [ + { + "name": "Contributor Name", + "email": "contributor@example.com" + } + ], + "repository": "https://github.com/user/repo", + "documentation": "https://docs.example.com", + "dependencies": { + "hatch": [...], + "python": [...], + "system": [...], + "docker": [...] + }, + "compatibility": { + "hatchling": ">=0.1.0", + "python": ">=3.8" + }, + "tools": [ + { + "name": "tool_name", + "description": "Tool description" + } + ], + "citations": { + "origin": "Origin citation", + "mcp": "MCP citation" + } +} +``` + +### Dependency Structures + +#### Hatch Dependencies + +```json +{ + "hatch": [ + { + "name": "package_name", + "version_constraint": ">=1.0.0" + } + ] +} +``` + +#### Python Dependencies + +```json +{ + "python": [ + { + "name": "requests", + "version_constraint": ">=2.28.0", + "package_manager": "pip" + } + ] +} +``` + +#### System Dependencies + +```json +{ + "system": [ + { + "name": "curl", + "version_constraint": ">=7.0.0", + "package_manager": "apt" + } + ] +} +``` + +#### Docker Dependencies + +```json +{ + "docker": [ + { + "name": "redis", + "version_constraint": ">=7.0.0", + "registry": "dockerhub" + } + ] +} +``` + +## Environment State + +### Environment Metadata + +Hatch stores environments as a mapping from environment name to metadata. The `environments.json` file looks like a dictionary where each key is the environment name and the value is the environment metadata. For example: + +```json +{ + "default": { + "name": "default", + "description": "Default environment", + "created_at": "2025-07-02T08:15:03.168827", + "packages": [], + "python_environment": true, + "python_env": { + "enabled": true, + "conda_env_name": "hatch_default", + "python_executable": "/home//miniforge3/envs/hatch_default/bin/python", + "created_at": "2025-07-07T22:19:33.122291", + "version": "3.13.5", + "requested_version": null, + "manager": "conda" + } + }, + "modeling": { + "name": "modeling", + "description": "An environment for modeling tools", + "created_at": "2025-08-08T16:18:29.711024", + "packages": [ + { + "name": "arithmetic_pkg", + "version": "1.2.0", + "type": "hatch", + "source": "/path/to/local/packages/arithmetic_pkg", + "installed_at": "2025-08-08T16:26:42.150821" + }, + { + "name": "base_pkg_1", + "version": "1.0.3", + "type": "hatch", + "source": "https://example.com/base_pkg_1-v1.0.3.zip", + "installed_at": "2025-08-24T12:57:42.431051" + } + ], + "python_environment": true, + "python_version": "3.12", + "python_env": { + "enabled": true, + "conda_env_name": "hatch_modeling", + "python_executable": "/home//miniforge3/envs/hatch_modeling/bin/python", + "created_at": "2025-08-08T16:18:29.711024", + "version": "3.12.11", + "requested_version": "3.12", + "manager": "conda" + } + } +} +``` + +### Current Environment Tracking + +The current environment is tracked in Hatch's cache directory in a file named `current_env`: `~/.hatch/envs/current_env` +It only contains the name of the current environment. + +```txt +my_env +``` diff --git a/docs/articles/devs/architecture/component_architecture.md b/docs/articles/devs/architecture/component_architecture.md new file mode 100644 index 0000000..8570b01 --- /dev/null +++ b/docs/articles/devs/architecture/component_architecture.md @@ -0,0 +1,291 @@ +# Component Architecture + +This article is about: + +- Detailed breakdown of major system components and their responsibilities +- Component relationships and data flow patterns +- Extension points and configuration options + +## Core Components + +![Component Architecture](../../../resources/images/architecture.svg) + +### Environment Management Components + +#### HatchEnvironmentManager (`hatch/environment_manager.py`) + +**Responsibilities:** + +- Core functionality for managing isolated environments +- Environment lifecycle and state management +- Delegates package installation to the DependencyInstallerOrchestrator +- Manages environment metadata and persistence + +**Key Features:** + +- Environment creation, removal, and listing +- Current environment tracking +- Environment metadata persistence +- Integration with package installation system + +#### PythonEnvironmentManager (`hatch/python_environment_manager.py`) + +**Responsibilities:** + +- Specialized management of Python environments via conda/mamba +- Python version detection and environment creation +- Python executable resolution and shell access +- Integration with system conda/mamba installations + +**Key Features:** + +- Automatic conda/mamba detection +- Python environment creation and management +- Environment diagnostic capabilities +- Shell integration for environment activation + +### Package System Components + +#### PackageLoader (`hatch/package_loader.py`) + +**Responsibilities:** + +- Loads and parses Hatch package metadata from local directories +- Validates package structure and metadata compliance +- Provides package inspection capabilities for CLI operations + +**Key Features:** + +- Package metadata loading and validation +- Package structure verification +- Local package installation support +- Integration with validation system + +#### TemplateGenerator (`hatch/template_generator.py`) + +**Responsibilities:** + +- Generates complete package templates with standard structure +- Creates boilerplate files including entry points and metadata +- Orchestrates template creation process for `hatch create` command + +**Key Features:** + +- Standard package template generation +- Customizable package metadata +- Boilerplate file creation +- Template validation + +### Registry System Components + +#### RegistryRetriever (`hatch/registry_retriever.py`) + +**Responsibilities:** + +- Handles registry interactions and package retrieval +- Implements caching strategies with configurable TTL +- Manages network fallback and error handling + +**Key Features:** + +- Package download and caching +- TTL-based cache management +- Network fallback strategies +- Registry authentication support + +#### RegistryExplorer (`hatch/registry_explorer.py`) + +**Responsibilities:** + +- Provides registry exploration and search capabilities +- Handles registry metadata parsing and package discovery +- Supports different registry sources and formats + +**Key Features:** + +- Package search and discovery +- Registry metadata parsing +- Multiple registry source support +- Package relationship analysis + +### Installation System Components + +#### DependencyInstallerOrchestrator (`hatch/installers/dependency_installation_orchestrator.py`) + +**Responsibilities:** + +- Coordinates multi-type dependency installation +- Manages installation planning and execution +- Provides progress reporting and error aggregation + +**Key Features:** + +- Multi-type dependency coordination +- Installation planning and optimization +- Progress tracking and reporting +- Error handling and recovery + +#### InstallationContext (`hatch/installers/installation_context.py`) + +**Responsibilities:** + +- Manages installation context and state +- Tracks installation progress and results +- Provides installer-specific context data + +**Key Features:** + +- Installation state management +- Progress callback mechanisms +- Context data storage +- Installation result tracking + +#### Installer Framework (`hatch/installers/`) + +**Base Interface:** `installer_base.py` + +- Defines common installer interface and base functionality +- Provides consistent API surface for all installer types +- Handles common error scenarios and logging + +**Specialized Installers:** + +- **PythonInstaller** - Python package installation via pip +- **SystemInstaller** - System package installation via package managers +- **DockerInstaller** - Docker image dependency management +- **HatchInstaller** - Hatch package dependency installation + +## Component Data Flow + +### Environment Creation Flow + +`CLI Command → HatchEnvironmentManager → PythonEnvironmentManager → Environment Metadata` + +1. CLI parses environment creation command and options +2. HatchEnvironmentManager creates environment structure and metadata +3. PythonEnvironmentManager optionally creates Python environment +4. Environment metadata is persisted for future operations + +### Package Installation Flow + +`CLI Command → PackageLoader → DependencyInstallerOrchestrator → Specialized Installers → InstallationContext` + +1. PackageLoader reads and validates package metadata +2. DependencyInstallerOrchestrator analyzes dependencies and creates installation plan +3. Specialized installers handle different dependency types in parallel +4. InstallationContext tracks installation state and progress +5. Package is registered in environment metadata + +### Registry Operations Flow + +`CLI Command → RegistryExplorer → RegistryRetriever → Local Cache → Package Operations` + +1. RegistryExplorer discovers and searches available packages +2. RegistryRetriever handles network operations and caching +3. Local cache is maintained with configurable TTL +4. Fallback strategies handle network issues and offline operation + +## Design Patterns in Components + +### Strategy Pattern Implementation + +- **Installer Framework**: Different installers implement common interface +- **Registry Retrieval**: Multiple strategies for different registry types +- **Python Environment Detection**: Support for multiple environment managers + +### Template Method Pattern Implementation + +- **Installation Orchestration**: Defines workflow while allowing installer-specific implementation +- **Package Template Generation**: Standard structure with customizable content +- **Environment Management**: Common lifecycle with specialized implementations + +### Factory Pattern Implementation + +- **Template Generator**: Creates packages with consistent structure +- **Installation Context**: Creates appropriate installer instances +- **Environment Creation**: Factory methods for different environment types + +## Extension Points + +### Adding New Installer Types + +**Extension Process:** + +1. Implement `InstallerBase` interface in new installer class +2. Register installer with `DependencyInstallerOrchestrator` +3. Add metadata schema support for new dependency type +4. Update documentation and tests + +**Key Interfaces:** + +- `install_dependency()` - Core installation logic +- `can_handle_dependency()` - Dependency type detection +- `get_dependency_type()` - Type identification + +### Registry Integration Extensions + +**Extension Process:** + +1. Extend `RegistryRetriever` for new registry sources +2. Implement appropriate caching strategy +3. Add error handling and fallback mechanisms +4. Update `RegistryExplorer` for new discovery methods + +### Python Environment Support Extensions + +**Extension Process:** + +1. Extend `PythonEnvironmentManager` detection capabilities +2. Add new environment manager implementations +3. Update diagnostic and shell integration capabilities + +## Configuration Points + +### Environment Configuration + +- **Environment Storage Directory** (`--envs-dir`) - Where environments are stored +- **Environment Metadata** - Persistent environment state and package tracking + +### Cache Configuration + +- **Cache Directory** (`--cache-dir`) - Local cache storage location +- **Cache TTL** (`--cache-ttl`) - Time-to-live for cached registry data +- **Cache Strategy** - Configurable caching behavior + +### Registry Configuration + +- **Registry Endpoints** - Primary and fallback registry URLs +- **Authentication** - Registry access credentials and tokens +- **Fallback Behavior** - Offline operation and error handling + +### Python Environment Configuration + +- **Manager Preferences** - Conda vs. mamba preference +- **Python Version Detection** - Automatic vs. explicit version selection +- **Environment Activation** - Shell integration preferences + +## Component Testing Strategy + +### Unit Testing + +- Individual component testing with mocked dependencies +- Interface compliance testing for extensible components +- Error condition and edge case testing + +### Integration Testing + +- Component interaction testing +- End-to-end workflow validation +- External dependency integration testing + +### Simulation Testing + +- Offline operation testing +- Network failure simulation +- Cache behavior validation + +## Related Documentation + +- [System Overview](./system_overview.md) - High-level architecture introduction +- [Implementation Guides](../implementation_guides/) - Technical implementation guidance for specific components +- [Development Processes](../development_processes/) - Development workflow and testing standards diff --git a/docs/articles/devs/architecture/index.md b/docs/articles/devs/architecture/index.md new file mode 100644 index 0000000..b1a9021 --- /dev/null +++ b/docs/articles/devs/architecture/index.md @@ -0,0 +1,47 @@ +# Architecture Documentation + +This section provides high-level architectural understanding of the Hatch package manager system. + +## Overview + +Hatch is a sophisticated package management system designed for the CrackingShells ecosystem, featuring modular architecture with clear separation of concerns across multiple layers. + +## Available Documentation + +### System Architecture + +- **[System Overview](./system_overview.md)** - High-level introduction to Hatch's architecture and core concepts +- **[Component Architecture](./component_architecture.md)** - Detailed breakdown of major system components and their relationships + +### Design Patterns + +Design patterns are covered within the main architecture documents: + +- **Installer Framework** - Strategy pattern implementation covered in [Component Architecture](./component_architecture.md#installation-system-components) +- **Environment Management** - Environment patterns covered in [Component Architecture](./component_architecture.md#environment-management-components) + +## Architecture Diagram + +The complete system architecture is documented in the [Architecture Diagram](../../resources/diagrams/architecture.puml), which provides a visual overview of all components and their relationships. + +## Key Architectural Principles + +1. **Modular Design** - Clear separation of concerns across components +2. **Extensibility** - Plugin-based architecture for installers and extensions +3. **Environment Isolation** - Robust environment management with metadata persistence +4. **Caching Strategy** - TTL-based caching for performance optimization + +## For New Developers + +If you're new to the Hatch codebase: + +1. Start with [System Overview](./system_overview.md) to understand the big picture +2. Review the [Architecture Diagram](../../resources/diagrams/architecture.puml) for visual context +3. Explore [Component Architecture](./component_architecture.md) for detailed component understanding +4. Check [Implementation Guides](../implementation_guides/) when ready to work on specific features + +## Related Documentation + +- [Implementation Guides](../implementation_guides/) - Technical how-to guides for specific components +- [Development Processes](../development_processes/) - Development workflow and standards +- [Contribution Guidelines](../contribution_guides/) - How to contribute to the project diff --git a/docs/articles/devs/architecture/system_overview.md b/docs/articles/devs/architecture/system_overview.md new file mode 100644 index 0000000..1d37ea0 --- /dev/null +++ b/docs/articles/devs/architecture/system_overview.md @@ -0,0 +1,151 @@ +# System Overview + +This article is about: + +- High-level introduction to Hatch's architecture and core concepts +- System components and their relationships +- Key architectural patterns and design decisions + +## Introduction + +Hatch is a sophisticated package management system designed for the CrackingShells ecosystem, featuring modular architecture with clear separation of concerns. It serves as the foundation for managing MCP server packages, environments, and registry interactions. + +## Components + +![Component Architecture](../../../resources/images/architecture.svg) + +(This image is big, don't hesitate to open it in a new tab) + +## Core System Layers + +### CLI Layer + +The command-line interface provides the primary user interaction point: + +- **`hatch/cli_hatch.py`** - Command-line interface with argument parsing and validation +- Delegates operations to appropriate management components +- Provides consistent user experience across all operations + +### Environment Management Layer + +Manages isolated environments for package installation: + +- **Environment Manager** - Core environment lifecycle and state management +- **Python Environment Manager** - Specialized Python environment integration via conda/mamba +- **Environment Isolation** - Ensures packages don't interfere between environments + +### Package System Layer + +Handles package operations and metadata: + +- **Package Loader** - Local package inspection and metadata validation +- **Template Generator** - Package template creation with boilerplate generation +- **Package Validation** - Ensures packages conform to schema standards + +### Registry System Layer + +Manages remote package discovery and retrieval: + +- **Registry Retriever** - Package downloads with caching and fallback handling +- **Registry Explorer** - Package discovery and search capabilities +- **Caching Strategy** - TTL-based caching for performance optimization + +### Installation System Layer + +Coordinates multi-type dependency installation: + +- **Dependency Orchestrator** - Coordinates installation across multiple dependency types +- **Installation Context** - Manages installation state and progress tracking +- **Installer Framework** - Extensible installer system using strategy pattern + +## System Architecture + +## Key Architectural Patterns + +### Strategy Pattern + +Used extensively in the installer framework: + +- Abstract installer interface defines common operations +- Concrete installers implement specific dependency types +- Registry-based installer discovery and instantiation + +### Template Method Pattern + +Applied in installation workflows: + +- Orchestrator defines installation workflow steps +- Individual installers implement specific installation logic +- Centralized consent management and progress reporting + +### Cache Management Pattern + +Implemented throughout the system: + +- TTL-based caching for registry data and packages +- Configurable cache directories and expiration policies +- Fallback mechanisms for offline operation + +## Component Relationships + +### Environment to Package Flow + +1. User creates or selects environment +2. Environment manager initializes environment metadata +3. Package operations are scoped to the active environment +4. Installation system coordinates dependency resolution + +### Package Installation Flow + +1. Package metadata is loaded and validated +2. Dependencies are analyzed and resolved +3. Installation orchestrator coordinates multi-type installation +4. Progress is tracked and reported to user + +### Registry Integration Flow + +1. Registry explorer discovers available packages +2. Registry retriever downloads and caches packages +3. Package loader validates downloaded packages +4. Installation system processes package dependencies + +## Design Principles + +### Modularity + +- Clear separation of concerns across components +- Well-defined interfaces between system layers +- Minimal coupling between unrelated components + +### Extensibility + +- Plugin-based architecture for installers +- Registry pattern for component discovery +- Abstract interfaces for easy extension + +### Reliability + +- Comprehensive error handling and recovery +- Robust caching with fallback mechanisms +- Environment isolation prevents interference + +### Performance + +- Efficient caching strategies reduce network calls +- Parallel installation where possible +- Optimized dependency resolution algorithms + +## For New Developers + +Understanding this system overview provides the foundation for working with Hatch: + +1. **Start Here** - This overview gives you the big picture +2. **Dive Deeper** - See [Component Architecture](./component_architecture.md) for detailed component information +3. **Get Hands-On** - Follow [Developer Onboarding](../development_processes/developer_onboarding.md) for practical next steps +4. **Implementation** - Check [Implementation Guides](../implementation_guides/) when ready to implement features + +## Related Documentation + +- [Component Architecture](./component_architecture.md) - Detailed component breakdown +- [Implementation Guides](../implementation_guides/) - Technical implementation guidance +- [Development Processes](../development_processes/) - Development workflow and standards diff --git a/docs/articles/devs/contribution_guides/how_to_contribute.md b/docs/articles/devs/contribution_guides/how_to_contribute.md new file mode 100644 index 0000000..0f92de5 --- /dev/null +++ b/docs/articles/devs/contribution_guides/how_to_contribute.md @@ -0,0 +1,229 @@ +# How to Contribute + +This article is about: + +- General contribution workflow and process for the Hatch project +- Branch naming conventions and submission requirements +- Community standards and expectations for contributors + +## Overview + +We welcome contributions to the Hatch project! This guide outlines the process for contributing code, documentation, and other improvements to help ensure smooth collaboration and high-quality contributions. + +## Before You Start + +### Prerequisites + +1. **Understand the System** - Review [Architecture Documentation](../architecture/) to understand Hatch's design +2. **Set Up Development Environment** - Follow [Development Environment Setup](../development_processes/development_environment_setup.md) +3. **Review Standards** - Familiarize yourself with [Testing Requirements](./testing_and_ci.md) and [Release Policies](./release_and_dependency_policy.md) + +### Planning Your Contribution + +- **Check Existing Issues** - Search [GitHub Issues](https://github.com/CrackingShells/Hatch/issues) for related work +- **Discuss Major Changes** - Open an issue to discuss significant changes before implementing +- **Review Implementation Guides** - Check [Implementation Guides](../implementation_guides/) for technical guidance + +## Contribution Workflow + +### 1. Fork and Clone + +```bash +# Fork the repository on GitHub +git fork https://github.com/CrackingShells/Hatch.git + +# Clone your fork locally +git clone https://github.com/YOUR_USERNAME/Hatch.git +cd Hatch + +# Add upstream remote +git remote add upstream https://github.com/CrackingShells/Hatch.git +``` + +### 2. Create Feature Branch + +Use descriptive branch names with appropriate prefixes: + +```bash +# Feature additions +git checkout -b feat/add-new-installer-type + +# Bug fixes +git checkout -b fix/environment-creation-error + +# Documentation updates +git checkout -b docs/update-architecture-guide +``` + +**Branch Naming Conventions:** + +- `feat/` - New features or enhancements +- `fix/` - Bug fixes +- `docs/` - Documentation changes +- `cicd/` - CI/CD pipeline changes + +### 3. Implement Your Changes + +#### Code Changes + +- Follow the organization's coding standards +- Write clear, self-documenting code with appropriate comments +- Include docstrings following the organization's docstring standards +- Implement comprehensive error handling and logging + +#### Testing Requirements + +- Write tests for all new functionality +- Follow the three-tier testing approach: Development, Regression, Feature +- Ensure tests pass locally before submitting +- Maintain or improve test coverage + +#### Documentation Updates + +- Update relevant documentation for new features +- Follow the organization's documentation guidelines +- Reference API docstrings rather than duplicating implementation details +- Maintain clear cross-references between related topics + +### 4. Commit Your Changes + +Write clear, descriptive commit messages: + +```bash +# Good commit messages +git commit -m "[Update] Add support for custom installer types" +git commit -m "[Fix] Resolve environment creation race condition" +git commit -m "[Docs - Minor] Typos in installation orchestration guide" + +# Include more detail in commit body for complex changes +git commit -m "[Feat] Implement parallel dependency installation + +- Add ThreadPoolExecutor for concurrent installations +- Implement dependency grouping for parallelization +- Add timeout handling for long-running installations +- Update tests to cover parallel execution scenarios" +``` + +### 5. Keep Your Branch Updated + +```bash +# Fetch latest changes from upstream +git fetch upstream + +# Rebase your branch on latest main +git rebase upstream/main + +# Resolve any conflicts and continue +git rebase --continue +``` + +### 6. Submit Pull Request + +#### Pull Request Guidelines + +- **Clear Title** - Summarize the change in the title +- **Detailed Description** - Explain what changes were made and why +- **Link Related Issues** - Reference any related GitHub issues +- **Testing Information** - Describe how the changes were tested +- **Breaking Changes** - Clearly document any breaking changes + +#### Pull Request Template + +```markdown +## Description +Brief description of the changes made. + +## Type of Change +- [ ] Bug fix (non-breaking change which fixes an issue) +- [ ] New feature (non-breaking change which adds functionality) +- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) +- [ ] Documentation update + +## Testing +- [ ] Tests pass locally +- [ ] New tests added for new functionality +- [ ] Manual testing performed + +## Related Issues +Fixes #(issue number) + +## Additional Notes +Any additional information or context about the changes. +``` + +## Code Review Process + +### What to Expect + +- **Initial Review** - Maintainers will review your pull request within a few days +- **Feedback** - You may receive requests for changes or improvements +- **Iteration** - Work with reviewers to address feedback and refine your contribution +- **Approval** - Once approved, your changes will be merged + +### Responding to Feedback + +- **Be Responsive** - Address feedback promptly and professionally +- **Ask Questions** - If feedback is unclear, ask for clarification +- **Make Requested Changes** - Implement suggested improvements +- **Update Tests** - Ensure tests still pass after making changes + +## Community Standards + +### Communication + +- **Be Respectful** - Treat all community members with respect and professionalism +- **Be Constructive** - Provide helpful, actionable feedback +- **Be Patient** - Understand that reviews take time and maintainers are volunteers + +### Quality Standards + +- **Follow Conventions** - Adhere to established coding and documentation standards +- **Test Thoroughly** - Ensure your changes work correctly and don't break existing functionality +- **Document Changes** - Provide clear documentation for new features and changes + +### Contribution Types + +#### Code Contributions + +- New features and enhancements +- Bug fixes and improvements +- Performance optimizations +- Refactoring and code cleanup + +#### Documentation Contributions + +- API documentation improvements +- Tutorial and guide updates +- Example code and usage patterns +- Translation and localization + +#### Testing Contributions + +- New test cases and scenarios +- Test infrastructure improvements +- Performance and load testing +- Integration test enhancements + +## Getting Help + +### Resources + +- **[Architecture Documentation](../architecture/)** - Understanding the system design +- **[Implementation Guides](../implementation_guides/)** - Technical implementation guidance +- **[Development Processes](../development_processes/)** - Development workflow and standards + +### Support Channels + +- **GitHub Issues** - For bug reports and feature requests +- **GitHub Discussions** - For questions and general discussion +- **Pull Request Comments** - For specific feedback on contributions + +## Recognition + +Contributors who make significant contributions to the Hatch project will be recognized in: + +- Project documentation and release notes +- Contributor acknowledgments +- Community highlights and announcements + +Thank you for contributing to the Hatch project! Your contributions help make package management better for the entire CrackingShells ecosystem. diff --git a/docs/articles/devs/contribution_guides/index.md b/docs/articles/devs/contribution_guides/index.md new file mode 100644 index 0000000..9da70d6 --- /dev/null +++ b/docs/articles/devs/contribution_guides/index.md @@ -0,0 +1,42 @@ +# Contribution Guidelines + +This section provides process-focused guidance for contributing to the Hatch project. + +## Overview + +These guidelines focus on the contribution process, community standards, and project policies rather than technical implementation details. For technical how-to guides, see [Implementation Guides](../implementation_guides/). + +## Available Guidelines + +### Core Contribution Process + +- **[How to Contribute](./how_to_contribute.md)** - General contribution workflow, branch naming, and submission process +- **[Release and Dependency Policy](./release_and_dependency_policy.md)** - Release management policies and dependency guidelines + +## Before Contributing + +1. **Review the Architecture** - Understand the system by reading [Architecture Documentation](../architecture/) +2. **Set Up Development Environment** - Follow [Development Environment Setup](../development_processes/development_environment_setup.md) +3. **Understand Testing Requirements** - Review [Testing Standards](../development_processes/testing_standards.md) +4. **Check Implementation Guides** - For technical guidance, see [Implementation Guides](../implementation_guides/) + +## Contribution Workflow + +1. **Plan Your Contribution** - Discuss significant changes in issues before implementing +2. **Follow Branch Naming** - Use proper branch prefixes (`feat/`, `fix/`, `docs/`, etc.) +3. **Implement Following Standards** - Adhere to code quality and testing requirements +4. **Submit for Review** - Create pull requests following the contribution guidelines +5. **Address Feedback** - Work with maintainers to refine your contribution + +## Community Standards + +- Be respectful and constructive in all interactions +- Follow the organization's coding and documentation standards +- Test your changes thoroughly before submission +- Provide clear descriptions of changes and their rationale + +## Related Documentation + +- [Development Processes](../development_processes/) - Development workflow and standards +- [Implementation Guides](../implementation_guides/) - Technical implementation guidance +- [Architecture](../architecture/) - System understanding for contributors diff --git a/docs/articles/devs/contribution_guides/release_policy.md b/docs/articles/devs/contribution_guides/release_policy.md new file mode 100644 index 0000000..2aa0d8f --- /dev/null +++ b/docs/articles/devs/contribution_guides/release_policy.md @@ -0,0 +1,158 @@ +# Release Policy + +This document records the project's release and dependency practices and, importantly, documents the implemented automated versioning flow used by CI and helper scripts. + +This article is about: + +- The repository's automated versioning helpers and where they live (`versioning/`) +- How version information is stored (`VERSION.meta`, `VERSION`) and when to update it +- The GitHub Actions that run the automation and create tags/releases +- Practical contributor guidance for interacting with the automation + +## Overview + +This file documents the real, implemented behavior for release/version automation, tag/release creation, and tag cleanup — not policies that are hypothetical. See the "Automated versioning" section for exact scripts, branch rules, and local commands. + +## Release Management + +### Versioning Strategy + +Hatch follows semantic versioning (SemVer) for public releases. The project additionally uses a structured, automated versioning helper that maintains both a human-readable, componentized file and a setuptools-compatible simple file. Key points: + +- **MAJOR** version: Incompatible API changes +- **MINOR** version: Backwards-compatible functionality additions +- **PATCH** version: Backwards-compatible bug fixes + +Automation rules (implemented in the repository) determine how prerelease/dev/build components are generated based on branch naming and actions (see "Automated versioning" below). + +### Version Files + +Each project maintains version information in two companion files (the repository uses a dual-file system used by the versioning helpers and CI): + +- `VERSION.meta` - Structured, human-readable key=value format that stores component fields (MAJOR, MINOR, PATCH, DEV_NUMBER, BUILD_NUMBER, BRANCH). Used as the canonical source for automated updates and CI. +- `VERSION` - Simple, setuptools-compatible version string derived from `VERSION.meta` (for building and packaging). This file is regenerated from `VERSION.meta` before builds. +- `pyproject.toml` - Package configuration with version specification + +Example from `Hatch/pyproject.toml`: + +```toml +[project] +name = "hatch" +version = "0.4.2" +description = "Package manager for Model Context Protocol servers" +dependencies = [ + "hatch-validator>=0.1.0", + "requests>=2.28.0", +] +``` + +### Release Process + +The release process is mostly automated via repository scripts and GitHub Actions. High-level steps: + +1. Version management and bumping are driven by branch names and CI (see "Automated versioning"). +2. CI runs tests and prepares a build artifact using the version resolved by the automation. +3. If CI succeeds, a job commits updated `VERSION`/`VERSION.meta` and creates a git tag with the resolved version string. +4. Pushed tags trigger the release workflow which creates a GitHub Release (pre-release for dev versions). +5. Optionally, scheduled/manual tag cleanup removes old dev/build tags. + +You generally should not edit `VERSION` or `VERSION.meta` by hand unless you have a specific reason — use the provided helper scripts or let CI manage version updates. + +See "How the automation works" for the exact flow and commands to run locally. + +## Automated versioning (scripts + workflows) + +The repository provides a small set of scripts and GitHub Actions that implement the automated bumping, tagging, and release flow. The important files are: + +- `versioning/version_manager.py` — core helper that reads/writes `VERSION.meta`, computes semantic version strings, and exposes commands: + - `--get` prints the current version string + - `--increment {major,minor,patch,dev,build}` increments a component and updates both files + - `--update-for-branch BRANCH` updates version fields according to the branch name and writes both `VERSION.meta` and `VERSION` + - `--simple` / helpers to write the simple `VERSION` file from the structured meta + +- `versioning/prepare_version.py` — small helper run before build that converts `VERSION.meta` into the simple `VERSION` file for setuptools compatibility (preserves `VERSION.meta`). + +- `versioning/tag_cleanup.py` — CI/manual helper to find and delete old `+build` and `.dev` tags according to configured age thresholds (dry-run mode by default). + +Workflows involved: + +- `.github/workflows/test_build.yml` — callable workflow used to: + - Run tests/builds + - Execute `python versioning/version_manager.py --update-for-branch ` to compute and write the new version (branch is passed from the calling workflow) + - Emit the computed version as a workflow output + - Run `python versioning/prepare_version.py` and build the package + - Upload `VERSION` and `VERSION.meta` as artifacts for downstream jobs + +- `.github/workflows/commit_version_tag.yml` — triggered on pushes to branches like `dev`, `main`, `feat/*`, `fix/*`. It: + - Calls/depends on the `test_build` workflow + - Downloads the `VERSION` files artifact + - Commits any changes to `VERSION`/`VERSION.meta` made by CI + - Creates and pushes a lightweight git tag named after the computed version (for example `v1.2.3` or `v1.2.3.dev4+build5`) + +- `.github/workflows/tag-release.yml` — fires on pushed tags matching the project's tag patterns and: + - Creates a GitHub Release for the tag + - Marks tags containing `.dev` as pre-releases + +- `.github/workflows/tag-cleanup.yml` — manual / (future: scheduled) workflow that runs `versioning/tag_cleanup.py` to remove old dev/build tags. + +Tagging conventions used by the automation: + +- Tags are created from the computed version string returned by `version_manager` and pushed by `commit_version_tag.yml`. +- Examples: `v1.2.3`, `v1.2.3.dev0`, `v1.2.3.dev0+build1`. +- Tags that include `.dev` are treated as pre-releases in the release workflow. + +## Branch-driven bump rules (summary) + +The `version_manager` logic implements these broad rules (read `versioning/version_manager.py` for full details): + +- `main` — clean release: no dev/build metadata; `DEV_NUMBER` and `BUILD_NUMBER` cleared. +- `dev` — prerelease/dev versions (increments dev number). +- `feat/*` (new feature branch) — creates/advances a minor/dev sequence; new feature branches may reset dev/build counters and start from e.g. `0`. +- `fix/*` — patch-level changes; build numbers are used to distinguish iterative work on the same fix branch. +- Other branches — treated as dev/prerelease in most cases. + +The manager writes `VERSION.meta` with component fields and `VERSION` with the setuptools-compatible string (derived from `VERSION.meta`). + +## How to run and test locally + +Quick commands you can run from the repository root (PowerShell examples): + +```powershell +# Print current computed version +python versioning/version_manager.py --get + +# Update version for a given branch (this writes both files) +python versioning/version_manager.py --update-for-branch dev + +# Increment a patch locally (writes both files) +python versioning/version_manager.py --increment patch --branch dev + +# Prepare simple VERSION file for a build (convert from VERSION.meta) +python versioning/prepare_version.py +``` + +Notes: + +- After running local updates, commit the updated `VERSION` and `VERSION.meta` if you intend to push the change. +- Prefer letting CI run `--update-for-branch` and perform the commit/tag steps automatically unless you need to perform an explicit offline bump. + +## Tag cleanup and maintenance + +Old `+build` and `.dev` tags are considered ephemeral. The `versioning/tag_cleanup.py` helper is provided to safely remove tags older than configured thresholds (dry-run first). The repository includes a manual GitHub Action (`tag-cleanup.yml`) that runs this helper; it can be scheduled once the policy is finalized. + +## Local bump contract (inputs/outputs) + +- Input: `VERSION.meta` (canonical), current git branch +- Output: updated `VERSION.meta`, `VERSION` (simple string), and on CI a git tag pushed to origin with the resolved version string +- Error modes: git unavailable, malformed `VERSION.meta` or permissions to push in CI + +## Guidance for contributors + +- Do not hand-edit `VERSION` except for emergency/manual bumps. Prefer using the helper (`version_manager.py`) or relying on CI automation. +- If you need a local pre-release for testing, use a branch name that follows the conventions (e.g., `feat/…`, `fix/…`, or `dev`) and call `--update-for-branch` locally. +- The GitHub Actions require repository write permissions for commits and tags; the `commit_version_tag` job sets `contents: write` to allow committing and pushing version files and tags. + +## Summary mapping to requirements + +- Automated versioning scripts: documented (`versioning/version_manager.py`, `versioning/prepare_version.py`, `versioning/tag_cleanup.py`) — Done +- GitHub Actions that run the automation and create tags/releases: documented (`.github/workflows/test_build.yml`, `.github/workflows/commit_version_tag.yml`, `.github/workflows/tag-release.yml`, `.github/workflows/tag-cleanup.yml`) — Done diff --git a/docs/articles/devs/development_processes/developer_onboarding.md b/docs/articles/devs/development_processes/developer_onboarding.md new file mode 100644 index 0000000..5109e06 --- /dev/null +++ b/docs/articles/devs/development_processes/developer_onboarding.md @@ -0,0 +1,184 @@ +# Developer Onboarding + +This article is about: + +- Quick start guide for new developers joining the Hatch project +- Essential knowledge and resources for getting productive quickly +- Step-by-step onboarding process from setup to first contribution + +## Welcome to Hatch Development + +This guide will help you get up to speed quickly as a new developer on the Hatch package manager project. Follow this step-by-step process to understand the system and make your first contribution. + +## Step 1: Understand What Hatch Is + +### Project Overview + +Hatch is a package management system designed for MCP server packages, environments, and registry interactions. + +### Key Concepts + +- **Environments** - Isolated spaces for package installations +- **Installers** - Modular components that handle different dependency types +- **Registry** - Remote package discovery and retrieval system +- **Orchestrator** - Coordinates multi-type dependency installation + +## Step 2: Set Up Your Development Environment + +### Prerequisites + +- Python 3.12+ installed +- Conda or Mamba for Python environment management +- Git for version control +- Code editor +- Terminal/command line access + +### Clone and Setup + +```bash +# Clone the repository +git clone https://github.com/CrackingShells/Hatch.git +cd Hatch + +# Create virtual environment +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate + +# Install dependencies +pip install -e . +``` + +### Verify Installation + +```bash +# Run basic tests to verify setup +python run_tests.py --integration + +# Test CLI functionality +hatch --help +``` + +## Step 3: Explore the Codebase + +### Start with High-Level Architecture + +1. **Read** [System Overview](../architecture/system_overview.md) - Understand the big picture +2. **Review** [Architecture Diagram](../../resources/diagrams/architecture.puml) - Visual system layout +3. **Examine** [Component Architecture](../architecture/component_architecture.md) - Detailed component breakdown + +### Key Files to Understand + +```txt +hatch/ +├── cli_hatch.py # Main CLI entry point +├── environment_manager.py # Environment lifecycle management +├── package_loader.py # Package loading and validation +├── registry_retriever.py # Package downloads and caching +└── installers/ # Installation system + ├── dependency_installation_orchestrator.py + ├── installer_base.py + └── [specific installers] +``` + +### Explore by Running Examples + +```bash +# Create a test environment +python -m hatch env create test-env + +# List environments +python -m hatch env list + +# Install a package (if available) +python -m hatch install example-package + +# Clean up +python -m hatch env remove test-env +``` + +## Step 4: Understand the Development Workflow + +### Testing Strategy + +Hatch uses a three-tier testing approach: + +- **Development tests** (`dev_test_*.py`) - Temporary validation during development +- **Regression tests** (`regression_test_*.py`) - Permanent tests preventing regressions +- **Feature tests** (`feature_test_*.py`) - Permanent tests for new features + +### Running Tests + +```bash +# Run all tests +python run_tests.py + +# Run specific test types +python run_tests.py --regression +python run_tests.py --feature + +# Run tests for specific component +python run_tests.py --pattern "*environment*" +``` + +### Code Quality Standards + +- Follow organization-wide Python coding standards +- Write comprehensive docstrings +- Implement proper error handling and logging +- Include progress reporting for long-running operations + +## Step 5: Make Your First Contribution + +### Choose a Good First Issue + +Look for issues labeled: + +- `good first issue` - Beginner-friendly tasks +- `documentation` - Documentation improvements +- `testing` - Test additions or improvements + +### Simple Contribution Ideas + +1. **Add a test case** - Find a component with incomplete test coverage +2. **Improve documentation** - Add examples or clarify existing docs +3. **Fix a small bug** - Look for simple bug reports +4. **Add error handling** - Improve error messages or edge case handling + +### Follow the Contribution Process + +1. **Create a branch** - Use descriptive naming compatible with automated versioning (`feat/`, `fix/`, `docs/`) +2. **Make your changes** - Follow coding standards and include tests +3. **Test thoroughly** - Ensure all tests pass +4. **Submit a pull request** - Follow the [contribution guidelines](../contribution_guides/how_to_contribute.md) + +## Step 6: Learn Advanced Topics + +### As You Get More Comfortable + +1. **Installer Framework** - Learn how to [add new installers](../implementation_guides/adding_installers.md) +2. **Orchestration System** - Understand [installation orchestration](../implementation_guides/installation_orchestration.md) +3. **Registry Integration** - Work with [registry systems](../implementation_guides/registry_integration.md) + +### Community + +- **GitHub Issues** - Bug reports and feature requests +- **GitHub Discussions** - Questions and general discussion +- **Pull Request Reviews** - Learn from code review feedback + +## Next Steps + +### After Your First Contribution + +1. **Reflect on the process** - What was easy? What was challenging? +2. **Identify areas of interest** - Which parts of the system interest you most? +3. **Take on larger tasks** - Gradually work on more complex features +4. **Help other new contributors** - Share your onboarding experience + +### Becoming a Regular Contributor + +1. **Specialize in an area** - Become an expert in specific components +2. **Review others' contributions** - Help with code reviews and testing +3. **Improve documentation** - Keep documentation current and helpful +4. **Mentor new developers** - Help onboard future contributors + +Welcome to the Hatch development community! We're excited to have you contribute to making package management better for the CrackingShells ecosystem. diff --git a/docs/articles/devs/development_processes/index.md b/docs/articles/devs/development_processes/index.md new file mode 100644 index 0000000..b8fc303 --- /dev/null +++ b/docs/articles/devs/development_processes/index.md @@ -0,0 +1,51 @@ +# Development Processes + +This section covers workflow, standards, and processes for effective development on the Hatch project. + +## Overview + +These documents provide the essential processes, standards, and workflows that developers need to follow when working on the Hatch codebase. They focus on "how to work" rather than "what to implement." + +## Available Documentation + +- **[Developer Onboarding](./developer_onboarding.md)** - Quick start guide for new developers joining the project +- **[Testing Standards](./testing_standards.md)** - Testing requirements, patterns, and best practices + +## Development Standards + +### Code Quality + +- Follow organization-wide Python coding standards +- Implement comprehensive error handling and logging +- Write clear, self-documenting code with appropriate comments +- Include docstrings following the organization's docstring standards + +### Testing Requirements + +- Write tests for all new functionality +- Follow the three-tier testing approach: Development, Regression, Feature +- Ensure adequate test coverage for critical paths +- Use the centralized test runner (`run_tests.py`) + +### Documentation Standards + +- Update documentation when implementing new features +- Follow the organization's documentation guidelines +- Reference API docstrings rather than duplicating implementation details +- Maintain clear cross-references between related topics + +## Quick Reference + +### Organization Standards + +All development must follow the [organization-wide](https://github.com/CrackingShells/.github) standards defined in: + +- `instructions/documentation.instructions.md` - Documentation standards +- `instructions/python_docstrings.instructions.md` - Docstring format requirements +- `instructions/testing.instructions.md` - Testing guidelines + +## Related Documentation + +- [Architecture](../architecture/) - Understanding the system you're working on +- [Implementation Guides](../implementation_guides/) - Technical how-to guides for specific features +- [Contribution Guidelines](../contribution_guides/) - Process for contributing your work diff --git a/docs/articles/devs/development_processes/testing_standards.md b/docs/articles/devs/development_processes/testing_standards.md new file mode 100644 index 0000000..5712f71 --- /dev/null +++ b/docs/articles/devs/development_processes/testing_standards.md @@ -0,0 +1,268 @@ +# Testing Standards + +This article is about: + +- Testing requirements, patterns, and best practices for Hatch development +- Test organization and naming conventions +- Testing tools and infrastructure usage + +## Overview + +Hatch follows comprehensive testing standards to ensure code quality, prevent regressions, and validate new functionality. All contributors must understand and follow these testing standards when making changes to the codebase. + +## Testing Philosophy + +### Quality Assurance Goals + +- **Prevent Regressions** - Ensure existing functionality continues to work +- **Validate New Features** - Confirm new functionality works as designed +- **Enable Refactoring** - Provide confidence when improving code structure +- **Document Behavior** - Tests serve as executable documentation + +### Testing Principles + +- **Test Early and Often** - Write tests as you develop, not after --> prefix with `dev_test_*.py` +- **Test at Multiple Levels** - Unit, integration, and end-to-end testing +- **Test Edge Cases** - Cover error conditions and boundary cases +- **Keep Tests Simple** - Each test should verify one specific behavior + +## Test Organization + +### Test Runner + +All tests are executed via the central test runner: `run_tests.py` located in project roots. `run_tests.py` dispatches to the Python standard library `unittest` test discovery by default, and it provides flags for selecting test types (development/regression/feature) and verbosity. + +Example usage: + +```bash +# Run all tests (uses unittest discovery) +python run_tests.py + +# Run specific test types +python run_tests.py --development +python run_tests.py --regression +python run_tests.py --feature + +# Run tests matching pattern (if supported by run_tests.py) +python run_tests.py --pattern "*environment*" + +# Run with verbose output +python run_tests.py --verbose +``` + +### Test File Naming Conventions + +In this repository the common and established pattern is `test_*.py` files under the `tests/` folder (for example `tests/test_python_installer.py`). + +Guidance: + +- Primary pattern: `test_.py` (e.g. `test_python_installer.py`, `test_registry.py`). +- Tests are organized by topic and typically include `unittest.TestCase` subclasses and helper functions. +- If you need to create temporary developer-only tests you may prefix them with `dev_` (for example `dev_test_new_feature.py`) or place them in a `tests/dev/` directory, but prefer landing permanent tests as `test_*.py` so they are discoverable by default. + +Examples you will find in the repository: + +```txt +tests/ +├── test_python_installer.py +├── test_registry.py +├── test_env_manip.py +└── test_python_environment_manager.py +``` + +### Test File Placement + +Place test files in dedicated `tests/` directories at the project root: + +```txt +Hatch/ +├── tests/ +│ ├── dev_test_environment_manager.py +│ ├── regression_test_package_installation.py +│ └── feature_test_python_env_integration.py +├── hatch/ +└── run_tests.py +``` + +## Test Types and Lifecycle + +### Test Types & Lifecycle + +The codebase primarily uses `unittest` test files named `test_*.py`. Tests fall into three practical categories (development, regression, feature) but the repository's pattern is to keep the discoverable filename as `test_*.py`. Use directory layout or filename prefixes (for example `tests/dev/` or `dev_test_*.py`) to mark transient development tests. + +Characteristics by category: + +- Development tests: temporary, may be placed under `tests/dev/` or prefixed with `dev_`. Remove or convert before merging. +- Regression tests: permanent, cover previously fixed bugs and stable behavior. Keep these in `tests/` with a clear name and thorough assertions. +- Feature tests: permanent, cover new feature behavior and edge cases; these can become regression tests over time. + +Key rule: make tests discoverable by `python -m unittest discover -s tests -p "test_*.py"` and use clear names and docstrings to describe purpose. + +### Repository test patterns (what you'll see) + +Practical patterns used across the existing `tests/` files — follow these so tests are consistent and maintainable: + +- Test files are named `test_*.py` and live under `tests/`. +- Tests use `unittest.TestCase` subclasses and the standard assertion methods (`self.assertEqual`, `self.assertTrue`, `self.assertRaises`, etc.). When adding tests, prefer the `unittest` assertion methods for clearer error messages and consistency. +- Use `setUp`/`tearDown` for per-test setup/cleanup. For slower integration suites use `setUpClass`/`tearDownClass` to prepare/clean shared resources. +- Temporary filesystem resources are created with `tempfile` and cleaned with `shutil.rmtree` in `tearDown`. +- Use `unittest.mock.patch` frequently; decorators are used to patch functions or methods at import paths (e.g., `@patch('hatch.module.Class.method')`). +- Some tests modify `sys.path` at module top-level to import local packages for direct testing; prefer installing the package in editable mode during development, but keep `sys.path` inserts when necessary for simple test execution. +- Integration tests often guard against missing external tools and call `unittest.SkipTest` or raise SkipTest in `setUpClass` to avoid running on systems without required dependencies. +- Tests commonly include a `if __name__ == '__main__':` guard that calls `unittest.main()` (often with `verbosity=2`) so tests can be run directly. +- Use `unittest-xml-reporting` in CI to produce xUnit XML reports if required by the CI system; tests themselves don't need to change to support this. + +When writing or updating tests, mirror these patterns so other contributors won't be surprised when cross-checking documentation and implementation. + +## Testing Patterns and Best Practices + +### Test Structure + +Follow the Arrange-Act-Assert pattern. Prefer `unittest.TestCase` methods to keep examples consistent with the repository tests: + +```python +import unittest + +class TestPackageLoading(unittest.TestCase): + def test_package_loading(self): + # Arrange - Set up test data and conditions + package_path = create_test_package() + loader = HatchPackageLoader() + + # Act - Perform the action being tested + metadata = loader.load_package(package_path) + + # Assert - Verify the expected outcome + self.assertEqual(metadata.name, "test-package") + self.assertEqual(metadata.version, "1.0.0") + +``` + +### Mocking External Dependencies + +Use mocks for external systems and dependencies (use `unittest.mock` which is part of the standard library): + +```python +from unittest.mock import Mock, patch +import unittest + +def test_registry_retrieval_with_network_error(): + """Test registry retrieval handles network errors gracefully.""" + with patch('requests.get') as mock_get: + mock_get.side_effect = requests.ConnectionError("Network error") + + retriever = RegistryRetriever() + + with unittest.TestCase().assertRaises(NetworkError): + retriever.retrieve_package("test-package") +``` + +### Fixture Usage + +In `unittest` use `setUp`/`tearDown` or class-level `setUpClass`/`tearDownClass` for shared fixtures. + +```python +import unittest + +class TestPackageInstallation(unittest.TestCase): + def setUp(self): + self.manager = HatchEnvironmentManager() + self.env_name = "test-fixture-env" + self.manager.create_environment(self.env_name) + + def tearDown(self): + if self.manager.environment_exists(self.env_name): + self.manager.remove_environment(self.env_name) + + def test_package_installation(self): + self.assertTrue(self.manager.environment_exists(self.env_name)) + # further assertions here +``` + +## Testing Tools and Infrastructure + +### Test Configuration + +Unittest does not use a central ini file; configuration is handled by `run_tests.py` and by CI job configuration. Recommended conventions: + +- Tests live under `tests/` and follow the file naming patterns described above (e.g. `dev_test_*.py`, `regression_test_*.py`, `feature_test_*.py`). +- Use `python -m unittest discover -s tests -p "*_test_*.py"` for discovery when running directly. +- Use `coverage` to collect coverage and enforce thresholds (examples below). + +## Testing Specific Components + +### Environment Management Testing + +```python +import unittest + +class TestEnvironmentIsolation(unittest.TestCase): + def test_environment_isolation(self): + """Test that environments are properly isolated.""" + manager = HatchEnvironmentManager() + + # Create two environments + env1 = manager.create_environment("env1") + env2 = manager.create_environment("env2") + + # Install different packages in each + env1.install_package("package-a") + env2.install_package("package-b") + + # Verify isolation + self.assertIn("package-a", env1.list_packages()) + self.assertNotIn("package-a", env2.list_packages()) + +``` + +### Installer Testing + +```python +import unittest + +class TestInstallerErrorHandling(unittest.TestCase): + def test_installer_error_handling(self): + """Test installer handles errors gracefully.""" + installer = PythonInstaller() + invalid_dependency = {"type": "python", "name": "nonexistent-package"} + context = InstallationContext() + + result = installer.install_dependency(invalid_dependency, context) + + self.assertFalse(result.success) + self.assertIn("not found", result.error_message.lower()) + +``` + +### Registry Testing + +```python +import unittest +from unittest.mock import patch + +class TestRegistryCaching(unittest.TestCase): + def test_registry_caching(self): + """Test registry caching behavior.""" + retriever = RegistryRetriever() + package_name = "test-package" + + # First retrieval should hit network + with patch('requests.get') as mock_get: + mock_get.return_value.json.return_value = {"version": "1.0.0"} + + result1 = retriever.get_package_info(package_name) + self.assertEqual(mock_get.call_count, 1) + + # Second retrieval should use cache + with patch('requests.get') as mock_get: + result2 = retriever.get_package_info(package_name) + self.assertEqual(mock_get.call_count, 0) # No network call + self.assertEqual(result1, result2) + +``` + +## Related Documentation + +- [Development Environment Setup](./development_environment_setup.md) - Setting up testing environment +- [Contribution Guidelines](../contribution_guides/how_to_contribute.md) - Testing requirements for contributions +- [Implementation Guides](../implementation_guides/) - Testing specific components diff --git a/docs/articles/devs/implementation_guides/adding_installers.md b/docs/articles/devs/implementation_guides/adding_installers.md new file mode 100644 index 0000000..8f56a2b --- /dev/null +++ b/docs/articles/devs/implementation_guides/adding_installers.md @@ -0,0 +1,113 @@ +# Adding New Installers + +**Quick Start:** Copy an existing installer, modify it, register it. Most installers are 100-200 lines. + +## When You Need This + +You want Hatch to install dependencies from a new source: + +- A package manager not yet supported (brew, pacman, etc.) +- A custom artifact repository +- Cloud services +- Version control systems beyond what's supported + +## The Pattern + +All installers implement `DependencyInstaller` and get registered with `installer_registry`. The orchestrator finds the right installer by calling `can_install()` on each one. + +**Core interface** (from `hatch/installers/installer_base.py`): + +```python +@property +def installer_type(self) -> str: # "python", "system", etc. + +def can_install(self, dependency: Dict[str, Any]) -> bool: # Can handle this? + +def install(self, dependency: Dict[str, Any], context: InstallationContext) -> InstallationResult: # Do it + +# and so on +``` + +## Implementation Steps + +### 1. Copy an Existing Installer + +Start with `hatch/installers/system_installer.py` - it's straightforward and handles subprocess calls well. + +### 2. Modify the Key Methods + +```python +# my_installer.py +class MyInstaller(DependencyInstaller): + @property + def installer_type(self) -> str: + return "my-type" # What goes in dependency["type"] + + def can_install(self, dependency: Dict[str, Any]) -> bool: + # Return True if you can handle this dependency + return dependency.get("type") == "my-type" + + def install(self, dependency: Dict[str, Any], context: InstallationContext) -> InstallationResult: + # Your installation logic here + name = dependency["name"] + # ... do the work ... + return InstallationResult( + dependency_name=name, + status=InstallationStatus.COMPLETED, + installed_path=some_path, + installed_version=some_version + ) +``` + +### 3. Register It + +Add to the bottom of your installer file: + +```python +from .registry import installer_registry +installer_registry.register_installer("my-type", MyInstaller) +``` + +That's it. The orchestrator will find and use your installer automatically. + +## Practical Tips + +**Error handling:** Catch exceptions and wrap them in `InstallationError`: + +```python +try: + # your installation logic +except Exception as e: + raise InstallationError(f"Failed to install {name}: {e}") from e +``` + +**Subprocess calls:** Use the patterns from `SystemInstaller._run_subprocess()` for external commands. + +**Progress reporting:** The `context` parameter has a `report_progress()` method if users need feedback. + +**Validation:** Implement `validate_dependency()` to check dependency format before attempting installation. + +## Testing + +Copy test patterns from `tests/test_system_installer.py`: + +- Mock external calls with `unittest.mock.patch` +- Test both success and failure cases +- Use `InstallationContext` with a temp directory for isolation + +## Real Examples + +Look at existing installers to understand patterns: + +- `SystemInstaller` - subprocess calls to apt +- `PythonInstaller` - subprocess calls to pip +- `DockerInstaller` - Docker API calls +- `HatchInstaller` - file operations for local packages + +## Common Gotchas + +**Don't overthink it:** Most installers are simple - check if you can handle it, then do the installation. + +**Context matters:** Use the `InstallationContext` for target paths, environment info, and progress reporting. + +**Error messages:** Make them actionable. "Permission denied installing X" is better than "Installation failed". \ No newline at end of file diff --git a/docs/articles/devs/implementation_guides/index.md b/docs/articles/devs/implementation_guides/index.md new file mode 100644 index 0000000..134d273 --- /dev/null +++ b/docs/articles/devs/implementation_guides/index.md @@ -0,0 +1,33 @@ +# Implementation Guides + +Quick, actionable guidance for extending Hatch functionality. These guides focus on **what you need to know** to get started, not exhaustive implementation details. + +## When to Use These Guides + +You're working on Hatch and need to: + +- Add support for a new dependency type (installer) +- Customize package loading or validation +- Work with the package registry system +- Understand how installation coordination works + +## Available Guides + +### Adding Functionality + +- **[Adding New Installers](./adding_installers.md)** - Support new dependency types (5-minute read) +- **[Package Loading Extensions](./package_loader_extensions.md)** - Custom package formats and validation (3-minute read) + +### Working with Existing Systems + +- **[Registry Integration](./registry_integration.md)** - Package discovery and caching (4-minute read) +- **[Installation Orchestration](./installation_orchestration.md)** - How dependency installation is coordinated (3-minute read) + +## Key Patterns to Know + +Hatch uses these patterns consistently: + +- **Strategy Pattern** - Multiple installers implement the same interface (`DependencyInstaller`) +- **Registry Pattern** - Global `installer_registry` maps dependency types to installers + +The code is well-documented and often clearer than additional documentation. diff --git a/docs/articles/devs/implementation_guides/installation_orchestration.md b/docs/articles/devs/implementation_guides/installation_orchestration.md new file mode 100644 index 0000000..87f745d --- /dev/null +++ b/docs/articles/devs/implementation_guides/installation_orchestration.md @@ -0,0 +1,286 @@ +# Installation Orchestration + +**Quick Start:** Coordinate complex package installations with dependencies, environment setup, and post-install tasks. + +> [!Warning] +> The orchestrator has not been specifically implemented with extensibility in mind. The current implementation is very specific to the current Hatch architecture and use cases. However, the following guide gives you general pointers in case you need a custom installation workflow. + +## When You Need This + +You want to customize how Hatch manages package installations: + +- Complex multi-step installation workflows +- Custom dependency resolution logic +- Environment-specific installation steps +- Integration with external installation tools + +## The Pattern + +The `InstallationOrchestrator` coordinates all installation components. You extend it to: + +1. Add custom installation steps +2. Modify dependency resolution +3. Handle specialized package types +4. Integrate external tools + +## Common Orchestrations + +Here are some ideas for custom implementations. These are NOT in the existing codebase. + +### Custom Installation Steps + +```python +# custom_orchestrator.py +class CustomInstallationOrchestrator(InstallationOrchestrator): + def install_package(self, package_name: str, version: str = None, target_env: str = None) -> bool: + # Pre-installation validation + if not self._validate_installation_requirements(package_name, version): + raise InstallationError(f"Requirements not met for {package_name}") + + # Custom installation steps + package_path = self._download_and_prepare(package_name, version) + self._run_pre_install_hooks(package_path) + + # Standard installation + success = super().install_package(package_name, version, target_env) + + if success: + self._run_post_install_hooks(package_path, target_env) + self._update_installation_registry(package_name, version, target_env) + + return success + + def _validate_installation_requirements(self, package_name: str, version: str) -> bool: + # Check system requirements, disk space, permissions, etc. + return True + + def _run_pre_install_hooks(self, package_path: Path): + # Custom pre-installation tasks + hook_script = package_path / "pre_install.py" + if hook_script.exists(): + subprocess.run([sys.executable, str(hook_script)], check=True) + + def _run_post_install_hooks(self, package_path: Path, target_env: str): + # Custom post-installation tasks + hook_script = package_path / "post_install.py" + if hook_script.exists(): + env = os.environ.copy() + env["HATCH_TARGET_ENV"] = target_env + subprocess.run([sys.executable, str(hook_script)], env=env, check=True) +``` + +### Dependency Resolution Strategy + +```python +class SmartDependencyOrchestrator(InstallationOrchestrator): + def __init__(self, conflict_resolution="latest"): + super().__init__() + self.conflict_resolution = conflict_resolution + + def resolve_dependencies(self, package_name: str, version: str) -> List[Tuple[str, str]]: + # Get package metadata + metadata = self.registry_retriever.get_package_metadata(package_name, version) + dependencies = metadata.get("dependencies", {}) + + # Build dependency tree + resolved = [] + for dep_name, dep_constraint in dependencies.items(): + dep_version = self._resolve_version_constraint(dep_name, dep_constraint) + resolved.append((dep_name, dep_version)) + + # Recursively resolve dependencies + sub_deps = self.resolve_dependencies(dep_name, dep_version) + resolved.extend(sub_deps) + + # Handle conflicts + return self._resolve_conflicts(resolved) + + def _resolve_version_constraint(self, package_name: str, constraint: str) -> str: + available_versions = self.registry_retriever.get_package_versions(package_name) + # Apply constraint logic (semver, etc.) + return self._pick_best_version(available_versions, constraint) + + def _resolve_conflicts(self, dependencies: List[Tuple[str, str]]) -> List[Tuple[str, str]]: + # Group by package name + by_package = {} + for name, version in dependencies: + if name not in by_package: + by_package[name] = [] + by_package[name].append(version) + + # Resolve conflicts based on strategy + resolved = [] + for package_name, versions in by_package.items(): + if len(versions) == 1: + resolved.append((package_name, versions[0])) + else: + chosen_version = self._choose_version(versions) + resolved.append((package_name, chosen_version)) + + return resolved +``` + +### Multi-Environment Installation + +```python +class MultiEnvOrchestrator(InstallationOrchestrator): + def install_to_environments(self, package_name: str, version: str, environments: List[str]) -> Dict[str, bool]: + results = {} + + for env in environments: + try: + # Environment-specific configuration + env_config = self._get_environment_config(env) + + # Install with environment-specific settings + success = self._install_to_specific_env(package_name, version, env, env_config) + results[env] = success + + except Exception as e: + results[env] = False + self._log_installation_error(env, package_name, version, e) + + return results + + def _get_environment_config(self, env: str) -> Dict: + config_map = { + "development": {"debug": True, "test_dependencies": True}, + "production": {"debug": False, "optimize": True}, + "testing": {"debug": True, "mock_external": True} + } + return config_map.get(env, {}) + + def _install_to_specific_env(self, package_name: str, version: str, env: str, config: Dict) -> bool: + # Custom installation logic per environment + if env == "production": + return self._production_install(package_name, version, config) + elif env == "development": + return self._development_install(package_name, version, config) + else: + return super().install_package(package_name, version, env) +``` + +### External Tool Integration + +```python +class IntegratedOrchestrator(InstallationOrchestrator): + def __init__(self, external_tools: Dict[str, str] = None): + super().__init__() + self.external_tools = external_tools or {} + + def install_package(self, package_name: str, version: str = None, target_env: str = None) -> bool: + # Check if package requires external tools + metadata = self.registry_retriever.get_package_metadata(package_name, version) + external_deps = metadata.get("external_dependencies", []) + + # Install external dependencies first + for ext_dep in external_deps: + if not self._install_external_dependency(ext_dep): + raise InstallationError(f"Failed to install external dependency: {ext_dep}") + + # Proceed with standard installation + return super().install_package(package_name, version, target_env) + + def _install_external_dependency(self, dependency: str) -> bool: + # Handle different external tools + if dependency.startswith("apt:"): + return self._install_via_apt(dependency[4:]) + elif dependency.startswith("brew:"): + return self._install_via_brew(dependency[5:]) + elif dependency.startswith("conda:"): + return self._install_via_conda(dependency[6:]) + else: + return self._install_via_generic_tool(dependency) +``` + +## Integration Points + +Here are illustrations of how to integrate your custom orchestrator. + +### With Environment Manager + +```python +# Orchestrator works with environment manager +env_manager = HatchEnvironmentManager() +orchestrator = CustomInstallationOrchestrator( + environment_manager=env_manager, + registry_retriever=custom_registry +) +``` + +### With Package Validation + +```python +class ValidatingOrchestrator(InstallationOrchestrator): + def install_package(self, package_name: str, version: str = None, target_env: str = None) -> bool: + # Download and validate package before installation + package_path = self.registry_retriever.download_package(package_name, version, self.temp_dir) + + if not self.package_validator.validate_package(package_path): + raise InstallationError(f"Package validation failed: {package_name}") + + return super().install_package(package_name, version, target_env) +``` + +### Configuration-Driven Orchestration + +```python +class ConfigurableOrchestrator(InstallationOrchestrator): + def __init__(self, config_path: Path): + super().__init__() + self.config = self._load_config(config_path) + self._setup_from_config() + + def _setup_from_config(self): + # Configure components based on config file + if "registry" in self.config: + self.registry_retriever = self._create_registry_from_config(self.config["registry"]) + + if "installers" in self.config: + self._register_installers_from_config(self.config["installers"]) +``` + +## Testing Orchestration + +Test complex workflows with mocks: + +```python +class TestCustomOrchestrator(unittest.TestCase): + def setUp(self): + self.mock_registry = Mock(spec=RegistryRetriever) + self.mock_env_manager = Mock(spec=HatchEnvironmentManager) + self.orchestrator = CustomInstallationOrchestrator( + registry_retriever=self.mock_registry, + environment_manager=self.mock_env_manager + ) + + def test_multi_step_installation(self): + # Set up mocks + self.mock_registry.download_package.return_value = Path("/tmp/test-pkg") + + # Test installation + success = self.orchestrator.install_package("test-pkg", "1.0.0") + + # Verify all steps were called + self.assertTrue(success) + self.mock_registry.download_package.assert_called_once() +``` + +## Practical Tips + +**Error recovery:** Implement rollback logic for failed installations. + +**Logging:** Log each step for debugging complex installation workflows. + +**Timeouts:** Set reasonable timeouts for external tool calls. + +**Parallel installations:** Be careful with concurrent installations - use locks when needed. + +**Configuration:** Make orchestration behavior configurable rather than hardcoded. + +## Real Examples + +Check existing patterns: + +- `InstallationOrchestrator` in `hatch/installation_orchestrator.py` - base implementation +- Tests in `tests/test_*orchestrator*.py` - orchestration testing patterns diff --git a/docs/articles/devs/implementation_guides/package_loader_extensions.md b/docs/articles/devs/implementation_guides/package_loader_extensions.md new file mode 100644 index 0000000..1c027a1 --- /dev/null +++ b/docs/articles/devs/implementation_guides/package_loader_extensions.md @@ -0,0 +1,523 @@ +# Package Loading Extensions + +**Quick Start:** Extend `HatchPackageLoader` to support new metadata formats or add custom validation rules. + +> [!Warning] +> The package loader has not been specifically implemented with extensibility in mind. The current implementation is very specific to the current Hatch architecture and use cases. However, the following guide gives you general pointers in case you need a custom package loading workflow. + +## When You Need This + +You want to customize how Hatch loads and validates packages: + +- Support metadata in YAML/TOML instead of JSON +- Add custom validation rules beyond schema validation +- Transform metadata for different environments +- Integrate with external validation systems + +## The Pattern + +The `HatchPackageLoader` loads `hatch_metadata.json` and validates it. You extend it to: + +1. Support additional metadata formats +2. Add custom validation steps +3. Transform metadata during loading + +## Common Extensions + +Here are some ideas for custom implementations. These are NOT in the existing codebase. + +### Supporting Multiple Metadata Formats + +```python +# custom_loader.py +class MultiFormatLoader(HatchPackageLoader): + def load_package(self, package_path: Path) -> PackageMetadata: + # Try different metadata file formats + for filename in ["hatch_metadata.json", "hatch_metadata.yaml", "pyproject.toml"]: + metadata_file = package_path / filename + if metadata_file.exists(): + metadata = self._load_by_format(metadata_file) + return self.validate_and_parse(metadata, package_path) + + raise PackageLoadError("No supported metadata file found") + + def _load_by_format(self, file_path: Path) -> Dict: + if file_path.suffix == ".json": + return json.load(file_path.open()) + elif file_path.suffix == ".yaml": + return yaml.safe_load(file_path.open()) + # etc. +``` + +### Custom Validation Rules + +```python +class ValidatingLoader(HatchPackageLoader): + def validate_and_parse(self, metadata: Dict, package_path: Path) -> PackageMetadata: + # Run standard validation first + result = super().validate_and_parse(metadata, package_path) + + # Add custom validation + self._validate_entry_points_exist(metadata, package_path) + self._validate_license_file_exists(metadata, package_path) + + return result + + def _validate_entry_points_exist(self, metadata: Dict, package_path: Path): + entry_point = metadata.get("entry_point") + if entry_point and not (package_path / entry_point).exists(): + raise ValidationError(f"Entry point file not found: {entry_point}") +``` + +### Environment-Specific Transformations + +```python +class EnvironmentLoader(HatchPackageLoader): + def __init__(self, target_env="production"): + super().__init__() + self.target_env = target_env + + def validate_and_parse(self, metadata: Dict, package_path: Path) -> PackageMetadata: + # Transform metadata for target environment + if self.target_env == "production": + # Remove development dependencies + metadata.get("dependencies", {}).pop("development", None) + + return super().validate_and_parse(metadata, package_path) +``` + +## Integration Points + +Similarly as above, here are illustrations of how to integrate your custom loader. + +### With Hatch Environment Manager + +Replace the default loader: + +```python +# In your environment manager code +custom_loader = MultiFormatLoader() +environment_manager = HatchEnvironmentManager(package_loader=custom_loader) +``` + +### With Registry Operations + +Registry operations use the loader for downloaded packages. Your custom loader will be used automatically if you pass it to the orchestrator. + +### With Validation Systems + +Integrate with external validators: + +```python +class SchemaValidatingLoader(HatchPackageLoader): + def __init__(self, external_validator): + super().__init__() + self.external_validator = external_validator + + def validate_and_parse(self, metadata: Dict, package_path: Path) -> PackageMetadata: + # Use external validation service + if not self.external_validator.validate(metadata): + raise ValidationError("External validation failed") + + return super().validate_and_parse(metadata, package_path) +``` + +## Testing Extensions + +Test your extensions like any other component: + +```python +class TestCustomLoader(unittest.TestCase): + def test_yaml_metadata_loading(self): + loader = MultiFormatLoader() + # Create test package with YAML metadata + metadata = loader.load_package(test_package_path) + self.assertEqual(metadata.name, "expected-name") +``` + +## Practical Tips + +**Start simple:** Most use cases need only 1-2 method overrides. + +**Chain validations:** Call `super().validate_and_parse()` first, then add your custom logic. + +**Error messages:** Make validation errors specific and actionable. + +**Performance:** Cache expensive validation operations if you're processing many packages. + +## Real Examples + +Check existing code for patterns: + +- `HatchPackageLoader` in `hatch/package_loader.py` - base implementation +- Tests in `tests/test_*loader*.py` - testing patterns +- Validation in `Hatch-Validator/` project - schema validation examples + +```python +class EnhancedPackageValidator: + """Enhanced package validator with custom rules.""" + + def __init__(self, base_validator): + self.base_validator = base_validator + self.custom_validators = [] + + def add_custom_validator(self, validator_func): + """Add custom validation function.""" + self.custom_validators.append(validator_func) + + def validate_package(self, metadata: Dict[str, Any], package_path: Path) -> ValidationResult: + """Validate package with base and custom validators.""" + # Run base schema validation + base_result = self.base_validator.validate(metadata) + + if not base_result.is_valid: + return base_result + + # Run custom validators + for validator in self.custom_validators: + custom_result = validator(metadata, package_path) + if not custom_result.is_valid: + return custom_result + + return ValidationResult(is_valid=True) + +# Example custom validators +def validate_entry_points_exist(metadata: Dict[str, Any], package_path: Path) -> ValidationResult: + """Validate that entry point files actually exist.""" + entry_points = metadata.get("entry_points", {}) + + for entry_point_name, entry_point_path in entry_points.items(): + full_path = package_path / entry_point_path + + if not full_path.exists(): + return ValidationResult( + is_valid=False, + error_message=f"Entry point file not found: {entry_point_path}" + ) + + return ValidationResult(is_valid=True) + +def validate_dependency_versions(metadata: Dict[str, Any], package_path: Path) -> ValidationResult: + """Validate dependency version specifications.""" + dependencies = metadata.get("dependencies", {}) + + for dep_type, dep_list in dependencies.items(): + for dependency in dep_list: + version = dependency.get("version") + if version and not _is_valid_version_spec(version): + return ValidationResult( + is_valid=False, + error_message=f"Invalid version specification: {version}" + ) + + return ValidationResult(is_valid=True) +``` + +### Package Metadata Processing + +Extend metadata processing for specialized use cases: + +```python +class MetadataProcessor: + """Process and transform package metadata.""" + + def __init__(self): + self.processors = [] + + def add_processor(self, processor_func): + """Add metadata processing function.""" + self.processors.append(processor_func) + + def process_metadata(self, metadata: Dict[str, Any]) -> Dict[str, Any]: + """Apply all processors to metadata.""" + processed_metadata = metadata.copy() + + for processor in self.processors: + processed_metadata = processor(processed_metadata) + + return processed_metadata + +# Example processors +def normalize_dependency_versions(metadata: Dict[str, Any]) -> Dict[str, Any]: + """Normalize dependency version specifications.""" + dependencies = metadata.get("dependencies", {}) + + for dep_type, dep_list in dependencies.items(): + for dependency in dep_list: + if "version" in dependency: + dependency["version"] = _normalize_version_spec(dependency["version"]) + + return metadata + +def resolve_template_variables(metadata: Dict[str, Any]) -> Dict[str, Any]: + """Resolve template variables in metadata.""" + template_vars = metadata.get("template_vars", {}) + + def replace_vars(obj): + if isinstance(obj, str): + for var_name, var_value in template_vars.items(): + obj = obj.replace(f"${{{var_name}}}", str(var_value)) + return obj + elif isinstance(obj, dict): + return {k: replace_vars(v) for k, v in obj.items()} + elif isinstance(obj, list): + return [replace_vars(item) for item in obj] + return obj + + return replace_vars(metadata) +``` + +## Advanced Package Loading Features + +### Lazy Loading and Caching + +Implement lazy loading and caching for performance: + +```python +class CachedPackageLoader(HatchPackageLoader): + """Package loader with caching support.""" + + def __init__(self, cache_ttl=3600): + super().__init__() + self.cache = {} + self.cache_ttl = cache_ttl + + def load_package(self, package_path: Path) -> PackageMetadata: + """Load package with caching.""" + cache_key = str(package_path.resolve()) + + # Check cache + if cache_key in self.cache: + cached_entry = self.cache[cache_key] + if not self._is_cache_expired(cached_entry): + return cached_entry["metadata"] + + # Load and cache + metadata = super().load_package(package_path) + self.cache[cache_key] = { + "metadata": metadata, + "timestamp": time.time() + } + + return metadata + + def _is_cache_expired(self, cache_entry: Dict[str, Any]) -> bool: + """Check if cache entry has expired.""" + return time.time() - cache_entry["timestamp"] > self.cache_ttl + + def invalidate_cache(self, package_path: Path = None): + """Invalidate cache for specific package or all packages.""" + if package_path: + cache_key = str(package_path.resolve()) + self.cache.pop(cache_key, None) + else: + self.cache.clear() +``` + +### Package Dependency Resolution + +Implement dependency resolution during package loading: + +```python +class DependencyResolvingLoader(HatchPackageLoader): + """Package loader with dependency resolution.""" + + def __init__(self, registry_retriever): + super().__init__() + self.registry_retriever = registry_retriever + + def load_package_with_dependencies(self, package_path: Path) -> PackageWithDependencies: + """Load package and resolve its dependencies.""" + metadata = self.load_package(package_path) + + resolved_dependencies = self._resolve_dependencies(metadata.dependencies) + + return PackageWithDependencies( + metadata=metadata, + resolved_dependencies=resolved_dependencies + ) + + def _resolve_dependencies(self, dependencies: Dict[str, List[Dict]]) -> Dict[str, List[ResolvedDependency]]: + """Resolve dependency specifications to concrete versions.""" + resolved = {} + + for dep_type, dep_list in dependencies.items(): + resolved[dep_type] = [] + + for dependency in dep_list: + resolved_dep = self._resolve_single_dependency(dependency) + resolved[dep_type].append(resolved_dep) + + return resolved + + def _resolve_single_dependency(self, dependency: Dict[str, Any]) -> ResolvedDependency: + """Resolve a single dependency specification.""" + name = dependency["name"] + version_spec = dependency.get("version", "latest") + + # Query registry for available versions + available_versions = self.registry_retriever.get_package_versions(name) + + # Resolve version specification + resolved_version = self._resolve_version_spec(version_spec, available_versions) + + return ResolvedDependency( + name=name, + requested_version=version_spec, + resolved_version=resolved_version, + source=dependency.get("source", "registry") + ) +``` + +### Package Transformation and Adaptation + +Transform packages for different environments or use cases: + +```python +class PackageTransformer: + """Transform packages for different environments.""" + + def __init__(self): + self.transformers = {} + + def register_transformer(self, target_env: str, transformer_func): + """Register transformer for specific environment.""" + self.transformers[target_env] = transformer_func + + def transform_package(self, metadata: PackageMetadata, target_env: str) -> PackageMetadata: + """Transform package for target environment.""" + if target_env not in self.transformers: + return metadata # No transformation needed + + transformer = self.transformers[target_env] + return transformer(metadata) + +# Example transformers +def transform_for_production(metadata: PackageMetadata) -> PackageMetadata: + """Transform package for production environment.""" + # Remove development dependencies + if "dependencies" in metadata.raw_data: + dependencies = metadata.raw_data["dependencies"] + dependencies.pop("development", None) + + # Set production-specific configuration + metadata.raw_data["environment"] = "production" + metadata.raw_data["debug"] = False + + return PackageMetadata(metadata.raw_data) + +def transform_for_testing(metadata: PackageMetadata) -> PackageMetadata: + """Transform package for testing environment.""" + # Add test-specific dependencies + test_deps = [ + {"name": "pytest", "version": ">=6.0.0"}, + {"name": "pytest-mock", "version": ">=3.0.0"} + ] + + if "dependencies" not in metadata.raw_data: + metadata.raw_data["dependencies"] = {} + + metadata.raw_data["dependencies"]["testing"] = test_deps + + return PackageMetadata(metadata.raw_data) +``` + +## Integration with Validation System + +### Schema Management + +Work with external schema validation: + +```python +class SchemaAwareLoader(HatchPackageLoader): + """Package loader with schema version management.""" + + def __init__(self, schema_manager): + super().__init__() + self.schema_manager = schema_manager + + def load_package(self, package_path: Path) -> PackageMetadata: + """Load package with appropriate schema validation.""" + metadata = self._load_raw_metadata(package_path) + + # Determine schema version + schema_version = self._determine_schema_version(metadata) + + # Get appropriate schema + schema = self.schema_manager.get_schema(schema_version) + + # Validate against schema + validation_result = schema.validate(metadata) + if not validation_result.is_valid: + raise ValidationError(f"Schema validation failed: {validation_result.errors}") + + return self.parse_metadata(metadata, package_path) + + def _determine_schema_version(self, metadata: Dict[str, Any]) -> str: + """Determine appropriate schema version for metadata.""" + # Check explicit schema version + if "schema_version" in metadata: + return metadata["schema_version"] + + # Infer from metadata structure + if "hatch_version" in metadata: + return self._map_hatch_version_to_schema(metadata["hatch_version"]) + + # Default to latest + return "latest" +``` + +## Testing Package Loading Extensions + +### Unit Testing + +```python +class TestCustomPackageLoader: + def test_yaml_metadata_loading(self): + """Test loading YAML metadata files.""" + loader = CustomPackageLoader() + + # Create test package with YAML metadata + test_package_path = self._create_test_package_yaml() + + metadata = loader.load_package(test_package_path) + + assert metadata.name == "test-package" + assert metadata.version == "1.0.0" + + def test_custom_validation_rules(self): + """Test custom validation rules.""" + validator = EnhancedPackageValidator(base_validator) + validator.add_custom_validator(validate_entry_points_exist) + + # Test with missing entry point file + metadata = {"entry_points": {"main": "nonexistent.py"}} + package_path = Path("/tmp/test-package") + + result = validator.validate_package(metadata, package_path) + + assert not result.is_valid + assert "Entry point file not found" in result.error_message +``` + +### Integration Testing + +```python +def test_package_loading_with_registry_integration(): + """Test package loading with registry dependency resolution.""" + registry_retriever = MockRegistryRetriever() + loader = DependencyResolvingLoader(registry_retriever) + + package_path = create_test_package_with_dependencies() + + package_with_deps = loader.load_package_with_dependencies(package_path) + + assert len(package_with_deps.resolved_dependencies["python"]) > 0 + assert all(dep.resolved_version for dep in package_with_deps.resolved_dependencies["python"]) +``` + +## Related Documentation + +- [Registry Integration](./registry_integration.md) - Working with package registries +- [Component Architecture](../architecture/component_architecture.md) - Package loading system architecture +- [Testing Standards](../development_processes/testing_standards.md) - Testing package loading extensions diff --git a/docs/articles/devs/implementation_guides/registry_integration.md b/docs/articles/devs/implementation_guides/registry_integration.md new file mode 100644 index 0000000..f24f879 --- /dev/null +++ b/docs/articles/devs/implementation_guides/registry_integration.md @@ -0,0 +1,225 @@ +# Registry Integration + +**Quick Start:** Integrate custom registries with Hatch's package discovery and installation system. + +> [!Warning] +> The registry retriever has not been specifically implemented with extensibility in mind. The current implementation is very specific to the current Hatch architecture and use cases. However, the following guide gives you general pointers in case you need a custom registry integration. + +## When You Need This + +You want to use package sources beyond the default Hatch registry: + +- Private/corporate package repositories +- Local development registries +- Third-party package sources with different APIs +- Registry mirrors for specific environments + +## The Pattern + +Hatch uses `RegistryRetriever` to find and download packages. You extend it to: + +1. Support different registry APIs +2. Add authentication/credentials +3. Implement caching strategies +4. Handle registry-specific metadata + +## Common Integrations + +Here are some ideas for custom implementations. These are NOT in the existing codebase. + +### Private Registry with Authentication + +```python +# private_registry.py +class PrivateRegistryRetriever(RegistryRetriever): + def __init__(self, base_url: str, api_key: str): + super().__init__() + self.base_url = base_url + self.api_key = api_key + + def download_package(self, package_name: str, version: str, target_dir: Path) -> Path: + headers = {"Authorization": f"Bearer {self.api_key}"} + download_url = f"{self.base_url}/packages/{package_name}/{version}/download" + + response = requests.get(download_url, headers=headers) + response.raise_for_status() + + package_file = target_dir / f"{package_name}-{version}.zip" + package_file.write_bytes(response.content) + return package_file + + def get_package_versions(self, package_name: str) -> List[str]: + headers = {"Authorization": f"Bearer {self.api_key}"} + url = f"{self.base_url}/packages/{package_name}/versions" + + response = requests.get(url, headers=headers) + response.raise_for_status() + return response.json()["versions"] +``` + +### Local Development Registry + +```python +class LocalRegistryRetriever(RegistryRetriever): + def __init__(self, registry_path: Path): + super().__init__() + self.registry_path = registry_path + + def download_package(self, package_name: str, version: str, target_dir: Path) -> Path: + source_path = self.registry_path / package_name / version + if not source_path.exists(): + raise PackageNotFoundError(f"Package {package_name}=={version} not found locally") + + # Copy to target directory + package_dir = target_dir / f"{package_name}-{version}" + shutil.copytree(source_path, package_dir) + return package_dir + + def get_package_versions(self, package_name: str) -> List[str]: + package_path = self.registry_path / package_name + if not package_path.exists(): + return [] + + return [d.name for d in package_path.iterdir() if d.is_dir()] +``` + +### Registry with Caching + +```python +class CachedRegistryRetriever(RegistryRetriever): + def __init__(self, upstream_retriever: RegistryRetriever, cache_dir: Path): + super().__init__() + self.upstream = upstream_retriever + self.cache_dir = cache_dir + self.cache_dir.mkdir(parents=True, exist_ok=True) + + def download_package(self, package_name: str, version: str, target_dir: Path) -> Path: + cache_key = f"{package_name}-{version}" + cached_path = self.cache_dir / cache_key + + if cached_path.exists(): + # Copy from cache + target_path = target_dir / cache_key + shutil.copytree(cached_path, target_path) + return target_path + + # Download and cache + package_path = self.upstream.download_package(package_name, version, target_dir) + shutil.copytree(package_path, cached_path) + return package_path +``` + +### Multi-Registry Fallback + +```python +class FallbackRegistryRetriever(RegistryRetriever): + def __init__(self, retrievers: List[RegistryRetriever]): + super().__init__() + self.retrievers = retrievers + + def download_package(self, package_name: str, version: str, target_dir: Path) -> Path: + for retriever in self.retrievers: + try: + return retriever.download_package(package_name, version, target_dir) + except PackageNotFoundError: + continue + + raise PackageNotFoundError(f"Package {package_name}=={version} not found in any registry") + + def get_package_versions(self, package_name: str) -> List[str]: + all_versions = set() + for retriever in self.retrievers: + try: + versions = retriever.get_package_versions(package_name) + all_versions.update(versions) + except Exception: + continue + + return sorted(all_versions) +``` + +## Integration Points + +Here are illustrations of how to integrate your custom registry retriever. + +### With Installation Orchestrator + +The orchestrator uses your registry retriever automatically: + +```python +# Configure custom registry +private_registry = PrivateRegistryRetriever("https://internal.company.com/registry", api_key) +orchestrator = InstallationOrchestrator(registry_retriever=private_registry) + +# Install from private registry +orchestrator.install_package("internal-tool", "1.0.0") +``` + +### With Environment Manager + +```python +# Set up environment with custom registry +registry = LocalRegistryRetriever(Path("/path/to/local/packages")) +env_manager = HatchEnvironmentManager(registry_retriever=registry) +``` + +### Registry Configuration + +Store registry settings in configuration: + +```python +class ConfigurableRegistryRetriever(RegistryRetriever): + def __init__(self, config: Dict): + super().__init__() + self.config = config + self.base_url = config["base_url"] + self.timeout = config.get("timeout", 30) + + @classmethod + def from_config_file(cls, config_path: Path): + with open(config_path) as f: + config = json.load(f) + return cls(config) +``` + +## Testing Registry Integrations + +Mock external services for testing: + +```python +class TestPrivateRegistry(unittest.TestCase): + @patch('requests.get') + def test_download_package(self, mock_get): + mock_response = Mock() + mock_response.content = b"fake package data" + mock_get.return_value = mock_response + + registry = PrivateRegistryRetriever("https://example.com", "fake-key") + package_path = registry.download_package("test-pkg", "1.0.0", Path("/tmp")) + + self.assertTrue(package_path.exists()) + mock_get.assert_called_with( + "https://example.com/packages/test-pkg/1.0.0/download", + headers={"Authorization": "Bearer fake-key"} + ) +``` + +## Practical Tips + +**Error handling:** Different registries have different error responses. Wrap in consistent exceptions. + +**Authentication:** Store credentials securely, not in code. Use environment variables or credential stores. + +**Performance:** Implement caching for frequently accessed packages and metadata. + +**Fallbacks:** Use multiple registries with fallback logic for reliability. + +**Validation:** Verify downloaded packages match expected checksums when available. + +## Real Examples + +Check existing patterns: + +- `RegistryRetriever` in `hatch_registry/registry_retriever.py` - base implementation +- Tests in `Hatch-Registry/tests/` - registry testing patterns +- Configuration in `test_settings.toml` - registry configuration examples diff --git a/docs/articles/devs/index.md b/docs/articles/devs/index.md new file mode 100644 index 0000000..da46046 --- /dev/null +++ b/docs/articles/devs/index.md @@ -0,0 +1,72 @@ +# Developer Documentation + +This section provides comprehensive documentation for developers and contributors working on the Hatch package manager. + +## Documentation Organization + +The developer documentation is organized into four main categories to serve different developer needs: + +### [Architecture](./architecture/) + +High-level system understanding and design patterns for developers getting familiar with the Hatch codebase. + +- System overview and component relationships +- Design patterns and architectural decisions +- Entry points for new developers + +### [Implementation Guides](./implementation_guides/) + +Technical how-to guides for implementing specific features and extending the system. + +- Adding new installer types +- Extending core functionality +- Working with specific system components + +### [Development Processes](./development_processes/) + +Workflow, standards, and processes for effective development on the Hatch project. + +- Testing standards and procedures +- Development environment setup +- Code quality and review processes + +### [Contribution Guidelines](./contribution_guides/) + +Process-focused guidance for contributing to the Hatch project. + +- How to contribute code and documentation +- Release and dependency policies +- Community standards and expectations + +## Quick Start for Developers + +### For New Developers + +1. **Start Here** → [Developer Onboarding](./development_processes/developer_onboarding.md) - Complete getting started guide +2. **Understand the System** → [System Overview](./architecture/system_overview.md) - High-level architecture introduction +3. **Make First Contribution** → [How to Contribute](./contribution_guides/how_to_contribute.md) - Contribution workflow and process + +### For Experienced Developers + +1. **Implementing Features** → [Implementation Guides](./implementation_guides/) - Technical how-to guides +2. **Understanding Components** → [Component Architecture](./architecture/component_architecture.md) - Detailed component breakdown +3. **Testing Requirements** → [Testing Standards](./development_processes/testing_standards.md) - Testing patterns and requirements +4. **Release Process** → [Release and Dependency Policy](./contribution_guides/release_and_dependency_policy.md) - Release management policies + +## Additional Resources + +- **[Architecture Diagram](../../resources/diagrams/architecture.puml)** - Visual overview of system components + +### External Resources + +- **[Source Code](../../hatch/)** - Main Hatch package source code +- **[GitHub Repository](https://github.com/CrackingShells/Hatch)** - Project repository +- **[Hatch Schemas](https://github.com/CrackingShells/Hatch-Schemas)** - Package metadata schemas +- **[Hatch Registry](https://github.com/CrackingShells/Hatch-Registry)** - Central package registry +- **[Hatch Validator](https://github.com/CrackingShells/Hatch-Validator)** - Package validation tools + +### Organization Standards + +- **[Organization Guidelines](https://github.com/CrackingShells/.github)** - Instructions and helper markdowns for + +For detailed API information, refer to the docstrings in the source code. diff --git a/docs/articles/users/CLIReference.md b/docs/articles/users/CLIReference.md new file mode 100644 index 0000000..fe458bb --- /dev/null +++ b/docs/articles/users/CLIReference.md @@ -0,0 +1,282 @@ +# CLI Reference + +This document is a compact reference of all Hatch CLI commands and options implemented in `hatch/cli_hatch.py` presented as tables for quick lookup. + +## Table of Contents + +- [Global options](#global-options) +- [Commands](#commands) + - [hatch create](#hatch-create) + - [hatch validate](#hatch-validate) + - [hatch env](#hatch-env-environment-management) + - [hatch env create](#hatch-env-create) + - [hatch env remove](#hatch-env-remove) + - [hatch env list](#hatch-env-list) + - [hatch env use](#hatch-env-use) + - [hatch env current](#hatch-env-current) + - [hatch env python](#hatch-env-python-advanced-python-environment-subcommands) + - [hatch env python init](#hatch-env-python-init) + - [hatch env python info](#hatch-env-python-info) + - [hatch env python remove](#hatch-env-python-remove) + - [hatch env python shell](#hatch-env-python-shell) + - [hatch package](#hatch-package-package-management) + - [hatch package add](#hatch-package-add) + - [hatch package remove](#hatch-package-remove) + - [hatch package list](#hatch-package-list) + +## Global options + +These flags are accepted by the top-level parser and apply to all commands unless overridden. + +| Flag | Type | Description | Default | +|------|------|-------------|---------| +| `--envs-dir` | path | Directory to store environments | `~/.hatch/envs` | +| `--cache-ttl` | int | Cache time-to-live in seconds | `86400` (1 day) | +| `--cache-dir` | path | Directory to store cached packages | `~/.hatch/cache` | + +## Commands + +Each top-level command has its own table. Use the Syntax line before the table to see how to call it. + +### `hatch create` + +Create a new package template. + +Syntax: + +`hatch create [--dir DIR] [--description DESC]` + +| Argument / Flag | Type | Description | Default | +|---:|---|---|---| +| `name` | string (positional) | Package name (required) | n/a | +| `--dir`, `-d` | path | Target directory for the template | current directory | +| `--description`, `-D` | string | Package description | empty string | + +Examples: + +`hatch create my_package` + +`hatch create my_package --dir ./packages --description "My awesome package"` + +--- + +### `hatch validate` + +Validate a package structure and metadata. + +Syntax: + +`hatch validate ` + +| Argument | Type | Description | +|---:|---|---| +| `package_dir` | path (positional) | Path to package directory to validate (required) | + +Examples: + +`hatch validate ./my_package` + +--- + +### `hatch env` (environment management) + +Top-level syntax: `hatch env ...` + +#### `hatch env create` + +Create a new Hatch environment bootstrapping a Python/conda environment. + +Syntax: + +`hatch env create [--description DESC] [--python-version VER] [--no-python] [--no-hatch-mcp-server] [--hatch_mcp_server_tag TAG]` + +| Argument / Flag | Type | Description | Default | +|---:|---|---|---| +| `name` | string (positional) | Environment name (required) | n/a | +| `--description`, `-D` | string | Human-readable environment description | empty string | +| `--python-version` | string | Python version to create (e.g., `3.11`) | none (manager default) | +| `--no-python` | flag | Do not create a Python environment (skip conda/mamba) | false | +| `--no-hatch-mcp-server` | flag | Do not install `hatch_mcp_server` wrapper | false | +| `--hatch_mcp_server_tag` | string | Git tag/branch for wrapper install (e.g., `dev`, `v0.1.0`) | none | + +#### `hatch env remove` + +Syntax: + +`hatch env remove ` + +| Argument | Type | Description | +|---:|---|---| +| `name` | string (positional) | Environment name to remove (required) | + +#### `hatch env list` + +Syntax: + +`hatch env list` + +Description: Lists all environments. When a Python manager (conda/mamba) is available additional status and manager info are displayed. + +#### `hatch env use` + +Syntax: + +`hatch env use ` + +| Argument | Type | Description | +|---:|---|---| +| `name` | string (positional) | Environment name to set as current (required) | + +#### `hatch env current` + +Syntax: + +`hatch env current` + +Description: Print the name of the current environment. + +--- + +### `hatch env python` (advanced Python environment subcommands) + +Top-level syntax: `hatch env python ...` + +#### `hatch env python init` + +Initialize or recreate a Python environment inside a Hatch environment. + +Syntax: + +`hatch env python init [--hatch_env NAME] [--python-version VER] [--force] [--no-hatch-mcp-server] [--hatch_mcp_server_tag TAG]` + +| Flag | Type | Description | Default | +|---:|---|---|---| +| `--hatch_env` | string | Hatch environment name (defaults to current env) | current environment | +| `--python-version` | string | Desired Python version (e.g., `3.12`) | none | +| `--force` | flag | Force recreation if it already exists | false | +| `--no-hatch-mcp-server` | flag | Skip installing `hatch_mcp_server` wrapper | false | +| `--hatch_mcp_server_tag` | string | Git tag/branch for wrapper installation | none | + +#### `hatch env python info` + +Show information about the Python environment for a Hatch environment. + +Syntax: + +`hatch env python info [--hatch_env NAME] [--detailed]` + +| Flag | Type | Description | Default | +|---:|---|---|---| +| `--hatch_env` | string | Hatch environment name (defaults to current) | current environment | +| `--detailed` | flag | Show additional diagnostics and package listing | false | + +When available this command prints: status, python executable, python version, conda env name, environment path, creation time, package count and package list. With `--detailed` it also prints diagnostics from the manager. + +#### `hatch env python add-hatch-mcp` + +Install the `hatch_mcp_server` wrapper into the Python environment of a Hatch env. + +Syntax: + +`hatch env python add-hatch-mcp [--hatch_env NAME] [--tag TAG]` + +| Flag | Type | Description | Default | +|---:|---|---|---| +| `--hatch_env` | string | Hatch environment name (defaults to current) | current environment | +| `--tag` | string | Git tag/branch for wrapper install | none | + +#### `hatch env python remove` + +Remove the Python environment associated with a Hatch environment. + +Syntax: + +`hatch env python remove [--hatch_env NAME] [--force]` + +| Flag | Type | Description | Default | +|---:|---|---|---| +| `--hatch_env` | string | Hatch environment name (defaults to current) | current environment | +| `--force` | flag | Skip confirmation prompt and force removal | false | + +#### `hatch env python shell` + +Launch a Python REPL or run a single command inside the Python environment. + +Syntax: + +`hatch env python shell [--hatch_env NAME] [--cmd CMD]` + +| Flag | Type | Description | Default | +|---:|---|---|---| +| `--hatch_env` | string | Hatch environment name (defaults to current) | current environment | +| `--cmd` | string | Command to execute inside the Python shell (optional) | none | + +--- + +### `hatch package` (package management) + +Top-level syntax: `hatch package ...` + +#### `hatch package add` + +Add a package (local path or registry name) into an environment. + +Syntax: + +`hatch package add [--env NAME] [--version VER] [--force-download] [--refresh-registry] [--auto-approve]` + +| Argument / Flag | Type | Description | Default | +|---:|---|---|---| +| `package_path_or_name` | string (positional) | Path to package directory or registry package name (required) | n/a | +| `--env`, `-e` | string | Target Hatch environment name (defaults to current) | current environment | +| `--version`, `-v` | string | Version for registry packages | none | +| `--force-download`, `-f` | flag | Force fetching even if cached | false | +| `--refresh-registry`, `-r` | flag | Refresh registry metadata before resolving | false | +| `--auto-approve` | flag | Automatically approve dependency installation prompts | false | + +Examples: + +`hatch package add ./my_package` + +`hatch package add registry_package --version 1.0.0 --env dev-env --auto-approve` + +#### `hatch package remove` + +Remove a package from a Hatch environment. + +Syntax: + +`hatch package remove [--env NAME]` + +| Argument / Flag | Type | Description | Default | +|---:|---|---|---| +| `package_name` | string (positional) | Name of the package to remove (required) | n/a | +| `--env`, `-e` | string | Hatch environment name (defaults to current) | current environment | + +#### `hatch package list` + +List packages installed in a Hatch environment. + +Syntax: + +`hatch package list [--env NAME]` + +| Flag | Type | Description | Default | +|---:|---|---|---| +| `--env`, `-e` | string | Hatch environment name (defaults to current) | current environment | + +Output: each package row includes name, version, hatch compliance flag, source URI and installation location. + +--- + +## Exit codes + +| Code | Meaning | +|---:|---| +| `0` | Success | +| `1` | Error or failure | + +## Notes + +- The implementation in `hatch/cli_hatch.py` does not provide a `--version` flag or a top-level `version` command. Use `hatch --help` to inspect available commands and options. +- This reference mirrors the command names and option names implemented in `hatch/cli_hatch.py`. If you change CLI arguments in code, update this file to keep documentation in sync. diff --git a/docs/articles/users/GettingStarted.md b/docs/articles/users/GettingStarted.md new file mode 100644 index 0000000..15fca2f --- /dev/null +++ b/docs/articles/users/GettingStarted.md @@ -0,0 +1,283 @@ +# Getting Started with Hatch + +Welcome to Hatch! This guide will help you understand what Hatch is, when to use it, and how to get started quickly. + +This article is about: + +- Understanding Hatch's purpose and capabilities +- System requirements and installation prerequisites +- Quick setup for immediate productivity +- Next steps for learning and exploration + +## What is Hatch? + +Hatch is a sophisticated package manager specifically designed for **Model Context Protocol (MCP) servers**. It solves the complexity of installing, managing, and deploying MCP servers by handling their diverse dependencies automatically. + +### Key Capabilities + +**Environment Isolation**: Create separate workspaces for different projects, similar to Python virtual environments but broader in scope. + +**Multi-Type Dependencies**: Handle system packages (apt), Python packages (pip), other Hatch packages, and Docker containers in a single, coordinated installation. + +**Intelligent Dependency Resolution**: Build dependency graphs, detect conflicts, and present clear installation plans before making changes. + +**Development-Focused Design**: Optimized for rapid development, testing, and demonstration of MCP server ecosystems. + +### When to Use Hatch + +**Ideal for**: + +- Developing and testing MCP servers +- Setting up demonstration environments +- Managing collections of related MCP servers +- Rapid prototyping with complex dependencies +- Educational scenarios and tutorials + +**Current limitations make it less suitable for**: + +- Production deployments (security enhancements needed) +- Multi-tenant systems (isolation improvements needed) +- High-security environments (integrity verification needed) + +See [Security and Trust](SecurityAndTrust.md) for detailed guidance on deployment scenarios. + +## System Requirements + +### Required Components + +**Python 3.12 or higher**: Hatch requires modern Python for reliable operation. + +**Git**: For cloning repositories and accessing package sources. + +**Network access**: For package registry access and validation services. + +### Recommended Components + +**Conda or Mamba**: For Python environment isolation within Hatch environments. Hatch will automatically detect and use these tools if available. + +**Docker**: For packages that include Docker-based dependencies. + +**Platform package manager**: + +- Linux: `apt` (Ubuntu/Debian) +- macOS: None supported (planed) +- Windows: None supported (planed) + +### Detection Behavior + +**Python Environment Detection**: Hatch looks for conda/mamba in standard locations: + +- Linux/macOS: `~/miniconda3/bin/conda`, `/opt/conda/bin/conda` +- Windows: `%USERPROFILE%\miniconda3\Scripts\conda.exe` + +If you have custom installations, Python environment features may be unavailable unless tools are in your PATH. + +## Quick Installation + +### Step 1: Install Prerequisites + +Ensure you have Python 3.12+ and Git installed: + +```bash +python --version # Should show 3.12 or higher +git --version # Should show Git version +``` + +For Python environment support, install conda or mamba: + +```bash +# Install Miniforge (includes conda and mamba) +# Download from: https://conda-forge.org/download/ +``` + +### Step 2: Install Hatch + +Clone and install Hatch from source: + +```bash +git clone https://github.com/CrackingShells/Hatch.git +cd Hatch +pip install -e . +``` + +### Step 3: Verify Installation + +Test that Hatch is working: + +```bash +hatch --help +``` + +You should see available commands. + +## First Steps + +### Explore existing commands + +You can suffix `--help` to any command to get more information about it. For example: + +```bash +hatch env --help +hatch package --help +``` + +### Create Your First Environment + +```bash +# Create a new environment for your project +# By default this will also initialize a Python environment with the latest Python version +hatch env create my-project + +# Switch to the new environment +hatch env use my-project + +# Verify you're in the correct environment +hatch env current + +``` + +### Add Your First Package + +```bash +# Add a package (from local directory or registry) +hatch package add example-mcp-server + +# List installed packages +hatch package list +``` + +For more in-depth information, please refer to the [tutorials](tutorials/01-getting-started/01-installation.md) section. + +## Understanding Hatch Concepts + +### Environments + +**Hatch Environments** are isolated workspaces that can contain: + +- MCP servers and their configurations +- Python environments (via conda/mamba) +- Package installations and dependencies +- Environment-specific settings + +Think of them as project-specific containers that keep different work separated. + +### Dependencies + +Hatch handles four types of dependencies: + +1. **System Dependencies**: OS packages like `curl`, `git`, installed via package managers +2. **Python Dependencies**: Python packages installed via pip +3. **Hatch Dependencies**: Other MCP servers and Hatch packages +4. **Docker Dependencies**: Container images for service dependencies + +### Installation Process + +When you install a package, Hatch: + +1. **Analyzes** the package metadata and builds a dependency graph +2. **Presents** an installation plan showing what will be installed +3. **Requests approval** unless you use `--auto-approve` +4. **Installs** dependencies in order: System → Python → Hatch → Docker +5. **Updates** environment state with the results + +## Common Workflows + +### Development Workflow + +```bash +# Set up project environment +hatch env create my-mcp-project +hatch env use my-mcp-project + +# Add development tools +hatch package add development-tools + +# Work on your project... + +# Test with different configurations +hatch env create test-env +hatch env use test-env +hatch package add ./my-project # Add local package for testing +``` + +### Package Authoring Workflow + +```bash +# Create package template +hatch create my-new-server + +# Edit package metadata +cd my-new-server +# Edit hatch_metadata.json + +# Validate package structure +hatch validate . + +# Test local installation +hatch package add . +``` + +### Automation Workflow + +```bash +# Validate before adding +hatch validate . + +# Add package (for automation, ensure non-interactive environment) +hatch package add . +``` + +## Learning Path + +Now that you have Hatch installed and understand the basics: + +### Next Steps + +1. **Complete the tutorial series**: + - [Environment Management](tutorials/02-environments/) - Advanced environment operations + - [Package Authoring](tutorials/03-author-package/) - Create your own packages + - [CI Automation](tutorials/04-ci-automation/) - Set up automated workflows + +2. **Explore reference documentation**: + - [CLI Reference](CLIReference.md) - Complete command documentation + - [Troubleshooting](Troubleshooting/CommonIssues.md) - Solutions to common issues + +3. **Understand limitations**: + - [Limits and Known Issues](../appendices/LimitsAndKnownIssues.md) - Current constraints and workarounds + - [Security and Trust](SecurityAndTrust.md) - Security model and deployment guidance + +### Getting Help + +**Documentation**: Start with the [troubleshooting guide](Troubleshooting/CommonIssues.md) for common issues. + +**Community**: Check the GitHub repository for issues and discussions. + +**Diagnostics**: Use these commands to gather information when seeking help: +```bash +hatch --version +hatch env list --verbose +hatch package list +python --version +which conda || echo "conda not found" +``` + +## Important Notes + +### Current Status + +Hatch v0.4.2 is optimized for **development and trusted network environments**. Key considerations: + +- **Package integrity**: Downloads are not cryptographically verified +- **Concurrent usage**: Multiple Hatch instances may conflict +- **System packages**: Version constraints are simplified for system dependencies +- **Network dependencies**: Registry access and validation require internet connectivity + +### Best Practices + +- **Use one Hatch instance per workspace** to avoid concurrent access issues +- **Test complex installations in isolated environments** before production use +- **Use `--dry-run` to preview installations** before making changes +- **Keep cache sizes manageable** by periodically cleaning old data +- **Use exact version pins for critical dependencies** where precision matters + +Welcome to Hatch! The tool is designed to make MCP server management straightforward while being transparent about its current capabilities and limitations. Start with simple scenarios and gradually explore more complex features as you become comfortable with the system. diff --git a/docs/articles/users/SecurityAndTrust.md b/docs/articles/users/SecurityAndTrust.md new file mode 100644 index 0000000..dd50868 --- /dev/null +++ b/docs/articles/users/SecurityAndTrust.md @@ -0,0 +1,195 @@ +# Security and Trust + +This article covers security considerations when using Hatch for package management and MCP server deployment. + +## Package Trust Model + +### Package Sources + +Hatch packages can come from different sources, each with different trust implications: + +- **Local packages** - Packages installed from local filesystem paths +- **Registry packages** - Packages downloaded from the Hatch registry + +### Metadata Fields Affecting Trust + +Several fields in `hatch_metadata.json` provide trust-related information: + +- **`repository`** - URL to the package's source code repository for verification +- **`license`** - License information with optional URI for license text +- **`author`** and **`contributors`** - Contact information for package maintainers +- **`citations`** - Attribution information for package origins and MCP compliance + +Verify these fields align with your security requirements before installation. + +## Installation Security + +### Privilege Requirements + +Different installer types have varying privilege implications: + +#### Python Installer (`python_installer.py`) + +- Installs Python packages via pip within conda/mamba environments +- Generally isolated to the specific Python environment +- May require network access for package downloads + +#### System Installer (`system_installer.py`) + +- Installs system packages using package managers like apt +- **Requires elevated privileges** for system-wide installations +- Can modify system state outside of Hatch environments + +#### Docker Installer (`docker_installer.py`) + +- Manages Docker image dependencies +- Requires Docker daemon access +- Images run with Docker's security model + +#### Hatch Installer (`hatch_installer.py`) + +- Handles other Hatch package dependencies +- Operates within Hatch's environment model + +### Installation Context + +Installer and environment components manage security context for installations and define common installer interfaces so different installer types behave consistently. Internal implementation filenames are omitted here; maintainers should keep operational details in internal docs where appropriate. + + +### Registry Security + +### Registry Retrieval + +Registry retrieval and exploration components handle interactions with the registry and mirrors. Key behaviors include: + +- **Caching** - Registry data is cached locally and the cache lifetime is configurable by administrators +- **Network fallback** - Multiple retrieval strategies and mirrors may be used for reliability +- **Error handling** - Graceful degradation when registry is unavailable + +### Cache Security + +Hatch stores registry caches in a configurable cache directory. Cache lifetime (TTL) is also configurable. Administrators in high-security environments should reduce cache lifetime, tighten cache permissions, and regularly rotate or clean caches. + +Recommended cache hardening: + +- Use a dedicated, user-owned cache directory with strict file permissions (owner-only where possible) +- Configure a short cache TTL in environments where registry content changes frequently or when supply-chain risk is a concern +- Consider running cache-clean tasks in CI or system maintenance schedules + + +### Environment Isolation + +### Python Environment Isolation + +When using Python environments via conda/mamba: + +- Each environment is isolated with its own Python installation +- Package installations are contained within the environment +- Environment paths are managed by the project's environment manager components + +### Hatch Environment Isolation + +Hatch environments provide: + +- Separate package namespaces +- Independent dependency resolution +- Isolated configuration and metadata + + +### Dependency Resolution Security + +### Dependency Chain Validation + +An installation orchestrator coordinates installation of package dependencies. Be aware that: + +- **Transitive dependencies** are automatically resolved and installed +- **Version constraints** may be satisfied by different package versions +- **Mixed dependency types** (Python, system, Docker, Hatch) may have different security profiles + +### Dependency Verification + +Always review dependency specifications in `hatch_metadata.json`: + +```json +{ + "dependencies": { + "python": [ + { + "name": "requests", + "version_constraint": ">=2.28.0", + "package_manager": "pip" + } + ], + "system": [ + { + "name": "curl", + "version_constraint": ">=7.0.0", + "package_manager": "apt" + } + ] + } +} +``` + +## Best Practices + +### Before Installation + +1. **Validate packages** using `hatch validate` before installation +2. **Review metadata** including dependencies, repository, and license information +3. **Verify package sources** and author information +4. **Check version constraints** to ensure expected dependency versions + +### During Installation + +1. **Use `--auto-approve` carefully** - only in trusted environments +2. **Monitor privilege escalation** for system package installations +3. **Review dependency resolution** output for unexpected packages + +### After Installation + +1. **Verify installed packages** using `hatch package list` +2. **Test package functionality** in isolated environments first +3. **Monitor environment health** using `hatch env python info --detailed` + +### Environment Management + +1. **Use separate environments** for different trust levels +2. **Regularly update** Python environments and dependencies +3. **Clean up unused environments** to reduce attack surface +4. **Use specific version constraints** rather than broad ranges when security is critical + +## Troubleshooting Security Issues + +### Registry Access Issues + +If registry access fails, check: + +- Network connectivity and firewall settings +- Registry cache status and TTL configuration +- Error messages from `registry_retriever.py` components + +### Permission Issues + +For installation failures: + +- Verify user permissions for target directories +- Check conda/mamba environment access +- Review Docker daemon permissions for Docker dependencies + +### Environment Corruption + +If environments become corrupted: + +- Use `hatch env python info --detailed` for diagnostics +- Consider removing and recreating environments +- Verify Python Environment Manager status + +## Reporting Security Issues + +When reporting security concerns: + +1. Include relevant CLI commands and error messages +2. Specify package sources and metadata details +3. Provide environment information from diagnostic commands +4. Follow responsible disclosure practices for vulnerabilities diff --git a/docs/articles/users/Troubleshooting/ReportIssues.md b/docs/articles/users/Troubleshooting/ReportIssues.md new file mode 100644 index 0000000..76edbe5 --- /dev/null +++ b/docs/articles/users/Troubleshooting/ReportIssues.md @@ -0,0 +1,132 @@ +# Common Issues + +This page is a concise troubleshooting landing page focused on collecting reproducible diagnostics for bug reports. + +This article is about: + +- what to include when reporting a problem +- copy-pasteable diagnostic commands (Linux/macOS and PowerShell variants) +- a short issue-report template you can paste into GitHub issues + +Keep this page small: concrete, reproducible examples and deep troubleshooting steps belong in separate, focused articles when we have real-world occurrences to document. + +## How to collect diagnostics (copy/paste) + +When filing an issue, include the exact commands you ran and the full output of the following commands. Use the PowerShell variants on Windows. + +Hatch and environment summary + +```bash +# POSIX (Linux/macOS) & PowerShell (Windows) +hatch env list +hatch env current +hatch package list +pip show hatch +``` + +Detailed Python environment info (replace `` with your Hatch environment name) + +```bash +hatch env python info --hatch_env --detailed +``` + +Registry & cache checks + +```bash +# Check registry connectivity by attempting a registry refresh (this will attempt to resolve the package) +hatch package add --refresh-registry + +# POSIX cache listing +ls -la ~/.hatch/cache/packages || true +``` + +```powershell +# PowerShell cache listing +hatch package add --refresh-registry +If (Test-Path "$env:USERPROFILE\.hatch\cache\packages") { Get-ChildItem -Path $env:USERPROFILE\.hatch\cache } else { Write-Output "No hatch cache directory" } +``` + +Capture failing command output (redirect and attach the log when filing an issue) + +```bash +# POSIX +hatch package add 2>&1 | tee hatch-package-add.log +``` + +```powershell +# PowerShell +hatch package add 2>&1 | Tee-Object -FilePath hatch-package-add.log +``` + +Validation / metadata checks + +```bash +# POSIX +python -m json.tool package_dir/hatch_metadata.json +``` + +```powershell +# PowerShell: pretty-print JSON +Get-Content package_dir\hatch_metadata.json | ConvertFrom-Json | ConvertTo-Json -Depth 10 +``` + +Useful system checks + +```bash +# POSIX +which conda || which mamba || true +df -h || true +``` + +```powershell +# PowerShell +where.exe conda,mamba +Get-PSDrive -PSProvider FileSystem | Select-Object Name, Free, Used, @{n='Size';e={$_.Free + $_.Used}} +``` + +Include relevant log files and timestamps when possible. Attach large logs rather than pasting them into issue body. + +## Issue report template + +Paste the following to [report an issue](https://github.com/CrackingShells/Hatch/issues): + +```markdown +### Environment: + +- OS: Windows 10/11 / macOS / Linux (include distro and version) +- Hatch version: + +### Command: + +- Exact command run: `hatch ...` + +### Symptom: + +- Paste exact error text and full output (attach log file if large). + +### Reproduction steps: + +1. ... +2. ... + +### Diagnostics run: + +- Paste outputs of commands from "How to collect diagnostics" + +### What I expect: + +- Short description of expected behavior + +### What happened: + +- Short description of actual behavior + +### Workarounds tried, if any: + +- list + +### Relevant files (attach): + +- `package_dir/hatch_metadata.json` +- `hatch-package-add.log` (if created) +``` diff --git a/docs/articles/users/tutorials/01-getting-started/01-installation.md b/docs/articles/users/tutorials/01-getting-started/01-installation.md new file mode 100644 index 0000000..b6d091b --- /dev/null +++ b/docs/articles/users/tutorials/01-getting-started/01-installation.md @@ -0,0 +1,84 @@ +# 01: Installation + +--- +**Concepts covered:** + +- How Hatch is distributed and installed +- CLI structure and available commands +- Configuration directories and cache management + +**Skills you will practice:** + +- Installing Hatch using pip +- Verifying the installation +- Exploring CLI help and available commands + +--- + +This article covers the installation of Hatch, a package manager for Model Context Protocol (MCP) servers. + +## Step 1: Install Hatch + +1. Ensure you have Python 3.12 or later installed on your system. +2. Install Hatch using pip: + + ```bash + git clone https://github.com/CrackingShells/Hatch.git + cd Hatch + pip install -e . + ``` + +3. Verify the installation by checking the available commands: + + ```bash + hatch --help + ``` + +## Step 2: Understand the CLI Structure + +Hatch provides several command groups: + +- `hatch create` - Create new package templates +- `hatch validate` - Validate package structures +- `hatch env` - Environment management commands +- `hatch package` - Package management commands + +### Global Configuration Options + +The CLI accepts the following global configuration options: + +- `--envs-dir` - Directory to store environments (default: `~/.hatch/envs`) +- `--cache-ttl` - Cache TTL in seconds (default: 86400 seconds = 1 day) +- `--cache-dir` - Directory to store cached packages (default: `~/.hatch/cache`) + +## Step 3: Explore Available Commands + +View detailed help for specific command groups: + +```bash +# Environment management +hatch env --help + +# Package management +hatch package --help +``` + +**Exercise:** +Explore the help output for the `create` command. What options are available for each? + +
+Solution + +```bash +# positional arguments: +# name Package name +# +# options: +# -h, --help show this help message and exit +# --dir DIR, -d DIR Target directory (default: current directory) +# --description DESCRIPTION, -D DESCRIPTION +# Package description +``` +
+ +> Next: [Create Environment](02-create-env.md) diff --git a/docs/articles/users/tutorials/01-getting-started/02-create-env.md b/docs/articles/users/tutorials/01-getting-started/02-create-env.md new file mode 100644 index 0000000..fc9243d --- /dev/null +++ b/docs/articles/users/tutorials/01-getting-started/02-create-env.md @@ -0,0 +1,102 @@ +# 02: Create Environment + +--- +**Concepts covered:** + +- What Hatch environments are and how they work +- Python environment integration via conda/mamba +- Environment metadata and configuration storage + +**Skills you will practice:** + +- Creating a new Hatch environment +- Creating environments with Python integration +- Managing environment options and configurations + +--- + +This article covers creating your first Hatch environment for managing MCP server packages. + +## Step 1: Create a Basic Environment + +Create a simple Hatch environment without Python integration: + +```bash +hatch env create my_first_env --description "My first Hatch environment" --no-python +``` + +This creates an environment that can manage Hatch packages but doesn't include a Python environment. + +> [!NOTE] +> Environment names must be alphanumeric and can contain underscores. + +## Step 2: Create an Environment with Python Integration + +For most use cases, you'll want Python integration for running MCP servers: + +```bash +hatch env create my_python_env --description "Environment with Python support" --python-version 3.11 +``` + +This command: + +- Creates a Hatch environment named `my_python_env` +- Sets up a Python 3.11 environment using conda/mamba +- Installs the hatch_mcp_server wrapper by default + +### Understanding the Options + +- `--python-version` - Specifies the Python version (e.g., 3.11, 3.12) +- `--no-python` - Skips Python environment creation +- `--no-hatch-mcp-server` - Skips installation of the hatch_mcp_server wrapper +- `--hatch_mcp_server_tag` - Specifies a git tag/branch for the wrapper (e.g., 'dev', 'v0.1.0') + +## Step 3: Verify Environment Creation + +List your environments to confirm they were created: + +```bash +hatch env list +``` + +You should see output similar to: + +```txt +Available environments: + my_first_env - My first Hatch environment + Python: Not configured + my_python_env - Environment with Python support + Python: 3.11.x (conda: my_python_env) + +Python Environment Manager: + Conda executable: /path/to/conda + Mamba executable: /path/to/mamba + Preferred manager: mamba +``` + +**Exercise:** +Initialize a Python environment inside `my_first_env`. Try both initializing without `hatch_mcp_server` wrapper and adding it afterwards. Hint: Use `hatch env python --help` to explore available Python subcommands and flags. + +
+Solution + +```bash + +# Initialize a Python environment inside my_first_env +hatch env python init --hatch_env my_first_env --no-hatch-mcp-server + +# Verify Python environment +hatch env python info --hatch_env my_first_env # hatch_mcp_server should not appear in the list of packages + +# Add hatch_mcp_server wrapper +hatch env python add-hatch-mcp --hatch_env my_first_env + +# Verify again +hatch env python info --hatch_env my_first_env # hatch_mcp_server should now appear in the list of packages +``` +
+ +In most use cases, you'll want to create environments with Python integration and the hatch_mcp_server wrapper. However, Hatch provides flexibility to customize your environments as needed. + +> Previous: [Installation](01-installation.md) +> Next: [Install Package](03-install-package.md) diff --git a/docs/articles/users/tutorials/01-getting-started/03-install-package.md b/docs/articles/users/tutorials/01-getting-started/03-install-package.md new file mode 100644 index 0000000..24590bf --- /dev/null +++ b/docs/articles/users/tutorials/01-getting-started/03-install-package.md @@ -0,0 +1,140 @@ +# 03: Install Package + +--- +**Concepts covered:** + +- Package installation from local directories and registry +- Dependency resolution and installation orchestration +- Registry caching and package sources + +**Skills you will practice:** + +- Installing packages from local paths +- Installing packages from the registry +- Managing installation options and dependencies + +--- + +This article covers installing MCP server packages into your Hatch environment. + +## Step 1: Set Your Current Environment + +Before installing packages, set the environment you want to work with: + +```bash +hatch env use my_python_env +``` + +Verify the current environment: + +```bash +hatch env current +``` + +## Step 2: Install from Registry + +Install a package from the Hatch registry by name: + +```bash +hatch package add base_pkg_1 +``` + +This will prompt you to confirm the installation plan. Type `y` to proceed: + +```txt +============================================================ +DEPENDENCY INSTALLATION PLAN +============================================================ +Main Package: base_pkg_1 v1.0.3 +Package Type: remote + +No additional dependencies to install. + +Total packages to install: 1 +============================================================ + +Proceed with installation? [y/N]: +``` + +For automated scenarios, use `--auto-approve` to skip confirmation prompts: + +```bash +hatch package add my-package --auto-approve +``` + +### Installation Options + +- `--env` / `-e` - Specify target environment (defaults to current) +- `--version` / `-v` - Specify package version (for registry packages) +- `--force-download` / `-f` - Force download even if package is cached +- `--refresh-registry` / `-r` - Force refresh of registry data +- `--auto-approve` - Automatically approve dependency installations + +## Step 3: Install a Local Package + +Hatch!'s package registry is almost empty as of now (until you contribute your own packages!). Hence, you will likely use Hatch! to help you install your own packages at first. You can do that by specifying the path to your local package directory: + +```bash +hatch package add /path/to/my-package +``` + +If you don't have a local package yet, you can create one using the `hatch create` command. This will be covered in the [Author Package](../03-author-package/01-create-package.md) tutorial. For now though, let us proceed with the next step. + +## Step 4: Verify Installation + +List installed packages in your environment: + +```bash +hatch package list --env my_python_env +``` + +Output shows package details: + +```txt +Packages in environment 'my_python_env': +my-package (1.0.0) Hatch compliant: true source: file:///path/to/package location: /env/path/my-package +``` + +## Step 5: Understanding Package Dependencies + +When you install a package, Hatch automatically resolves and installs dependencies defined in the package's `hatch_metadata.json`. Hatch supports four types of dependencies: + +- **Hatch dependencies** - Other Hatch packages required +- **Python dependencies** - Python packages installed via pip +- **System dependencies** - System packages installed via package managers like apt +- **Docker dependencies** - Docker images required for the package + +As you could see in the earlier example, `base_pkg_1` has no dependencies and the installation plan reflected that: + +```txt +No additional dependencies to install. +``` + +However, any existing dependencies will be clearly stated in the installation plan. **Always review the installation plan before approving it.** + +The dependency installation orchestrator handles the installation flow and manages different installer types. Dependencies are always installed following the order ``system -> python -> hatch -> docker``. + +**Exercise:** +Try re-installing `base_pkg_1` with `--force-download` and `--refresh-registry` flags. Then try installing the same package again without these flags to see the caching behavior. + +
+Solution + +```bash +# First installation with force options +hatch package add base_pkg_1 --force-download --refresh-registry + +# Second installation (should use cache) +hatch package add base_pkg_1 +``` + +The second installation should be faster due to caching. In particular, you should see the statement: + +```txt +YYYY-MM-DD HH:MM:SS - hatch.package_loader - INFO - Using cached package base_pkg_1 v1.0.3 +``` + +
+ +> Previous: [Create Environment](02-create-env.md) +> Next: [Checkpoint](04-checkpoint.md) diff --git a/docs/articles/users/tutorials/01-getting-started/04-checkpoint.md b/docs/articles/users/tutorials/01-getting-started/04-checkpoint.md new file mode 100644 index 0000000..1ac3e73 --- /dev/null +++ b/docs/articles/users/tutorials/01-getting-started/04-checkpoint.md @@ -0,0 +1,14 @@ +# Checkpoint: Getting Started + +**What you've accomplished:** + +- Installed Hatch and explored the CLI structure +- Created environments with and without Python integration +- Installed packages from local paths and understood dependency management + +**Next up:** + +- Learn advanced environment management techniques +- Explore Python environment management in detail + +Ready to dive deeper into environment management? Let's continue with [Environment Management](../02-environments/01-manage-envs.md)! diff --git a/docs/articles/users/tutorials/02-environments/01-manage-envs.md b/docs/articles/users/tutorials/02-environments/01-manage-envs.md new file mode 100644 index 0000000..184b8b1 --- /dev/null +++ b/docs/articles/users/tutorials/02-environments/01-manage-envs.md @@ -0,0 +1,144 @@ +# 01: Manage Environments + +--- +**Concepts covered:** + +- Environment lifecycle management +- Environment switching and current environment tracking +- Environment metadata and descriptions + +**Skills you will practice:** + +- Listing and inspecting environments +- Switching between environments +- Removing environments when no longer needed + +--- + +## Understanding Hatch Environments + +Hatch environments provide isolated workspaces for different MCP server collections. Each environment can have: + +- Its own package installations +- Optional Python environment (via conda/mamba) +- Independent configuration and state + +**Limitation Note**: Multiple `hatch` instances modifying environments simultaneously may cause conflicts. Use one instance per workspace for reliability. + +## Step 1: List All Environments + +View all available environments with their details: + +```bash +hatch env list +``` + +Example output: + +```txt +Available environments: +* my_python_env - Environment with Python support + Python: 3.11.9 (conda: my_python_env) + my_first_env - My first Hatch environment + Python: 3.13.5 (conda: my_first_env) + +Python Environment Manager: + Conda executable: /usr/local/bin/conda + Mamba executable: /usr/local/bin/mamba + Preferred manager: mamba +``` + +The `*` indicates the current active environment. + +## Step 2: Switch Between Environments + +Change your current working environment: + +```bash +hatch env use my_first_env +``` + +Verify the switch: + +```bash +hatch env current +``` + +This command shows: + +``` +Current environment: my_first_env +``` + +## Step 3: Remove Environments + +Remove an environment you no longer need: + +```bash +hatch env remove my_first_env +``` + +**Important:** This removes both the Hatch environment and any associated Python environment. Make sure to back up any important data first. + +## Step 4: Understanding Environment Information + +The `env list` command provides detailed information: + +- **Environment name and description** - Basic identification +- **Current environment marker (*)** - Shows which environment is active +- **Python environment status** - Shows Python version and conda environment name +- **Python Environment Manager status** - Shows available conda/mamba executables + +If conda/mamba is not available, you'll see: + +``` +Python Environment Manager: Conda/mamba not available +``` + +## Step 5: Managing Multiple Environments + +You can maintain multiple environments for different projects: + +```bash +# Project-specific environments +hatch env create project-a --description "Environment for Project A" --python-version 3.11 +hatch env create project-b --description "Environment for Project B" --python-version 3.12 + +# Switch between them as needed +hatch env use project-a +# Work on project A... + +hatch env use project-b +# Work on project B... +``` + +**Exercise:** +Create three environments with different Python versions, switch between them, and observe how the current environment changes. Then remove one of the environments. + +
+Solution + +```bash +# Create environments +hatch env create env-311 --python-version 3.11 --description "Python 3.11 environment" +hatch env create env-312 --python-version 3.12 --description "Python 3.12 environment" +hatch env create env-313 --python-version 3.13 --description "Python 3.13 environment" + +# Switch between them +hatch env use env-311 +hatch env current # Should show env-311 + +hatch env use env-312 +hatch env current # Should show env-312 + +# List to see all three +hatch env list + +# Remove one +hatch env remove env-313 +``` + +
+ +> Previous: [Getting Started Checkpoint](../01-getting-started/04-checkpoint.md) +> Next: [Python Environment Management](02-python-env.md) diff --git a/docs/articles/users/tutorials/02-environments/02-python-env.md b/docs/articles/users/tutorials/02-environments/02-python-env.md new file mode 100644 index 0000000..a260baa --- /dev/null +++ b/docs/articles/users/tutorials/02-environments/02-python-env.md @@ -0,0 +1,166 @@ +# 02: Python Environment Management + +--- +**Concepts covered:** + +- Advanced Python environment operations +- Python environment initialization and configuration +- Environment diagnostics and troubleshooting +- Hatch MCP server wrapper management + +**Skills you will practice:** + +- Initializing Python environments in existing Hatch environments +- Viewing detailed Python environment information +- Managing the hatch_mcp_server wrapper +- Launching Python shells within environments + +--- + +This article covers advanced Python environment management within Hatch environments. + +## Step 1: Initialize Python Environment + +Add a Python environment to an existing Hatch environment: + +```bash +# Initialize in current environment +hatch env python init --python-version 3.11 + +# Initialize in specific environment +hatch env python init --hatch_env my_env --python-version 3.12 --force +``` + +### Python Initialization Options + +- `--hatch_env` - Target Hatch environment (defaults to current) +- `--python-version` - Python version (e.g., 3.11, 3.12) +- `--force` - Force recreation if Python environment exists +- `--no-hatch-mcp-server` - Skip hatch_mcp_server wrapper installation +- `--hatch_mcp_server_tag` - Git tag/branch for wrapper (e.g., 'dev', 'v0.1.0') + +## Step 2: View Python Environment Information + +Get detailed information about a Python environment: + +```bash +# Basic info for current environment +hatch env python info + +# Detailed info for specific environment +hatch env python info --hatch_env my_env --detailed +``` + +Example output: +``` +Python environment info for 'my_env': + Status: Active + Python executable: /path/to/conda/envs/my_env/bin/python + Python version: 3.11.9 + Conda environment: my_env + Environment path: /path/to/conda/envs/my_env + Created: 2025-08-24T10:30:00 + Package count: 25 + Packages: + - fastmcp (1.0.0) + - mcp (1.0.0) + - hatch-mcp-server (0.1.0) + +Diagnostics: + conda_available: true + mamba_available: true + environment_exists: true + python_executable_valid: true +``` + +## Step 3: Manage Hatch MCP Server Wrapper + +The hatch_mcp_server wrapper provides integration between Hatch packages and MCP servers. + +Add the wrapper to an environment: +```bash +hatch env python add-hatch-mcp --hatch_env my_env --tag v0.1.0 +``` + +The wrapper is typically installed automatically when creating Python environments, but you can add it manually or update to specific versions. + +## Step 4: Launch Python Shell + +Access the Python environment directly: + +```bash +# Interactive shell +hatch env python shell --hatch_env my_env + +# Run specific command +hatch env python shell --hatch_env my_env --cmd "import sys; print(sys.version)" +``` + +## Step 5: Remove Python Environment + +Remove only the Python environment while keeping the Hatch environment: + +```bash +# With confirmation prompt +hatch env python remove --hatch_env my_env + +# Force removal without prompt +hatch env python remove --hatch_env my_env --force +``` + +This allows you to remove Python dependencies without affecting Hatch package installations. + +## Step 6: Troubleshooting Python Environments + +Use the `--detailed` flag to diagnose issues: + +```bash +hatch env python info --detailed +``` + +Common diagnostic information includes: + +- **conda_available** - Whether conda is installed and accessible +- **manager_executable** - The path to conda/mamba executable that will be used +- **environment_path** - The path to the conda environment +- **python_executable** - The path to the Python executable used to run the MCP servers in the Hatch! packages. + +**Exercise:** + +- Create a Hatch environment without Python, then add Python support afterwards. View the detailed information before and after Python initialization. +- Remove the Python environment and verify it's cleanly removed while keeping the Hatch environment intact. +- Remove the Hatch environment and verify it's cleanly removed. + +
+Solution + +```bash +# Create environment without Python +hatch env create test_python --no-python --description "Test Python addition" + +# Check info (should show no Python) +hatch env python info --hatch_env test_python --detailed + +# Add Python environment +hatch env python init --hatch_env test_python --python-version 3.11 + +# Check info again (should show Python details) +hatch env python info --hatch_env test_python --detailed + +# Remove Python environment +hatch env python remove --hatch_env test_python --force + +# Check info (should show no Python again) +hatch env python info --hatch_env test_python --detailed + +# Remove Hatch environment +hatch env remove test_python + +# Verify removal +hatch env list +``` + +
+ +> Previous: [Manage Environments](01-manage-envs.md) +> Next: [Checkpoint](03-checkpoint.md) diff --git a/docs/articles/users/tutorials/02-environments/03-checkpoint.md b/docs/articles/users/tutorials/02-environments/03-checkpoint.md new file mode 100644 index 0000000..1e720af --- /dev/null +++ b/docs/articles/users/tutorials/02-environments/03-checkpoint.md @@ -0,0 +1,15 @@ +# Checkpoint: Environment Management + +**What you've accomplished:** + +- Mastered environment listing, switching, and removal +- Learned advanced Python environment initialization and management +- Explored environment diagnostics and troubleshooting +- Understood hatch_mcp_server wrapper management + +**Next up:** + +- Learn how to author your own Hatch packages +- Explore package template generation and metadata editing + +Ready to create your own packages? Let's move on to [Package Authoring](../03-author-package/01-generate-template.md)! diff --git a/docs/articles/users/tutorials/03-author-package/01-generate-template.md b/docs/articles/users/tutorials/03-author-package/01-generate-template.md new file mode 100644 index 0000000..5ef70ee --- /dev/null +++ b/docs/articles/users/tutorials/03-author-package/01-generate-template.md @@ -0,0 +1,158 @@ +# 01: Generate Template + +--- +**Concepts covered:** + +- Package template structure and generated files +- Template generator functionality and default configurations +- Package naming conventions and directory organization + +**Skills you will practice:** + +- Creating new package templates using the CLI +- Understanding generated file purposes and structure +- Customizing template generation with descriptions and directories + +--- + +This article covers generating new Hatch package templates to start developing MCP server packages. + +## Step 1: Generate Basic Template + +Create a new package template with the default settings: + +```bash +hatch create my_new_package +``` + +This generates a package directory `my_new_package/` in the current directory with the following structure: + +```txt +my_new_package/ +├── __init__.py +├── mcp_server.py +├── hatch_mcp_server_entry.py +├── hatch_metadata.json +└── README.md +``` + +## Step 2: Customize Template Generation + +Generate a template with custom options: + +```bash +hatch create advanced-package --dir ./packages --description "An advanced MCP server package" +``` + +### Template Generation Options + +- `--dir` / `-d` - Target directory (default: current directory) +- `--description` / `-D` - Package description (appears in metadata and README) + +## Step 3: Understanding Generated Files + +Let's examine each generated file: + +### `__init__.py` + +```python +# Hatch package initialization +``` + +Standard Python package initialization file. It is empty by default; there is not much of a reason in the case of a hatch package to put anything in this file afterward. Indeed, a hatch package wraps an MCP server which purpose it not to expose any public API but only to provide a set of functionalities for LLMs. Hence, `__init__.py` exists only for python imports within the package. + +### `mcp_server.py` + +Contains the core MCP server implementation using FastMCP: + +```python +from mcp.server.fastmcp import FastMCP + +mcp = FastMCP("my_new_package", log_level="WARNING") + +@mcp.tool() +def example_tool(param: str) -> str: + """Example tool function. + + Args: + param (str): Example parameter. + + Returns: + str: Example result.""" + + return f"Processed: {param}" + +if __name__ == "__main__": + mcp.run() +``` + +This is where the magic stems from. You can add as many tools as you want by adding more functions decorated with `@mcp.tool()`. The tools are the actual functions that will be called by the LLMs. The tools are defined by the function signature and its docstring. **The docstring is used by the LLMs to understand what the tool does and how to use it, thus it is crucial to write it correctly.** +You will likely spend most of your time in this file. + +### `hatch_mcp_server_entry.py` + +Wraps the MCP server for Hatch integration: + +```python +from hatch_mcp_server import HatchMCP +from mcp_server import mcp + +hatch_mcp = HatchMCP("my_new_package", + fast_mcp=mcp, + origin_citation="Origin citation for my_new_package", + mcp_citation="MCP citation for my_new_package") + +if __name__ == "__main__": + hatch_mcp.server.run() +``` + +`HatchMCP`can be found in another [repository](https://github.com/CrackingShells/hatch-mcp-server) of the __Cracking Shells__ organization. Its purpose is to expose MCP resources by default to provide support for scientific citation of the underlying MCP server and its tools. We believe this is critical for the attribution and transparency in the context of massive tools usage by autonomous systems such as so-called scientific agents. + +That being the case, this wrapper is currently only used within the software suite of the __Cracking Shells__ organization (typically [Hatchling](https://github.com/CrackingShells/Hatchling)). + +### `hatch_metadata.json` + +Contains package metadata following the Hatch schema (see next article for details). + +### `README.md` + +Basic documentation template with package description and tool listing. + +We strongly recommend that you include in this file typical information about the package, its tools, and how to use them. In particular, despite the stochastic nature of the LLMs, what typical prompts triggers tool usage; and what is the typical output given by an LLM. + +## Step 4: Package Naming Guidelines + +Follow these conventions for package names: + +- Use lowercase letters, numbers, and underscores only +- Start with a letter +- Use descriptive, meaningful names +- Avoid conflicts with existing packages + +**Exercise:** +Generate three different packages: one with default settings, one in a custom directory, and one with a detailed description. Examine the differences in the generated files. + +
+Solution + +```bash +# Default package +hatch create basic_package + +# Custom directory +hatch create custom_package --dir ./my-packages + +# With description +hatch create described-package --description "A package that demonstrates detailed descriptions" + +# Examine the differences +cat basic-package/hatch_metadata.json +cat my-packages/custom_package/hatch_metadata.json +cat described-package/hatch_metadata.json +``` + +The description should appear in the metadata and README files. +
+ +> Previous: [Environment Management Checkpoint](../02-environments/03-checkpoint.md) +> Next: [Edit Metadata](02-edit-metadata.md) diff --git a/docs/articles/users/tutorials/03-author-package/02-edit-metadata.md b/docs/articles/users/tutorials/03-author-package/02-edit-metadata.md new file mode 100644 index 0000000..0270661 --- /dev/null +++ b/docs/articles/users/tutorials/03-author-package/02-edit-metadata.md @@ -0,0 +1,231 @@ +# 02: Edit Metadata + +--- +**Concepts covered:** + +- Hatch package metadata schema structure and validation +- Required and optional metadata fields +- Dependency specification and compatibility constraints +- Package versioning and licensing + +**Skills you will practice:** + +- Editing the hatch_metadata.json file +- Understanding schema validation requirements +- Configuring dependencies and compatibility settings +- Setting up proper package metadata for distribution + +--- + +This article covers editing the `hatch_metadata.json` file that defines your package's metadata and dependencies. + +## Understanding the Metadata Schema + +The `hatch_metadata.json` file follows the schema defined in `Hatch-Schemas/package/v1.2.0/hatch_pkg_metadata_schema.json`. + +The package's metadata are not critical to the package's functionality but are critical to its distribution and installation by the Hatch ecosystem. For example, [dependencies](#step-3-define-dependencies) information are leveraged during the package's installation to automatically resolve and install any required dependencies. The list of [tools](#step-5-define-tools-and-citations) your package provides is used during package submission to the online registry to confirm the content of your mcp server. Except these, the metadata are not used at runtime simply helps providing information to users and transparency to the community. + +## Step 1: Required Fields + +Every package must include these fields: + +```json +{ + "package_schema_version": "1.2.0", + "name": "package_name", + "version": "0.1.0", + "entry_point": "hatch_mcp_server_entry.py", + "description": "Package description", + "tags": [], + "author": { + "name": "Author Name" + }, + "license": { + "name": "MIT" + } +} +``` + +## Step 2: Configure Package Information + +Edit the basic package information: + +```json +{ + "package_schema_version": "1.2.0", + "name": "my_awesome_package", + "version": "1.0.0", + "description": "An awesome MCP server package that does amazing things", + "tags": ["mcp", "server", "automation", "productivity"], + "author": { + "name": "Your Name", + "email": "your.email@example.com" + }, + "contributors": [ + { + "name": "Contributor Name", + "email": "contributor@example.com" + } + ], + "license": { + "name": "MIT", + "uri": "https://opensource.org/licenses/MIT" + }, + "repository": "https://github.com/yourusername/my-awesome-package", + "documentation": "https://your-docs-site.com", + "entry_point": "hatch_mcp_server_entry.py" +} +``` + +## Step 3: Define Dependencies + +Configure different types of dependencies: + +```json +{ + "dependencies": { + "hatch": [ + { + "name": "another_hatch_package", + "version_constraint": ">=1.0.0" + } + ], + "python": [ + { + "name": "requests", + "version_constraint": ">=2.28.0", + "package_manager": "pip" + }, + { + "name": "numpy", + "version_constraint": "==1.24.0", + "package_manager": "pip" + } + ], + "system": [ + { + "name": "curl", + "version_constraint": ">=7.0.0", + "package_manager": "apt" + } + ], + "docker": [ + { + "name": "redis", + "version_constraint": ">=7.0.0", + "registry": "dockerhub" + } + ] + } +} +``` + +### Version Constraint Patterns + +- `==1.0.0` - Exact version +- `>=1.0.0` - Minimum version +- `<=2.0.0` - Maximum version +- `!=1.5.0` - Exclude specific version + +## Step 4: Set Compatibility Requirements + +Define compatibility constraints: + +```json +{ + "compatibility": { + "hatchling": ">=0.1.0", + "python": ">=3.8" + } +} +``` + +## Step 5: Define Tools and Citations + +Document the tools your package provides: + +```json +{ + "tools": [ + { + "name": "search_web", + "description": "Search the web for information" + }, + { + "name": "analyze_data", + "description": "Analyze data sets and generate reports" + } + ], + "citations": { + "origin": "Based on the WebSearch project by Example Corp", + "mcp": "Implements MCP specification version 1.0" + } +} +``` + +## Step 6: Validate Metadata Structure + +The schema enforces several validation rules: + +- **Package names** must match pattern `^[a-z0-9_]+$` +- **Versions** must follow semantic versioning `^\d+(\.\d+)*$` +- **Email addresses** must be valid email format +- **URIs** must be valid URI format +- **Version constraints** must match pattern `^\s*(==|>=|<=|!=)\s*\d+(\.\d+)*$` + +**Exercise:** +Create a complete metadata file for a hypothetical "weather-service" package that depends on the requests library and provides weather-related tools. + +
+Solution + +```json +{ + "package_schema_version": "1.2.0", + "name": "weather_service", + "version": "1.0.0", + "description": "MCP server for weather information and forecasting", + "tags": ["weather", "forecast", "api", "mcp"], + "author": { + "name": "Weather Developer", + "email": "dev@weather-service.com" + }, + "license": { + "name": "MIT", + "uri": "https://opensource.org/licenses/MIT" + }, + "repository": "https://github.com/weather/weather-service", + "entry_point": "hatch_mcp_server_entry.py", + "dependencies": { + "python": [ + { + "name": "requests", + "version_constraint": ">=2.28.0", + "package_manager": "pip" + } + ] + }, + "compatibility": { + "python": ">=3.8" + }, + "tools": [ + { + "name": "get_current_weather", + "description": "Get current weather for a location" + }, + { + "name": "get_forecast", + "description": "Get weather forecast for a location" + } + ], + "citations": { + "origin": "Weather data provided by OpenWeather API", + "mcp": "Implements MCP specification for weather services" + } +} +``` + +
+ +> Previous: [Generate Template](01-generate-template.md) +> Next: [Validate and Install](03-validate-and-install.md) diff --git a/docs/articles/users/tutorials/03-author-package/03-validate-and-install.md b/docs/articles/users/tutorials/03-author-package/03-validate-and-install.md new file mode 100644 index 0000000..cf3924c --- /dev/null +++ b/docs/articles/users/tutorials/03-author-package/03-validate-and-install.md @@ -0,0 +1,195 @@ +# 03: Validate and Install + +--- +**Concepts covered:** + +- Package validation process and schema verification +- Dependency resolution and installation orchestration +- Package installation flows and error handling + +**Skills you will practice:** + +- Validating package structure and metadata +- Installing packages from local directories +- Understanding validation feedback and error messages + +--- + +## Understanding Validation + +This article covers validating your package and installing it into a Hatch environment. + +The `validate` CLI subcommand constructs a `HatchPackageValidator` with `version='latest'` and `allow_local_dependencies=True`, passing the environment manager's registry data into the validator. On success the command exits with code 0; on failure it exits with code 1 and prints grouped validation errors (the CLI prints category-level errors except for the special `valid` and `metadata` categories). + +## Step 1: Validate Package Structure + +Before installing or submitting your package to the online registry, validate your package meets Hatch requirements: + +```bash +hatch validate /path/to/my-package +``` + +The validation process checks: + +- **Metadata schema compliance** - Ensures `hatch_metadata.json` follows the latest schema +- **Required files presence** - Verifies entry point and essential files exist +- **Field validation** - Checks naming patterns, version formats, and constraints +- **Dependency specifications** - Validates dependency definitions and constraints + +### Successful Validation + +```txt +Package validation SUCCESSFUL: /path/to/my-package +``` + +The command will exit with status code 0 when validation succeeds. + +### Failed Validation + +```txt +Package validation FAILED: /path/to/my-package +``` + +When validation fails the CLI prints a failure line and then — if detailed results are available — lists errors grouped by category. The CLI deliberately skips printing the `valid` and `metadata` categories in that grouped output and only prints categories which are invalid and include an `errors` list. + +Example failure output (category grouping): + +```txt +Package validation FAILED: C:\path\to\my-package + +Entry Point Errors: + - Missing required field 'entry_point' + - Entry point 'hatch_mcp_server_entry.py' not found + +Dependency Errors: + - Dependency 'some_pkg' not found in registry +``` + +Note that `metadata` category details may not be printed in the grouped list by the CLI loop; important metadata validation failures may still be reported elsewhere by the validator object or surfaced in tooling that consumes the validator results. + +The command will exit with status code 1 when validation fails. + +## Step 2: Fix Common Validation Issues + +### Invalid Package Name + +```json +// ❌ Invalid - contains hyphens +"name": "my-package" + +// ✅ Valid - uses underscores +"name": "my_package" +``` + +### Invalid Version Format + +```json +// ❌ Invalid - not semantic versioning +"version": "1.0" + +// ✅ Valid - proper semantic version +"version": "1.0.0" +``` + +### Missing Required Fields + +Ensure all required fields are present: + +- `package_schema_version` +- `name` +- `version` +- `entry_point` +- `description` +- `tags` +- `author` +- `license` + +## Step 3: Install Validated Package + +Once validation passes, install the package: + +```bash +# Set target environment +hatch env use my_dev_env + +# Install the package +hatch package add /path/to/my-package +``` + +The installation process involves: + +1. **Package loading** - The `package_loader` reads and parses the package metadata +2. **Dependency resolution** - The system identifies all required dependencies +3. **Installation orchestration** - The `dependency_installation_orchestrator` coordinates installation +4. **Installer execution** - Different installers handle Hatch, Python, system, and Docker dependencies + +### Installation Components + +The installation system uses several specialized installers: + +- **`hatch_installer.py`** - Handles Hatch package dependencies +- **`python_installer.py`** - Manages Python package installation via pip +- **`system_installer.py`** - Handles system package installation +- **`docker_installer.py`** - Manages Docker image dependencies +- **`installer_base.py`** - Provides common installer interface + +## Step 4: Verify Installation + +Confirm the package was installed successfully: + +```bash +hatch package list +``` + +Output should show your package: + +```txt +Packages in environment 'my_dev_env': +my_package (1.0.0) Hatch compliant: true source: file:///path/to/my-package location: /env/path/my_package +``` + +## Step 5: Test Package Functionality + +After installation, test that your MCP server works: + +```bash +# Test the package entry point +python /env/path/my_package/hatch_mcp_server_entry.py +``` + +**Exercise:** +Create a package with intentional validation errors (invalid name, missing fields), attempt validation, fix the errors, and then successfully validate and install the package. + +
+Solution + +```bash +# 1. Create package with errors +hatch create test-package # Invalid name with hyphens + +# Edit hatch_metadata.json to introduce errors: +# - Change name to include hyphens +# - Remove required field like "description" +# - Use invalid version format + +# 2. Attempt validation (should fail) +hatch validate test-package + +# 3. Fix errors: +# - Change name to "test_package" +# - Add missing required fields +# - Use proper version format like "1.0.0" + +# 4. Validate again (should succeed) +hatch validate test-package + +# 5. Install the corrected package +hatch env use my_dev_env +hatch package add ./test-package +hatch package list +``` + +
+ +> Previous: [Edit Metadata](02-edit-metadata.md) +> Next: [Checkpoint](04-checkpoint.md) diff --git a/docs/articles/users/tutorials/03-author-package/04-checkpoint.md b/docs/articles/users/tutorials/03-author-package/04-checkpoint.md new file mode 100644 index 0000000..b3c72d5 --- /dev/null +++ b/docs/articles/users/tutorials/03-author-package/04-checkpoint.md @@ -0,0 +1,16 @@ +# Checkpoint: Package Authoring + +**What you've accomplished:** + +- Generated package templates with customization options +- Mastered the hatch_metadata.json schema and field requirements +- Learned validation processes and error resolution +- Successfully installed locally developed packages + +**Next steps:** + +- Explore advanced CLI features and configuration options +- Learn about security considerations and troubleshooting +- Understand the development architecture for contributing + +You now have the fundamental skills to create, validate, and install Hatch packages. For more advanced topics, explore the [CLI Reference](../../CLIReference.md) and [Security and Trust](../../SecurityAndTrust.md) guides. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..f12bca2 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,72 @@ +# Hatch Documentation + +Welcome to the documentation for Hatch, the official package manager for the Hatch! ecosystem. + +## Overview + +Hatch provides powerful tools for managing MCP server packages, environments, and interacting with the Hatch registry. It serves as the package management foundation for [Hatchling](https://github.com/CrackingShells/Hatchling) and other projects in the ecosystem. + +## Documentation Sections + +### For Users + +- **[Getting Started](./articles/users/GettingStarted.md)** - Quick start guide for using Hatch +- **[Command Reference](./articles/users/CLIReference.md)** - Complete CLI command documentation +- **[Tutorials Start](./articles/users/tutorials/01-getting-started/01-installation.md)** - Step-by-step guides for your journey from installation to authoring Hatch packages for MCP server easy sharing. + +### For Developers + +Comprehensive documentation for developers and contributors working on the Hatch codebase. + +#### [Architecture](./articles/devs/architecture/) + +High-level system understanding and design patterns for developers getting familiar with the Hatch codebase. + +- [System Overview](./articles/devs/architecture/system_overview.md) - Introduction to Hatch's architecture +- [Component Architecture](./articles/devs/architecture/component_architecture.md) - Detailed component breakdown + +#### [Implementation Guides](./articles/devs/implementation_guides/) + +Technical how-to guides for implementing specific features and extending the system. + +- [Adding New Installers](./articles/devs/implementation_guides/adding_installers.md) - Implementing new dependency installer types +- [Registry Integration](./articles/devs/implementation_guides/registry_integration.md) - Working with package registries + +#### [Development Processes](./articles/devs/development_processes/) + +Workflow, standards, and processes for effective development on the Hatch project. + +- [Developer Onboarding](./articles/devs/development_processes/developer_onboarding.md) - Setting up your development environment +- [Testing Standards](./articles/devs/development_processes/testing_standards.md) - Testing requirements and best practices + +#### [Contribution Guidelines](./articles/devs/contribution_guides/) + +Process-focused guidance for contributing to the Hatch project. + +- [How to Contribute](./articles/devs/contribution_guides/how_to_contribute.md) - General contribution workflow +- [Release Policy](./articles/devs/contribution_guides/release_policy.md) - Release management policies + +## Quick Links + +- **[Architecture Diagram](./resources/diagrams/architecture.puml)** - Visual overview of system components +- **[Source Code](../hatch/)** - Main Hatch package source code +- **[GitHub Repository](https://github.com/CrackingShells/Hatch)** - Project repository +- **[Hatchling Integration](https://github.com/CrackingShells/Hatchling)** - Primary consumer of Hatch + +## Additional Resources + +### Reference Materials + +- **[Glossary](./articles/appendices/glossary.md)** - Key terms and definitions +- **[State and Data Models](./articles/appendices/state_and_data_models.md)** - Data structures and state management + +### External Resources + +- **[Hatch Schemas](https://github.com/CrackingShells/Hatch-Schemas)** - Package metadata schemas +- **[Hatch Registry](https://github.com/CrackingShells/Hatch-Registry)** - Central package registry +- **[Hatch Validator](https://github.com/CrackingShells/Hatch-Validator)** - Package validation tools + +## Getting Help + +- Search existing [GitHub Issues](https://github.com/CrackingShells/Hatch/issues) +- Create a new issue for bugs or feature requests diff --git a/docs/resources/diagrams/architecture.puml b/docs/resources/diagrams/architecture.puml new file mode 100644 index 0000000..9dd8ebd --- /dev/null +++ b/docs/resources/diagrams/architecture.puml @@ -0,0 +1,90 @@ +@startuml architecture +!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml + +LAYOUT_WITH_LEGEND() + +title Hatch Architecture Overview + +Container_Boundary(cli, "CLI Layer") { + Component(cli_hatch, "CLI Interface", "Python", "Command-line interface\nArgument parsing and validation") +} + +Container_Boundary(core, "Core Management") { + Component(env_manager, "Environment Manager", "Python", "Environment lifecycle\nMetadata persistence\nCurrent environment tracking") + Component(python_env_manager, "Python Environment Manager", "Python", "Conda/mamba integration\nPython executable resolution\nPackage installation coordination") +} + +Container_Boundary(package_system, "Package System") { + Component(package_loader, "Package Loader", "Python", "Local package inspection\nMetadata validation\nStructure verification") + Component(template_generator, "Template Generator", "Python", "Package template creation\nBoilerplate file generation\nDefault metadata setup") +} + +Container_Boundary(registry, "Registry System") { + Component(registry_retriever, "Registry Retriever", "Python", "Package downloads\nCaching with TTL\nNetwork fallback handling") + Component(registry_explorer, "Registry Explorer", "Python", "Package discovery\nSearch capabilities\nRegistry metadata parsing") +} + +Container_Boundary(installation, "Installation System") { + Component(orchestrator, "Dependency Orchestrator", "Python", "Multi-type dependency coordination\nInstallation planning\nProgress reporting") + Component(installation_context, "Installation Context", "Python", "Installation state management\nEnvironment isolation\nProgress tracking") + Component(installer_base, "Installer Base", "Python", "Common installer interface\nError handling patterns\nResult aggregation") + + Component(python_installer, "Python Installer", "Python", "Pip package installation\nConda environment integration") + Component(system_installer, "System Installer", "Python", "System package installation\nPrivilege management\nAPT/package manager integration") + Component(docker_installer, "Docker Installer", "Python", "Docker image management\nRegistry authentication\nImage version handling") + Component(hatch_installer, "Hatch Installer", "Python", "Hatch package dependencies\nRecursive installation\nPackage registration") +} + +Container_Boundary(validation, "Validation System") { + Component(hatch_validator, "Hatch Validator", "Python", "Schema validation\nPackage compliance checking\nMetadata verification") +} + +Container_Boundary(external, "External Systems") { + ComponentDb(conda_mamba, "Conda/Mamba", "Environment Manager", "Python environment creation\nPackage management") + ComponentDb(docker_engine, "Docker Engine", "Container Runtime", "Image management\nContainer execution") + ComponentDb(system_pm, "System Package Managers", "OS Tools", "APT, YUM, etc.\nSystem-level dependencies") + ComponentDb(registry_remote, "Package Registry", "Remote Service", "Package repository\nMetadata distribution") + ComponentDb(schemas, "Hatch Schemas", "JSON Schema", "Package metadata schema\nValidation rules") +} + +' CLI relationships +Rel(cli_hatch, env_manager, "Manages environments") +Rel(cli_hatch, package_loader, "Loads and validates packages") +Rel(cli_hatch, template_generator, "Creates package templates") + +' Core management relationships +Rel(env_manager, python_env_manager, "Delegates Python operations") +Rel(env_manager, orchestrator, "Coordinates installations") +Rel(python_env_manager, conda_mamba, "Creates Python environments") + +' Package system relationships +Rel(package_loader, hatch_validator, "Validates packages") +Rel(template_generator, schemas, "Uses schema for templates") + +' Registry system relationships +Rel(registry_retriever, registry_remote, "Downloads packages") +Rel(registry_explorer, registry_remote, "Searches packages") + +' Installation system relationships +Rel(orchestrator, installation_context, "Manages context") +Rel(orchestrator, python_installer, "Installs Python packages") +Rel(orchestrator, system_installer, "Installs system packages") +Rel(orchestrator, docker_installer, "Installs Docker images") +Rel(orchestrator, hatch_installer, "Installs Hatch packages") + +Rel(python_installer, installer_base, "Implements") +Rel(system_installer, installer_base, "Implements") +Rel(docker_installer, installer_base, "Implements") +Rel(hatch_installer, installer_base, "Implements") + +' External system relationships +Rel(python_installer, conda_mamba, "Installs packages via pip") +Rel(system_installer, system_pm, "Installs system packages") +Rel(docker_installer, docker_engine, "Manages Docker images") +Rel(hatch_installer, package_loader, "Loads dependency packages") + +' Validation relationships +Rel(hatch_validator, schemas, "Uses validation schemas") + +SHOW_LEGEND() +@enduml diff --git a/docs/resources/images/Logo/hatch_wide_dark_bg_transparent.png b/docs/resources/images/Logo/hatch_wide_dark_bg_transparent.png new file mode 100644 index 0000000..b7bef7e Binary files /dev/null and b/docs/resources/images/Logo/hatch_wide_dark_bg_transparent.png differ diff --git a/docs/resources/images/Logo/hatch_wide_light_bg_transparent.png b/docs/resources/images/Logo/hatch_wide_light_bg_transparent.png new file mode 100644 index 0000000..bb040f1 Binary files /dev/null and b/docs/resources/images/Logo/hatch_wide_light_bg_transparent.png differ diff --git a/docs/resources/images/architecture.svg b/docs/resources/images/architecture.svg new file mode 100644 index 0000000..c31f7cd --- /dev/null +++ b/docs/resources/images/architecture.svg @@ -0,0 +1 @@ +Hatch Architecture OverviewHatch Architecture OverviewCLI Layer[container]Core Management[container]Package System[container]Registry System[container]Installation System[container]Validation System[container]External Systems[container]CLI Interface[Python] Command-line interfaceArgument parsing andvalidationEnvironment Manager[Python] Environment lifecycleMetadata persistenceCurrent environmenttrackingPython EnvironmentManager[Python] Conda/mamba integrationPython executableresolutionPackage installationcoordinationPackage Loader[Python] Local package inspectionMetadata validationStructure verificationTemplate Generator[Python] Package template creationBoilerplate file generationDefault metadata setupRegistry Retriever[Python] Package downloadsCaching with TTLNetwork fallback handlingRegistry Explorer[Python] Package discoverySearch capabilitiesRegistry metadata parsingDependencyOrchestrator[Python] Multi-type dependencycoordinationInstallation planningProgress reportingInstallation Context[Python] Installation statemanagementEnvironment isolationProgress trackingInstaller Base[Python] Common installer interfaceError handling patternsResult aggregationPython Installer[Python] Pip package installationConda environmentintegrationSystem Installer[Python] System package installationPrivilege managementAPT/package managerintegrationDocker Installer[Python] Docker image managementRegistry authenticationImage version handlingHatch Installer[Python] Hatch packagedependenciesRecursive installationPackage registrationHatch Validator[Python] Schema validationPackage compliancecheckingMetadata verificationConda/Mamba[Environment Manager] Python environmentcreationPackage managementDocker Engine[Container Runtime] Image managementContainer executionSystem PackageManagers[OS Tools] APT, YUM, etc.System-level dependenciesPackage Registry[Remote Service] Package repositoryMetadata distributionHatch Schemas[JSON Schema] Package metadata schemaValidation rulesManagesenvironmentsLoads and validatespackagesCreates packagetemplatesDelegates PythonoperationsCoordinatesinstallationsCreates PythonenvironmentsValidates packagesUses schema fortemplatesDownloads packagesSearches packagesManages contextInstalls PythonpackagesInstalls systempackagesInstalls DockerimagesInstalls HatchpackagesImplementsImplementsImplementsImplementsInstalls packages viapipInstalls systempackagesManages DockerimagesLoads dependencypackagesUses validationschemasLegend ▯ component ▯ container boundary  \ No newline at end of file