Env substitution

AGENTS.md

@@ -8,6 +8,31 @@
 - **Format**: `go fmt ./...`
 - **Dependencies**: `go mod tidy`
 
+## UI and Output Modes
+MCPHost supports multiple output modes for different use cases:
+
+### Standard Mode (Default)
+- Full-featured terminal UI with rich styling and formatting
+- Interactive message display with proper spacing and visual hierarchy
+- Suitable for regular interactive usage
+
+### Compact Mode (`--compact`)
+- **Location**: `internal/ui/compact_renderer.go`, `internal/ui/cli.go`
+- **Purpose**: Simplified output format without fancy styling for better readability in automation contexts
+- **Features**: 
+  - Single-line message format where possible
+  - Minimal visual styling and spacing
+  - Consistent symbol-based prefixes (🧑, 🤖, 🔧, etc.)
+  - Optimized for scripting and log parsing
+- **Usage**: Add `--compact` flag to any mcphost command
+- **Note**: Has no effect when combined with `--quiet` (mutually exclusive)
+
+### Quiet Mode (`--quiet`)
+- Suppresses all UI elements except the AI response
+- Only works with `--prompt` (non-interactive mode)
+- Ideal for shell scripting and piping output to other commands
+- **Note**: When `--quiet` is used, `--compact` has no effect since no UI elements are shown
+
 ## Code Style Guidelines
 - **Package structure**: `pkg/` for reusable packages, `cmd/` for CLI commands
 - **Imports**: Standard library first, then third-party, then local packages with blank lines between groups
@@ -83,33 +108,57 @@ MCPHost includes several builtin MCP servers for common functionality:
 - **Features**: HTTP/HTTPS support, HTML parsing, markdown conversion, size limits
 - **Security**: 5MB response limit, configurable timeouts, localhost-aware HTTPS upgrade
 
+## Environment Variable Substitution System
+MCPHost includes a comprehensive environment variable substitution system for secure configuration management.
+
+### Implementation Details
+- **Location**: `internal/config/substitution.go` (core logic), `internal/config/substitution_test.go` (unit tests)
+- **Purpose**: Replace `${env://VAR}` and `${env://VAR:-default}` patterns with environment variable values
+- **Integration**: Works in both config files and script frontmatter/prompts
+
+### Substitution Components
+- **EnvSubstituter**: Handles environment variable substitution
+- **ArgsSubstituter**: Handles script argument substitution (refactored from existing code)
+- **Shared Parsing Logic**: `parseVariableWithDefault()` function used by both substituters
+
+### Processing Order
+1. **Config Loading**: `cmd/root.go` → `loadConfigWithEnvSubstitution()` → env substitution → YAML/JSON parsing
+2. **Script Mode**: `cmd/script.go` → `parseScriptContent()` → env substitution → YAML parsing → args substitution
+
+### Security Features
+- **No Shell Execution**: Direct environment variable lookup using `os.Getenv()`
+- **Error Handling**: Clear error messages for missing required variables
+- **Validation**: Regex-based pattern matching with comprehensive validation
+
 ## Script System
 MCPHost includes a powerful script system for automation and reusable workflows.
 
 ### Script Features
 - **Location**: `cmd/script.go` (main implementation), `cmd/script_test.go` (comprehensive tests)
-- **Purpose**: Execute YAML-based automation scripts with variable substitution
+- **Purpose**: Execute YAML-based automation scripts with dual variable substitution
 - **Format**: YAML frontmatter + prompt content in single executable files
 
 ### Variable Substitution System
-- **Required Variables**: `${variable}` - Must be provided via `--args:variable value`
-- **Optional Variables**: `${variable:-default}` - Uses default if not provided
+- **Environment Variables**: `${env://VAR}` and `${env://VAR:-default}` - Processed first
+- **Script Arguments**: `${variable}` and `${variable:-default}` - Processed after environment variables
 - **Features**: 
   - Bash-style default syntax for familiarity
-  - Empty defaults supported: `${var:-}`
+  - Empty defaults supported: `${var:-}` and `${env://VAR:-}`
   - Complex defaults: paths, URLs, commands with spaces
   - Full backward compatibility with existing scripts
-- **Implementation**: Regex-based parsing with comprehensive validation
+- **Implementation**: Dual-phase substitution with shared parsing logic
 
 ### Script Examples
 - **Location**: `examples/scripts/` directory
-- **Demo Script**: `default-values-demo.sh` - Showcases new default values feature
-- **Usage Examples**: Multiple scenarios from simple defaults to complex overrides
+- **Demo Script**: `default-values-demo.sh` - Showcases script argument default values
+- **Env Substitution Script**: `env-substitution-script.sh` - Demonstrates environment variable usage
+- **Usage Examples**: Multiple scenarios from simple defaults to complex environment/args combinations
 
 ### Testing
-- **Test Coverage**: 25+ test cases covering all variable scenarios
-- **Edge Cases**: Empty defaults, complex values, mixed required/optional variables
-- **Backward Compatibility**: Ensures existing scripts continue working unchanged
+- **Unit Tests**: 25+ test cases in `internal/config/substitution_test.go` covering all substitution scenarios
+- **Integration Tests**: `internal/config/integration_test.go` and `cmd/script_integration_test.go`
+- **Edge Cases**: Empty defaults, complex values, mixed required/optional variables, processing order
+- **Backward Compatibility**: Ensures existing scripts and configs continue working unchanged
 
 ## Authentication System
 MCPHost includes optional OAuth authentication for Anthropic Claude as an alternative to API keys.
@@ -124,6 +173,22 @@ MCPHost includes optional OAuth authentication for Anthropic Claude as an altern
 - **Features**: PKCE security, automatic token refresh, encrypted storage, browser-based flow
 - **Priority**: OAuth credentials > API keys (environment variables/flags)
 
+## Recent Features
+
+### Environment Variable Substitution (New)
+- **Feature**: Added support for `${env://VAR}` and `${env://VAR:-default}` syntax in config files and scripts
+- **Implementation**: 
+  - `internal/config/substitution.go`: Core substitution logic with shared parsing
+  - `cmd/root.go`: Config loading integration with `loadConfigWithEnvSubstitution()`
+  - `cmd/script.go`: Script parsing integration with dual-phase substitution
+- **Security**: Environment variables processed safely without shell execution
+- **Compatibility**: Full backward compatibility with existing configurations and scripts
+- **Testing**: Comprehensive test suite with 25+ unit tests and integration tests
+
+### Processing Flow
+1. **Config Files**: Raw content → Env substitution → YAML/JSON parsing → Viper config
+2. **Scripts**: Raw content → Env substitution → YAML parsing → Args substitution → Final config
+
 ## Recent Bug Fixes
 
 ### Legacy MCP Server Configuration Fix

README.md

@@ -94,6 +94,42 @@ MCPHost will automatically create a configuration file in your home directory if
 
 You can also specify a custom location using the `--config` flag.
 
+### Environment Variable Substitution
+
+MCPHost supports environment variable substitution in both config files and script frontmatter using the syntax:
+- **`${env://VAR}`** - Required environment variable (fails if not set)
+- **`${env://VAR:-default}`** - Optional environment variable with default value
+
+This allows you to keep sensitive information like API keys in environment variables while maintaining flexible configuration.
+
+**Example:**
+```yaml
+mcpServers:
+  github:
+    type: local
+    command: ["docker", "run", "-i", "--rm", "-e", "GITHUB_PERSONAL_ACCESS_TOKEN=${env://GITHUB_TOKEN}", "ghcr.io/github/github-mcp-server"]
+    environment:
+      DEBUG: "${env://DEBUG:-false}"
+      LOG_LEVEL: "${env://LOG_LEVEL:-info}"
+
+model: "${env://MODEL:-anthropic:claude-sonnet-4-20250514}"
+provider-api-key: "${env://OPENAI_API_KEY}"  # Required - will fail if not set
+```
+
+**Usage:**
+```bash
+# Set required environment variables
+export GITHUB_TOKEN="ghp_your_token_here"
+export OPENAI_API_KEY="your_openai_key"
+
+# Optionally override defaults
+export DEBUG="true"
+export MODEL="openai:gpt-4"
+
+# Run mcphost
+mcphost
+```
+
 ### Simplified Configuration Schema
 
 MCPHost now supports a simplified configuration schema with three server types:
@@ -105,19 +141,28 @@ For local MCP servers that run commands on your machine:
   "mcpServers": {
     "filesystem": {
       "type": "local",
-      "command": ["npx", "@modelcontextprotocol/server-filesystem", "/tmp"],
+      "command": ["npx", "@modelcontextprotocol/server-filesystem", "${env://WORK_DIR:-/tmp}"],
       "environment": {
-        "DEBUG": "true",
-        "LOG_LEVEL": "info"
+        "DEBUG": "${env://DEBUG:-false}",
+        "LOG_LEVEL": "${env://LOG_LEVEL:-info}",
+        "API_TOKEN": "${env://FS_API_TOKEN}"
       },
       "allowedTools": ["read_file", "write_file"],
       "excludedTools": ["delete_file"]
     },
+    "github": {
+      "type": "local",
+      "command": ["docker", "run", "-i", "--rm", "-e", "GITHUB_PERSONAL_ACCESS_TOKEN=${env://GITHUB_TOKEN}", "ghcr.io/github/github-mcp-server"],
+      "environment": {
+        "DEBUG": "${env://DEBUG:-false}"
+      }
+    },
     "sqlite": {
       "type": "local",
-      "command": ["uvx", "mcp-server-sqlite", "--db-path", "/tmp/foo.db"],
+      "command": ["uvx", "mcp-server-sqlite", "--db-path", "${env://DB_PATH:-/tmp/foo.db}"],
       "environment": {
-        "SQLITE_DEBUG": "1"
+        "SQLITE_DEBUG": "${env://DEBUG:-0}",
+        "DATABASE_URL": "${env://DATABASE_URL:-sqlite:///tmp/foo.db}"
       }
     }
   }
@@ -138,12 +183,12 @@ For remote MCP servers accessible via HTTP:
   "mcpServers": {
     "websearch": {
       "type": "remote",
-      "url": "https://api.example.com/mcp",
-      "headers": ["Authorization: Bearer your-api-token"]
+      "url": "${env://WEBSEARCH_URL:-https://api.example.com/mcp}",
+      "headers": ["Authorization: Bearer ${env://WEBSEARCH_TOKEN}"]
     },
     "weather": {
       "type": "remote", 
-      "url": "https://weather-mcp.example.com"
+      "url": "${env://WEATHER_URL:-https://weather-mcp.example.com}"
     }
   }
 }
@@ -165,7 +210,7 @@ For builtin MCP servers that run in-process for optimal performance:
       "type": "builtin",
       "name": "fs",
       "options": {
-        "allowed_directories": ["/tmp", "/home/user/documents"]
+        "allowed_directories": ["${env://WORK_DIR:-/tmp}", "${env://HOME}/documents"]
       },
       "allowedTools": ["read_file", "write_file", "list_directory"]
     },
@@ -414,7 +459,12 @@ Each in their own environment. Give me the URL of each app
 
 #### Variable Substitution
 
-Scripts support variable substitution using `${variable}` syntax with optional default values. Variables can be provided via command line arguments:
+Scripts support both environment variable substitution and script argument substitution:
+
+1. **Environment Variables**: `${env://VAR}` and `${env://VAR:-default}` - Processed first
+2. **Script Arguments**: `${variable}` and `${variable:-default}` - Processed after environment variables
+
+Variables can be provided via command line arguments:
 
 ```bash
 # Script with variables
@@ -423,38 +473,55 @@ mcphost script myscript.sh --args:directory /tmp --args:name "John"
 
 ##### Variable Syntax
 
-MCPHost supports two variable syntaxes:
+MCPHost supports these variable syntaxes:
 
-1. **Required Variables**: `${variable}` - Must be provided via `--args:variable value`
-2. **Optional Variables with Defaults**: `${variable:-default}` - Uses default if not provided
+1. **Required Environment Variables**: `${env://VAR}` - Must be set in environment
+2. **Optional Environment Variables**: `${env://VAR:-default}` - Uses default if not set
+3. **Required Script Arguments**: `${variable}` - Must be provided via `--args:variable value`
+4. **Optional Script Arguments**: `${variable:-default}` - Uses default if not provided
 
-Example script with mixed variables:
+Example script with mixed environment variables and script arguments:
 ```yaml
 #!/usr/bin/env -S mcphost script
 ---
 mcpServers:
+  github:
+    type: "local"
+    command: ["gh", "api"]
+    environment:
+      GITHUB_TOKEN: "${env://GITHUB_TOKEN}"
+      DEBUG: "${env://DEBUG:-false}"
+
   filesystem:
     type: "local"
-    command: ["npx", "-y", "@modelcontextprotocol/server-filesystem", "${directory:-/tmp}"]
+    command: ["npx", "-y", "@modelcontextprotocol/server-filesystem", "${env://WORK_DIR:-/tmp}"]
+
+model: "${env://MODEL:-anthropic:claude-sonnet-4-20250514}"
 ---
-Hello ${name:-World}! Please list the files in ${directory:-/tmp} and tell me about them.
-Use the ${command:-ls} command to show file details.
+Hello ${name:-World}! Please list ${repo_type:-public} repositories for user ${username}.
+Working directory is ${env://WORK_DIR:-/tmp}.
+Use the ${command:-gh} command to fetch ${count:-10} repositories.
 ```
 
 ##### Usage Examples
 
 ```bash
-# Uses all defaults: name="World", directory="/tmp", command="ls"
+# Set environment variables first
+export GITHUB_TOKEN="ghp_your_token_here"
+export DEBUG="true"
+export WORK_DIR="/home/user/projects"
+
+# Uses env vars and defaults: name="World", repo_type="public", command="gh", count="10"
 mcphost script myscript.sh
 
-# Override specific variables
-mcphost script myscript.sh --args:name "John"
+# Override specific script arguments
+mcphost script myscript.sh --args:name "John" --args:username "alice"
 
-# Override multiple variables  
-mcphost script myscript.sh --args:name "John" --args:directory "/home/john"
+# Override multiple script arguments  
+mcphost script myscript.sh --args:name "John" --args:username "alice" --args:repo_type "private"
 
-# Mix of provided and default values
-mcphost script myscript.sh --args:name "Alice" --args:command "ls -la"
+# Mix of env vars, provided args, and default values
+mcphost script myscript.sh --args:name "Alice" --args:command "gh api" --args:count "5"
 ```
 
 ##### Default Value Features
@@ -464,7 +531,11 @@ mcphost script myscript.sh --args:name "Alice" --args:command "ls -la"
 - **Spaces in defaults**: `${msg:-Hello World}` - Supports spaces in default values
 - **Backward compatibility**: Existing `${variable}` syntax continues to work unchanged
 
-**Important**: Only variables without defaults (e.g., `${directory}`, `${name}`) are required. Variables with defaults are optional and will use their default value if not provided via `--args:variable value` syntax.
+**Important**: 
+- Environment variables without defaults (e.g., `${env://GITHUB_TOKEN}`) are required and must be set in the environment
+- Script arguments without defaults (e.g., `${username}`) are required and must be provided via `--args:variable value` syntax
+- Variables with defaults are optional and will use their default value if not provided
+- Environment variables are processed first, then script arguments
 
 #### Script Features
 
@@ -552,7 +623,10 @@ mcphost --model openai:<your-model-name> \
 # Single prompt with full UI
 mcphost -p "List files in the current directory"
 
-# Quiet mode for scripting (only AI response output)
+# Compact mode for cleaner output without fancy styling
+mcphost -p "List files in the current directory" --compact
+
+# Quiet mode for scripting (only AI response output, no UI elements)
 mcphost -p "What is the capital of France?" --quiet
 
 # Use in shell scripts
@@ -573,6 +647,7 @@ mcphost -p "Generate a random UUID" --quiet | tr '[:lower:]' '[:upper:]'
 - `-m, --model string`: Model to use (format: provider:model) (default "anthropic:claude-sonnet-4-20250514")
 - `-p, --prompt string`: **Run in non-interactive mode with the given prompt**
 - `--quiet`: **Suppress all output except the AI response (only works with --prompt)**
+- `--compact`: **Enable compact output mode without fancy styling (ideal for scripting and automation)**
 - `--stream`: Enable streaming responses (default: true, use `--stream=false` to disable)
 
 ### Authentication Subcommands
@@ -704,12 +779,34 @@ curl -X POST http://localhost:8080/process \
 ```
 
 ### Tips for Scripting
-- Use `--quiet` flag to get clean output suitable for parsing
+- Use `--quiet` flag to get clean output suitable for parsing (only AI response, no UI)
+- Use `--compact` flag for simplified output without fancy styling (when you want to see UI elements)
+- Note: `--compact` and `--quiet` are mutually exclusive - `--compact` has no effect with `--quiet`
+- **Use environment variables for sensitive data** like API keys instead of hardcoding them
+- **Use `${env://VAR}` syntax** in config files and scripts for environment variable substitution
 - Combine with standard Unix tools (`grep`, `awk`, `sed`, etc.)
 - Set appropriate timeouts for long-running operations
 - Handle errors appropriately in your scripts
 - Use environment variables for API keys in production
 
+#### Environment Variable Best Practices
+```bash
+# Set sensitive variables in environment
+export GITHUB_TOKEN="ghp_your_token_here"
+export OPENAI_API_KEY="your_openai_key"
+export DATABASE_URL="postgresql://user:pass@localhost/db"
+
+# Use in config files
+mcpServers:
+  github:
+    environment:
+      GITHUB_TOKEN: "${env://GITHUB_TOKEN}"
+      DEBUG: "${env://DEBUG:-false}"
+
+# Use in scripts
+mcphost script my-script.sh --args:username alice
+```
+
 ## MCP Server Compatibility 🔌
 
 MCPHost can work with any MCP-compliant server. For examples and reference implementations, see the [MCP Servers Repository](https://github.com/modelcontextprotocol/servers).