[MNT] Change examples to v5 + small fixes

.github/workflows/test.yml

@@ -83,7 +83,7 @@ jobs:
       - name: Show dependencies
         run: python -m pip list
 
-      - name: Test with pytest
+      - name: Test core package
         run: |
           python -m pytest src/hyperactive -p no:warnings
 
@@ -117,7 +117,7 @@ jobs:
       - name: Show dependencies
         run: python -m pip list
 
-      - name: Test with pytest
+      - name: Test core package
         run: |
           python -m pytest src/hyperactive -p no:warnings
 
@@ -150,3 +150,54 @@ jobs:
       - name: Run sklearn integration tests for ${{ matrix.sklearn-version }}
         run: |
           python -m pytest -x -p no:warnings src/hyperactive/integrations/sklearn/
+
+  test-examples:
+    name: test-examples
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Check for example changes
+        id: check-examples
+        run: |
+          if [ "${{ github.event_name }}" == "push" ]; then
+            # For pushes, compare with previous commit
+            CHANGED_FILES=$(git diff --name-only HEAD~1 HEAD | grep "^examples/" || true)
+          else
+            # For pull requests, compare with base branch
+            CHANGED_FILES=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep "^examples/" || true)
+          fi
+
+          if [ -n "$CHANGED_FILES" ]; then
+            echo "examples_changed=true" >> $GITHUB_OUTPUT
+            echo "Examples changed:"
+            echo "$CHANGED_FILES"
+          else
+            echo "examples_changed=false" >> $GITHUB_OUTPUT
+            echo "No example files changed"
+          fi
+
+      - name: Install dependencies
+        if: steps.check-examples.outputs.examples_changed == 'true'
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install build
+          make install-all-extras-for-test
+
+      - name: Show dependencies
+        if: steps.check-examples.outputs.examples_changed == 'true'
+        run: python -m pip list
+
+      - name: Test examples
+        if: steps.check-examples.outputs.examples_changed == 'true'
+        run: |
+          python -m pytest examples/ -v -p no:warnings

examples/__init__.py

@@ -0,0 +1 @@
+"""Test package for Hyperactive library functionality."""

examples/gfo/README.md

@@ -0,0 +1,147 @@
+# GFO (Gradient-Free Optimization) Examples
+
+This directory contains examples demonstrating the GFO optimization backend in Hyperactive. GFO provides a collection of gradient-free optimization algorithms that don't require derivative information, making them suitable for black-box optimization problems like hyperparameter tuning.
+
+## Quick Start
+
+Run any example directly:
+```bash
+python random_search_example.py
+python hill_climbing_example.py
+python bayesian_optimization_example.py
+# ... etc
+```
+
+## Available Algorithms
+
+| Algorithm | Type | Best For | Characteristics |
+|-----------|------|----------|-----------------|
+| [RandomSearch](random_search_example.py) | Random | Baselines, high-dim spaces | Simple, parallel-friendly |
+| [GridSearch](grid_search_example.py) | Exhaustive | Small discrete spaces | Complete coverage |
+| [HillClimbing](hill_climbing_example.py) | Local Search | Fast convergence | Exploits local structure |
+| [StochasticHillClimbing](stochastic_hill_climbing_example.py) | Local Search | Plateau handling | Probabilistic moves |
+| [RepulsingHillClimbing](repulsing_hill_climbing_example.py) | Local Search | Multi-optima discovery | Memory-based repulsion |
+| [RandomRestartHillClimbing](random_restart_hill_climbing_example.py) | Multi-start | Multi-modal problems | Multiple hill climbs |
+| [SimulatedAnnealing](simulated_annealing_example.py) | Probabilistic | Escaping local optima | Temperature cooling |
+| [ParallelTempering](parallel_tempering_example.py) | MCMC | Complex landscapes | Multi-temperature chains |
+| [PatternSearch](pattern_search_example.py) | Direct Search | Noisy functions | Systematic patterns |
+| [PowellsMethod](powells_method_example.py) | Coordinate Descent | Smooth functions | Conjugate directions |
+| [DownhillSimplexOptimizer](downhill_simplex_example.py) | Simplex | Low-dimensional | Nelder-Mead method |
+| [DirectAlgorithm](direct_algorithm_example.py) | Global | Lipschitz functions | Space division |
+| [LipschitzOptimizer](lipschitz_optimizer_example.py) | Global | Smooth functions | Theoretical guarantees |
+| [SpiralOptimization](spiral_optimization_example.py) | Nature-inspired | Mixed problems | Spiral search patterns |
+| [ParticleSwarmOptimizer](particle_swarm_example.py) | Swarm Intelligence | Continuous problems | Population-based |
+| [GeneticAlgorithm](genetic_algorithm_example.py) | Evolutionary | Multi-modal problems | Selection/crossover/mutation |
+| [EvolutionStrategy](evolution_strategy_example.py) | Evolutionary | Self-adaptive | Strategy parameter evolution |
+| [DifferentialEvolution](differential_evolution_example.py) | Evolutionary | Continuous problems | Vector differences |
+| [BayesianOptimizer](bayesian_optimization_example.py) | Probabilistic | Expensive evaluations | Gaussian Process |
+| [TreeStructuredParzenEstimators](tree_structured_parzen_estimators_example.py) | Bayesian | Mixed parameters | TPE algorithm |
+| [ForestOptimizer](forest_optimizer_example.py) | Surrogate-based | Non-linear problems | Random Forest surrogates |
+
+## Algorithm Categories
+
+### Simple Algorithms
+- **RandomSearch**: Pure random sampling - excellent baseline
+- **GridSearch**: Exhaustive enumeration of discrete grids
+
+### Local Search Methods
+- **HillClimbing**: Greedy local search from starting points
+- **SimulatedAnnealing**: Hill climbing with probabilistic escapes
+- **StochasticHillClimbing**: Hill climbing with random moves
+
+### Population-Based Methods
+- **ParticleSwarmOptimizer**: Swarm intelligence optimization
+- **GeneticAlgorithm**: Evolutionary computation approach
+- **DifferentialEvolution**: Evolution strategy for continuous spaces
+
+### Advanced Methods
+- **BayesianOptimizer**: Gaussian Process-based optimization
+- **TreeStructuredParzenEstimators**: TPE algorithm (similar to Optuna's TPE)
+- **ForestOptimizer**: Random forest-based surrogate optimization
+
+## Choosing the Right Algorithm
+
+### Quick Decision Tree
+
+1. **Need a baseline?** → RandomSearch
+2. **Small discrete space?** → GridSearch
+3. **Expensive evaluations?** → BayesianOptimizer
+4. **Continuous optimization?** → ParticleSwarmOptimizer or BayesianOptimizer
+5. **Fast local optimization?** → HillClimbing
+6. **General purpose?** → RandomSearch or BayesianOptimizer
+
+### Problem Characteristics
+
+**Function Properties:**
+- Smooth, continuous → BayesianOptimizer, ParticleSwarmOptimizer
+- Noisy, discontinuous → RandomSearch, GeneticAlgorithm
+- Multi-modal → ParticleSwarmOptimizer, GeneticAlgorithm
+
+**Search Space:**
+- Low dimensional (< 10) → GridSearch, BayesianOptimizer
+- High dimensional (> 20) → RandomSearch, GeneticAlgorithm
+- Mixed types → RandomSearch, TreeStructuredParzenEstimators
+
+**Computational Budget:**
+- Low budget (< 50 trials) → RandomSearch, HillClimbing
+- Medium budget (50-200) → BayesianOptimizer, ParticleSwarmOptimizer
+- High budget (200+) → GridSearch, GeneticAlgorithm
+
+## Common Configuration
+
+All GFO algorithms support:
+- **Random state**: `random_state=42` for reproducibility
+- **Early stopping**: `early_stopping=10` trials without improvement
+- **Max score**: `max_score=0.99` stop when target reached
+- **Warm start**: `initialize={"warm_start": [points]}` initial solutions
+
+## Advanced Usage
+
+### Sequential Optimization
+```python
+# Phase 1: Global exploration
+random_opt = RandomSearch(n_trials=20, ...)
+initial_results = random_opt.solve()
+
+# Phase 2: Local refinement
+hill_opt = HillClimbing(
+    n_trials=30,
+    initialize={"warm_start": [initial_results]}
+)
+final_results = hill_opt.solve()
+```
+
+### Parameter Space Design
+
+**Continuous Spaces:**
+```python
+param_space = {
+    "learning_rate": (0.001, 0.1),      # Log scale recommended
+    "n_estimators": (10, 1000),         # Integer range
+    "regularization": (0.0, 1.0),       # Bounded continuous
+}
+```
+
+**Discrete/Categorical:**
+```python
+param_space = {
+    "algorithm": ["adam", "sgd", "rmsprop"],  # Categorical
+    "layers": [1, 2, 3, 4, 5],               # Discrete integers
+    "activation": ["relu", "tanh", "sigmoid"], # Categorical
+}
+```
+
+## Performance Tips
+
+1. **Start simple**: RandomSearch is often surprisingly competitive
+2. **Use warm starts**: Provide good initial points when available
+3. **Set early stopping**: Avoid wasted evaluations on poor runs
+4. **Match algorithm to problem**: Consider function smoothness and dimensionality
+5. **Reproducibility**: Always set `random_state` for consistent results
+
+## Further Reading
+
+- [Gradient-Free Optimization Overview](https://en.wikipedia.org/wiki/Derivative-free_optimization)
+- [Bayesian Optimization Tutorial](https://arxiv.org/abs/1807.02811)
+- [Evolutionary Algorithms Survey](https://ieeexplore.ieee.org/document/6900297)
+- [Hyperparameter Optimization Review](https://arxiv.org/abs/1502.02127)

examples/gfo/bayesian_optimization_example.py

@@ -0,0 +1,75 @@
+"""
+Bayesian Optimization Example - Gaussian Process-based Optimization
+
+Bayesian optimization uses a probabilistic model (typically Gaussian Process) to
+model the objective function and an acquisition function to decide where to sample
+next. This approach is highly sample-efficient and particularly useful when
+function evaluations are expensive.
+
+Characteristics:
+- Sample-efficient optimization
+- Uses Gaussian Process to model objective function
+- Balances exploration vs exploitation via acquisition functions
+- Excellent for expensive function evaluations
+- Handles uncertainty quantification naturally
+"""
+
+import numpy as np
+from sklearn.datasets import load_wine
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import cross_val_score
+
+from hyperactive.experiment.integrations import SklearnCvExperiment
+from hyperactive.opt.gfo import BayesianOptimizer
+
+# Load dataset
+X, y = load_wine(return_X_y=True)
+print(f"Dataset: Wine classification ({X.shape[0]} samples, {X.shape[1]} features)")
+
+# Create experiment
+estimator = RandomForestClassifier(random_state=42)
+experiment = SklearnCvExperiment(estimator=estimator, X=X, y=y, cv=3)
+
+# Define search space - discrete values for Bayesian optimization
+search_space = {
+    "n_estimators": list(range(10, 201, 10)),  # Discrete integer values (step 10)
+    "max_depth": list(range(1, 21)),  # Discrete integer values
+    "min_samples_split": list(range(2, 21)),  # Discrete integer values
+    "min_samples_leaf": list(range(1, 11)),  # Discrete integer values
+}
+
+# Configure Bayesian Optimization
+# Provide some initial good points to help the GP model initialization
+warm_start_points = [
+    {
+        "n_estimators": 100,
+        "max_depth": 10,
+        "min_samples_split": 5,
+        "min_samples_leaf": 2,
+    },
+    {
+        "n_estimators": 50,
+        "max_depth": 15,
+        "min_samples_split": 3,
+        "min_samples_leaf": 1,
+    },
+]
+
+optimizer = BayesianOptimizer(
+    search_space=search_space,
+    n_iter=15,
+    random_state=42,
+    initialize={"warm_start": warm_start_points},
+    experiment=experiment,
+)
+
+# Run optimization
+# Bayesian optimization builds a GP model of the objective function
+# and uses acquisition functions (like Expected Improvement) to select
+# the most promising points to evaluate next
+best_params = optimizer.solve()
+
+# Results
+print("\n=== Results ===")
+print(f"Best parameters: {best_params}")
+print("Bayesian optimization completed successfully")

examples/gfo/differential_evolution_example.py

@@ -0,0 +1,59 @@
+"""
+Differential Evolution Example - Vector-Based Evolutionary Algorithm
+
+Differential Evolution is a population-based evolutionary algorithm that uses
+difference vectors between population members to generate new candidate solutions.
+It's particularly effective for continuous optimization problems and has robust
+convergence properties across various problem types.
+
+Characteristics:
+- Population-based with vector difference operations
+- Self-adapting through mutation and crossover
+- Robust convergence across different problem types
+- Effective for continuous and mixed optimization
+- Simple parameter control with good default behavior
+"""
+
+import numpy as np
+from sklearn.datasets import load_wine
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import cross_val_score
+
+from hyperactive.experiment.integrations import SklearnCvExperiment
+from hyperactive.opt.gfo import DifferentialEvolution
+
+# Load dataset
+X, y = load_wine(return_X_y=True)
+print(f"Dataset: Wine classification ({X.shape[0]} samples, {X.shape[1]} features)")
+
+# Create experiment
+estimator = RandomForestClassifier(random_state=42)
+experiment = SklearnCvExperiment(estimator=estimator, X=X, y=y, cv=3)
+
+# Define search space
+search_space = {
+    "n_estimators": list(range(10, 201, 5)),     # Discrete integer values (broader range)
+    "max_depth": list(range(1, 21)),             # Discrete integer values
+    "min_samples_split": list(range(2, 21)),     # Discrete integer values
+    "min_samples_leaf": list(range(1, 11)),      # Discrete integer values
+}
+
+# Configure Differential Evolution
+optimizer = DifferentialEvolution(
+    search_space=search_space,
+    n_iter=50,
+    random_state=42,
+    experiment=experiment
+)
+
+# Run optimization
+# Differential evolution uses difference vectors between population members
+# For each target vector, it creates a mutant vector using weighted differences
+# from other population members, then applies crossover to generate trial vector
+# The trial vector replaces the target if it has better fitness
+best_params = optimizer.solve()
+
+# Results
+print("\n=== Results ===")
+print(f"Best parameters: {best_params}")
+print("Differential evolution optimization completed successfully")