docs(KDP): smart processing for custom pipelines

Author: Gandalfdore

docs/distribution_aware_encoder.md

@@ -1,7 +1,7 @@
 # Distribution-Aware Encoder
 
 ## Overview
-The Distribution-Aware Encoder is an advanced preprocessing layer that automatically detects and handles various types of data distributions. It uses TensorFlow Probability (tfp) for accurate modeling and applies specialized transformations while preserving the statistical properties of the data.
+The **Distribution-Aware Encoder** is an advanced preprocessing layer that automatically detects and handles various types of data distributions. It leverages TensorFlow Probability (tfp) for accurate modeling and applies specialized transformations while preserving the statistical properties of the data.
 
 ## Features
 
@@ -80,7 +80,7 @@ The Distribution-Aware Encoder is an advanced preprocessing layer that automatic
 
 ### Basic Usage
 
-The capability only works with numerical features!
+The Distribution-Aware Encoder works seamlessly (and only) with numerical features. Enable it by setting `use_distribution_aware=True` in the `PreprocessingModel`.
 
 ```python
 from kdp.processor import PreprocessingModel

docs/features.md

@@ -13,7 +13,7 @@ Explore various methods to define numerical features tailored to your needs:
         "feat1": "float",
         "feat2": "FLOAT",
         "feat3": "FLOAT_NORMALIZED",
-        "feat3": "FLOAT_RESCALED",
+        "feat4": "FLOAT_RESCALED",
         ...
     }
     ```
@@ -50,7 +50,7 @@ Explore various methods to define numerical features tailored to your needs:
         "feat3": NumericalFeature(
             name="feat3",
             feature_type=FeatureType.FLOAT_DISCRETIZED,
-            bin_boundaries=[(1, 10)],
+            bin_boundaries=[0.0, 1.0, 2.0],
         ),
         "feat4": NumericalFeature(
             name="feat4",
@@ -60,6 +60,10 @@ Explore various methods to define numerical features tailored to your needs:
     }
     ```
 
+### 📊 **Distribution-Aware Encoding**
+
+Enhance your numerical feature processing by leveraging the **Distribution-Aware Encoder**. This allows automatic or manual detection of data distributions, applying appropriate transformations to preserve the integrity and statistical properties of your data.
+
 Here's how the numeric preprocessing pipeline looks:
 
 ![Numeric Feature Pipeline](imgs/num_feature_pipeline.png)

docs/index.md

@@ -65,6 +65,8 @@ features_specs = {
 ppr = PreprocessingModel(
     path_data="data/my_data.csv",
     features_specs=features_spec,
+    use_distribution_aware=True,  # Enable Distribution-Aware Encoding
+    distribution_aware_bins=1000, # Set number of bins for finer data encoding
 )
 # construct the preprocessing pipelines
 ppr.build_preprocessor()

docs/kdp_overview.md

@@ -33,6 +33,17 @@ KDP (Keras Data Processor) is a powerful preprocessing library designed to strea
   - Dynamic feature filtering
   - Interpretable weights
 
+### 4. 📈 **Distribution-Aware Encoder**
+- **Automatic Distribution Detection**
+  - Identifies underlying data distributions (e.g., Normal, Heavy-Tailed, Multimodal, etc.)
+  - Applies specialized transformations to preserve statistical properties
+- **Adaptive Transformations**
+  - Learns optimal parameters during training
+  - Adjusts to data distribution changes dynamically
+- **Robust Handling**
+  - Manages sparse and periodic data effectively
+  - Ensures numerical stability across transformations
+
 ## 🏗️ Architecture Overview
 
 ```mermaid