The univariate level is inconsistent when another variable is present.

[folmos-at-orange]

Description

When performing data preparation with Khiops, the level of variable depends on other variables being present. This is counter intuitive as the univariate data preparation should depend solely on the variable at hand.

I attach an example to reproduce this (inconsistent-level.zip). Simply execute the inconsistent_level.py script to obtain the results.

The example consist in a two-column data table with a numerical variable y_score and a categorical variable y. We perform a data preparation with target y with the vanilla dictionary

Dictionary y_score
{
  Numerical y_score;
  Categorical y;
};

which gives to y_score a level of 0.0588805. Then we perform data preparation with the following modified dictionary:

Dictionary y_score_with_copy
{
  Numerical y_score;
  Categorical y;
  Categorical y_copy = CopyC(y);
};

which gives y_score a level of 0.0533369. I also noticed the construction cost went from 0.693147 to 1.38629.

Questions/Ideas

• Is this a feature or a bug?
• When the level is low this may pass from having a discretization model to not having one.

Context

Khiops 11.0.0-b.0

Environment variables:
        KHIOPS_API_MODE true
        KHIOPS_DRIVERS_PATH     /opt/homebrew/Caskroom/miniforge/base/envs/khcalib-env/lib
Internal environment variables:
        None
System resources :
        Host number 1
        Physical cores on system 10
        Logical processes on system 10
        Available memory on system 16.0 GB (Logical 10.0 GB)
        Available disk space on system 193.2 GB
        hostname        MPI ranks       logical memory  disk    cores
        MC-JWMHQQ2JYL   0,1,2,3,4,5,6,7,...     10.0 GB 193.2 GB        10
System
system=Darwin
release=24.5.0
version=Darwin Kernel Version 24.5.0: Tue Apr 22 19:54:25 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6020

[marcboulle]

This behavior is intentional in the MODL approach to prevent overfitting.

Let us illustrate this with an example. Suppose that, for a given sample size, the data preparation of each descriptive variable is very robust, with a very low probability $p=\frac{1}{1000}$ of obtaining a strictly positive $Level$ for a random descriptive variable, that is, the probability of wrongly identifying a random variable as informative.
Then, if your dataset contains 1000 random descriptive variables, the probability of identifying at least one of them as informative approaches 1. More precisely, this probability is $1 - (1-\frac{1}{1000})^{1000}$ ~ $1 - \frac{1}{e}$ ~ 63%.

To avoid such overfitting, the MODL approach incorporates a construction cost in addition to the data preparation cost. See https://khiops.org/learn/autofeature_engineering/, section Optimization Criterion. When no feature is constructed in a dataset with $K$ native variables, this construction cost reduces to a selection cost, which represents the choice of a descriptive variable to preprocess among the $K$ variables. The probability of selecting any particular variable is $\frac{1}{K}$, corresponding to a coding length of $\log K$.

The approach summarized at https://khiops.org/learn/autofeature_engineering/ and detailed in the reference papers (https://khiops.org/references/) has since been improved by introducing the null model:

• The null model describes the target variable without using any descriptive variables.
  • It is essentially a multinomial model for the distribution of the $J$ target values in a dataset of size $N$.
  • Its coding length is $\log \binom{N + J - 1}{J - 1} + \log \binom{N}{N_1, N_2, \dots, N_J}$.
• An additional hyper-prior parameter is introduced for the preprocessing of each descriptive variable, alongside the construction cost and data preparation cost.
  • This parameter reflects the choice of being informative or not, with an equiprobability $p_{null} = \frac{1}{2}$, corresponding to a coding length of $c_{null} = \log 2$.
  • This extra prior enhances robustness, as the null cost is challenging to beat.
  • The null cost is independent of the preprocessing model, whether discretization or value grouping.
  • If a descriptive variable is chosen as informative, its preprocessing model is finely tuned by considering only models with at least two intervals or groups.

Finally, let's illustrate this with your sample:

• In the first case, with a single descriptive variable $K=1$, the construction cost is $c_{null} + \log K = \log 2 \approx 0.693147$.
• In the second case, with two descriptive variables $K=2$, the construction cost is $c_{null} + \log 2 = 2 \log 2 \approx 1.38629$.

[folmos-at-orange]

Thanks for the answer @marcboulle !