11.0.1

Version 11.0.1

New features

Text analysis performance:

• Parallelization of token collection for text analysis: significant speedup on large text datasets

New Khiops Desktop Visualization Application:

• a single application that replaces the previous Khiops Visualization and Covisualization applications

Fixes

• Fix optimization issue with ngram tokenization on very large text datasets (100,000+ ngrams)
• Fix trailing quote inconsistency between Windows and UNIX for very long quoted strings
• Fix DLL loading error messages encoding and improve diagnostics on Windows
• Improve DLL dependency search using LoadLibraryExW on Windows
• Fix misconfiguration of KHIOPS_DRIVERS_PATH
• Activate Raw GUI mode when external drivers are installed
• Fields containing double-quotes in the middle now raise proper warnings
• Add warning for instances x variables coclustering with less than two variables

Distribution

• New khiops-core pip package published on PyPI
• New khiops-kni pip package for the Khiops Native Interface, including Python bindings and tutorial examples
• Ubuntu 26.04 support added

11.0.1-rc.2

This release is for testing purposes only and there is no support for it.
Go to https://khiops.org to install the latest supported version.

11.0.0

Version 11.0.0

New major version

Khiops 11 - what's new

Text data:

• new Text type for variables in tabular or multi-table schema
• Automatic feature construction from Text variables

SNB classifier for sparse data: extension to sparse data

Random forests for regression

Khiops interpretation and reinforcement:

• Instance-based interpretation of scores
• Exact computation of Shapley values
• Importance of SNB selected variables is now computing using their mean absolute Shapley value
• Build an interpretation dictionary, to deploy interpretation values
• Build a reinforcement dictionary, to deploy reinforcement scores based on lever variables

Histograms: Optimal histograms for univariate data exploration

Coclustering instances x variables: extension of existing variable x variable coclustering
for joint density estimation, to instances x variables coclustering for exploratory analysis.

Visualization tools:

• visualization: new panel to visualize histograms
• covisualization: accounting for the case of instances x variables coclustering

Simplified ergonomy

• simplification of panels and fields, everywhere, as much as possible
• fast path: to train a model without a dictionary
• visualization of reports and editing of dictionaries from the graphical interface

Extended scenario-based management of Khiops, with control structures and a parameter file in JSON format

Detailed evolutions

Text data

• new type Text available in Khiops dictionaries
  • Text variables can contain up to 1,000,000 bytes
  • Categorical variables are now limited to 1,000 bytes
• type detected in automatic "build dictionary" feature
• automatic feature construction
  • parameter "number of text features ", with default value 10,000
  • text features:
    • words: default automatic tokenization
    • ngrams: black-box using ngrams of bytes, for blob-like variables
    • tokens: open to user defined tokenization
• new derivations rules for Text variables
  • TextLoadFile: load a Text variable from a text file, up to 1,000,000 chars, replacing end of lines by whitespaces
  • FromText, ToText: conversion with categorial variables
  • rules similar to those related to categorical variables:
    • TextLength, TextLeft, TextRight, TextMiddle,
    • TextTokenLength, TextTokenLeft, TextTokenRight, TextTokenMiddle,
    • TextTranslate, TextSearch, TextReplace, TextReplaceAll
    • TextRegexMatch, TextRegexSearch, , TextRegexReplace, TextRegexReplaceAll
    • TextToUpper, TextToLower,
    • TextConcat, TextHash, TextEncrypt
    • GetText(Entity, Text)
• new type TextList: list of Text variables, to avoid scalability problems when concatenating Text variables from a corpus
  • dedicated derivation rules
    • creation: TextList(text1, text2, ...), TextListConcat(textList1, textList2, ...)
    • Inspection: TextListSize, TextListAt
    • extract from sub-tables: GetTextList, TableAllTexts, TableAllTextLists

Optimal histograms

• by default in unsupervised learning (without target variable), the new MODL preprocessing methods are activated
  • numerical variables: optimal histogram are built to for accurate density estimation and usefull exploratory analysis
  • categorical variable: optimal number of frequent value are kept, with the rare values in a default group
• former unsupervised preprocessing methods can still be used if specified

Random forest for classification

• now produced even in case of grouped target values

Preprocessing

• in supervised learning, MODL is now the only available method
  • all other alternative methods are removed
• max part number is now the only constraint that can be specified
  • it is an "universal" constraint that applies to all preprocessing methods:
    discretization/grouping, supervised/unsupervised, univariate/bivariate

Timestamp year extension

• the maximum year has been extended from 4000 to 9999 to improve automatic type recognition when year 9999 is used in databases

Visualization: see the Khiops visualization and coviszualisation release notes

Khiops reports files .khj

• "variable statistics"
  • new field "parts" in the case of unsupervised learning
  • field "missingNumber" is now also available for catageorical variables
  • new field "sparseMissingNumber" to count the number of present values in sparse data blocks (technical field, not visualized)
• "variablesDetailedStatistics"
  • new sub-section "modlHistograms" in the case of unsupervised learning with MODL optimal histigram for numerical variables
    • "histogramNumber": number of available histograms, sorted by increasing granularities
    • "intrepretableHistogramNumber": number of interpretable histogrammes (potentaiily one histogram less)
    • "truncationEpsilon": truncation epsilon used by the TMH (Truncation Management Heuristic) (0 if no truncation detected in data)
    • "removedSingularIntervalNumber": number of singular intervals removed from the finest histogram to obtain the first interpretable histogram
    • "granularities", "intervalNumbers", "peakIntervalNumbers", "spikeIntervalNumbers",
      "emptyIntervalNumbers", "levels", "informationRates": vector of histogram properties
    • "histograms": array of histograms, each with "bounds" and "frequencies" vectors
• "modelingReport/trainedPredictorsDetails/selectedVariables"
  • selected variable related to pairs now two additional fields "name1" and "name2"
• "treeDetails"
  • new sub-section "targetPartition" to summarize the target partitioon per tree in cas of regression trees

Khiops coclustering reports files .khcj

• extended to support instances x variables coclustering
• support for deprecated format .khc has been removed

Simplification of data paths for multi-table schema

• each data path refers to a Table or Entity variable and identifies a data file
• main table: empty data path
• star schema: data paths are table or entity variable names for secondary tables
• snowflake schema: data paths are lists of variable names separated by '/'
• external tables: start with a data root prefixed with '/', referring to the referenced Root dictionary

Simplified Khiops ergonomics: main evolutions

• fast path: analyze a database without specifying a dictionary, just fill in the "Database File" and click "Train Model"
  to detect format, build dictionary, and train automatically
• dictionary management:
  • removed "Data dictionary" pane
  • extended "Data dictionary" menu with "Dictionary management" (opens a dialog similar to the previous pane)
  • new button "Edit dictionary file" to open the dictionary in a text editor
• "Train database" pane: simplified layout with sub-panes for "Sampling" and "Selection"
• "Help" menu: added "Quick start" sub-menu
• "Parameters" pane:
  • sub-pane "Predictors/Feature engineering"
    • new field "Keep selected variables only"
    • new field "Max number of text features" (default: 10,000)
    • field "Max number of constructed variables" (new default: 1,000)
  • sub-pane "Predictors/Advanced predictor parameters"
    • new field "Do data preparation only" (removes fields like "Selective Naive Bayes", "Baseline Predictor", etc.)
    • new button "Text feature parameters" to choose among 'words', 'ngrams', 'tokens'
  • sub-pane "Preprocessing":
    • removed "Discretization" and "Value grouping" sub-panes (all supervised preprocessing now uses MODL)
    • new button "Advanced unsupervised parameters" for standard choices
    • new field "Max part number": universal constraint on all preprocessing steps
• "Results" pane:
  • now only two fields:
    • "Analysis report" (replaces previous "Results files directory" and "Result files prefix")
    • "Short Description"
  • two buttons:
    • "Export as xls" (replaces previous report fields)
    • "Visualize report" (opens visualization directly)
• "Tool" Menu: new sub-menus "Interpret model" and "Reinforce model" for new features

Simplified Khiops coclustering ergonomics: Main Evolutions

• similar simplifications as Khiops tools
• new menu items under "Tools" to enable Khiops coclustering to be used independently of Khiops
  • "Check database", "Sort data table by key", "Deploy model"
• new option in "Parameters" pane:
  • "Coclustering Type": choose between "Variable coclustering" and "Instances x Variables coclustering"

Khiops dictionaries

• Comments are now allowed alongside labels for entities, variables, and variable blocks, using '//' as a prefix
  • Dictionary: comment lines before declaration, internal comments before '}', label being the first line
    before the comment acting as a title
  • Variable: comments before declaration, label at end of line
  • Variable block: comments before start '{', internal comments before '}', label at end of block
• Impact on .kdicj files:
  • Optional fields "label" as string, "comments" and "internalComments" as lists of strings

Integration improvements

• extended scenario-based management with control structures and JSON parameter files
• new environment variable KHIOPS_API_MODE for better API integration:
  • Default: no behavior change, result files stored relative to input database, suffixes imposed as needed
  • If set to true (e.g., in khiops python), result file names are used as-is

Samples:

• new sample 'NegativeAirlineTweets', small text classification problem, with one single text variable
• new sample 'WineReviews', regression probleme, with 13 variables, including three text variables

Many minor corrections and improvements

11.0.0-rc.2

This release is for testing purposes only and there is no support for it.
Go to https://khiops.org to install the latest supported version.

10.3.2

Bug fix of the sort algorithm: The singleton chunks were not always concatenated.
Other minor improvements.

10.3.1

• Bug fix of the sort algorithm: The singleton chunks don't need to be sorted, but if the input and output file
  separators are different, they need to be rewritten to change the file separator.
• Minor improvement of the Java GUI (better handling of the value changes for spinner widgets).
• Other minor improvements and bug fixes.

10.3.0

Linux packages have been improved.
Better integration of the file drivers (currently S3 and GCS).
Streamlined distributed computing support integration.

10.2.4

Bug fix:

• Fix MPI executable detection in "Conda-based" environments on Linux and MacOS.

Minor improvement:

• Reduce launching time of the GUI on Windows.

10.2.3

Khiops is now available for Ubuntu 24.04.

The GUI now uses all available processor cores if instructed to do so.

New version of visualization and covisualization tools, with improvements and bug fixes.

Internal improvements linked to packaging, in particular with the systematic use of the 'khiops_env' script.

Other minor improvements and bug fixes.

10.2.2

On Windows, the Khiops installation program now comes with the open-source Java JRE JustJ instead of Oracle's JDK.
Khiops is now available for debian 11 and 12: the library used to parallelize Khiops is now OpenMPI. MPICH remains for conda packages only.

New version of visualization and covisualization tools, with many improvements and bug fixes.

Minor improvements:

• new '-s' flag to display system information (useful for debugging).
• change in exit status: return 0 on success, 1 on fatal error (if there is an error or warning in the log file, the exit code is 0).

Other minor improvements and bug fixes.