MuSEvo

MuSEvo is a research prototype for assessing the robustness of multilingual summarization evaluation metrics using evolutionary adversarial Natural Language Processing (NLP) attacks.

The project accompanies the paper:

Bello-Orgaz, Gema, et al. “MuSEvo: Assessing the Robustness of Multilingual Summarization Metrics through Evolutionary Adversarial Natural Language Processing-based Attacks.” The 26th International Conference on Computational Science and Its Applications (ICCSA 2026), 2026.
Paper: https://kclpure.kcl.ac.uk/portal/files/377351968/ICCSA_MuSEVO-8.pdf

Overview

Automatic summarization metrics are widely used to compare generated summaries against reference summaries. However, metric scores can be sensitive to small textual perturbations, especially in multilingual settings. MuSEvo explores this issue by generating adversarial variants of summaries and analysing how evaluation metrics respond to those changes.

At a high level, MuSEvo:

• applies evolutionary search to generate adversarial text perturbations;
• evaluates the robustness of summarization metrics across languages;
• supports experiments for multilingual summarization evaluation;
• helps identify cases where metric scores change despite limited semantic change;
• provides tooling for reproducible adversarial evaluation experiments.

Usage

The exact entry points depend on the scripts included in the repository. A typical workflow is:

1. Prepare multilingual source documents, reference summaries, and system summaries.
2. Select the summarization metric or metrics to evaluate.
3. Configure the evolutionary adversarial attack.
4. Run MuSEvo to generate adversarial summary variants.
5. Analyse the robustness of the selected metrics.

Citation

If you use MuSEvo in your research, please cite:

@inproceedings{belloorgaz2026musevo,
  title     = {MuSEvo: Assessing the Robustness of Multilingual Summarization Metrics through Evolutionary Adversarial Natural Language Processing-based Attacks},
  author    = {Bello-Orgaz, Gema and others},
  booktitle = {The 26th International Conference on Computational Science and Its Applications},
  year      = {2026},
  url       = {https://kclpure.kcl.ac.uk/portal/files/377351968/ICCSA_MuSEVO-8.pdf}
}

Paper

• MuSEvo: Assessing the Robustness of Multilingual Summarization Metrics through Evolutionary Adversarial Natural Language Processing-based Attacks