PRIM

Official repository of PRIM: Towards Practical In-Image Multilingual Machine Translation (EMNLP 2025 Main).

📂 Dataset

• PRIM Test Set:
• Training Set:

After downloading, you can check images in the training set with ./dataset/check_lmdb.ipynb

We also provide:

• iimt30k_lmdb.py → Convert IIMT30k dataset into lmdb format, for training our model.
• check_lmdb_iimt30k.ipynb → Check images in IIMT30k lmdb format.

🚀 Training

VisTrans training involves two stages. Training logs are recorded via Weights & Biases.

Stage 1️⃣ Vision

• Configs & scripts are in ./vision
• Modify configs

Note

train_lmdb_path, val_lmdb_path: the list of paths to training and validation lmdb;
spm_path: the sentencepiece model. For our data, please use ./multilingual_spm/total-35kbpe.model; for IIMT30k, please use ./iimt30k_spm/total-10kbpe.model;
src_language_tags, tgt_language_tags: the languages corresponding to train_lmdb_path and val_lmdb_path;
save_checkpoint_dir: path to save checkpoints.

Assign the --config, and run the training script:

cd ./vision/script
bash wandb_train.sh

After the training is finished, run inference to obtained the tokenized code sequence of images in training and validation sets:

cd ./vision/script
bash infer_code.sh

After that, you can get several files that stores code sequences, for example:

# train.de
# 7500 7500 7500 7500 7500 2209 6536 ...... 2209 7500 7500 7500 7500 7500
# ......

Stage 2️⃣ Translation

• Configs & scripts are in ./translation
• Modify configs

Note

train_code_file_path, val_code_file_path: the code file inferenced by infer_code.sh in stage 1. Make sure the vision_config and vision_ckpt are the same as in infer_code.sh;
subspm_path: the sentencepiece model for gererating char representation. For our data, please use ./multilingual_spm/total-char.model; for IIMT30k, please use ./iimt30k_spm/total-char.model.

Assign the --config, and run the training script:

cd ./translation/script
bash wandb_train.sh

🔍 Inference

We have released our trained model at

We provide ready-to-use scripts:

• PRIM test set: ./infer_prim.sh
• IIMT30k: ./infer_iimt30k.sh

Note

Set correct paths for config and checkpoint (Stage 2). The "vision_config" and "vision_ckpt" will be loaded automatically in translation config, which means you do not need to specify them in the inference bash scripts, so please verify that they are correct in the config trans.json.

📊 Evaluation

Evaluation consists of:

• Textual quality → BLEU, COMET (via ocr.py)
• Visual quality → FID

The example of evaluation commands are shown in eval_prim.sh and eval_iimt30k.sh.

Make sure to update paths before running.

📖 Citation

If you find our work helpful, we would greatly appreciate it if you could cite our paper:

@inproceedings{tian-etal-2025-prim,
    title = "{PRIM}: Towards Practical In-Image Multilingual Machine Translation",
    author = "Tian, Yanzhi  and
      Liu, Zeming  and
      Liu, Zhengyang  and
      Feng, Chong  and
      Li, Xin  and
      Huang, Heyan  and
      Guo, Yuhang",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.691/",
    pages = "13693--13708",
    ISBN = "979-8-89176-332-6",
    abstract = "In-Image Machine Translation (IIMT) aims to translate images containing texts from one language to another. Current research of end-to-end IIMT mainly conducts on synthetic data, with simple background, single font, fixed text position, and bilingual translation, which can not fully reflect real world, causing a significant gap between the research and practical conditions. To facilitate research of IIMT in real-world scenarios, we explore Practical In-Image Multilingual Machine Translation (IIMMT). In order to convince the lack of publicly available data, we annotate the PRIM dataset, which contains real-world captured one-line text images with complex background, various fonts, diverse text positions, and supports multilingual translation directions. We propose an end-to-end model VisTrans to handle the challenge of practical conditions in PRIM, which processes visual text and background information in the image separately, ensuring the capability of multilingual translation while improving the visual quality. Experimental results indicate the VisTrans achieves a better translation quality and visual effect compared to other models. The code and dataset are available at: https://github.com/BITHLP/PRIM."
}