Skip to content

uvavision/visual-pivoting

main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Visual-Pivot

[Findings of EMNLP 2020] Using Visual Feature Space as a Pivot Across Languages

Abstract

Our work aims to leverage visual feature space to pass information across languages. We show that models trained to generate textual captions in more than one language conditioned on an input image can leverage their jointly trained feature space during inference to pivot across languages. We particularly demonstrate improved quality on a caption generated from an input image, by leveraging a caption in a second language. More importantly, we demonstrate that even without conditioning on any visual input, the model demonstrates to have learned implicitly to perform to some extent machine translation from one language to another in their shared visual feature space. We show results in German-English, and Japanese-English language pairs that pave the way for using the visual world to learn a common representation for language.

Requirements

  • Python 3
  • Pytorch > 1.0

Data

Training: image captioning task for multi30k

CUDA_VISIBLE_DEVICES=[gpu id] python train.py --language2 DE --data_folder=[data folder] --data_name [your data name] --save_name [your saved model name]

Inference:

image + source language(DE or EN) -> target language(DE or EN):

CUDA_VISIBLE_DEVICES=[gpu id] python pivoting.py --lr 0.001 --layer conv13 --data_folder=[data folder] --decoder_mode="lstm" \
--checkpoint=[checkpoint] --data_name [your data name] --source [DE or EN] --target [DE or EN] --max_iter 20 --type normal 

source language(DE or EN) -> target language(DE or EN):

CUDA_VISIBLE_DEVICES=[gpu id] python pivoting.py --lr 0.01 --layer conv13 --data_folder=[data folder] --decoder_mode="lstm" \
--checkpoint=[checkpoint] --data_name [your data name] --source [DE or EN] --target [DE or EN] --max_iter 40 --type random

Kudos:

This project is developed based on feedback-prop and image Captioning, thanks!

About

[EMNLP 2020] Using Visual Feature Space as a Pivot Across Languages

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages