Skip to content

Code for importance-weighted domain alignment, and the paper “Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations”.

License

Notifications You must be signed in to change notification settings

rxian/domain-alignment

Repository files navigation

Importance-Weighted Domain Alignment

Modules for performing class-importance-weighted domain alignment (IWDA) in PyTorch for unsupervised domain adaptation, as well as on text and token classifiers built upon pre-trained language models using the 🤗 Transformers library. The alignment is implemented using adversarial training with Wasserstein-1 critic loss and zero-centered gradient penalty.

The modules are found in domain_alignment.py. They are W1CriticWithImWeights, W1CriticWithImWeightsEstimation, and ImWeightsEstimator. See run_text_cls.py, run_token_cls.py and below for example usage.

Example: Unsupervised Cross-Lingual Learning

One usage of IWDA is unsupervised cross-lingual transfer of pre-trained language models on downstream tasks, discussed and evaluated in our ICLR 2022 paper “Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations”.

The commands for the following unsupervised cross-lingual transfer tasks are provided in run_iclr2022.sh (also check package requirements.txt):

  • mBERT transfer from English to Spanish for named-entity recognition on CoNLL-2002 and 2003 datasets.
  • mBERT transfer from English to Japanese for sentiment analysis on Multilingual Amazon Reviews Corpus.

Results on CoNLL NER with mBERT (average of 5 runs):

Method de es nl
Zero-shot 69.77 74.14 78.28
IWDA 72.56 76.11 78.63
IWDA (oracle) 72.58 76.48 79.17

Results on MARC sentiment analysis with mBERT (average of 3 runs):

Method de es fr ja zh
Zero-shot 44.80 46.49 46.02 37.37 38.48
IWDA 51.94 49.77 49.78 42.62 44.04
IWDA (oracle) 51.95 50.83 50.01 44.91 45.96

References

The BibTeX entry for our paper is:

@inproceedings{xian2022crosslingual,
  title={Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations},
  author={Ruicheng Xian and Heng Ji and Han Zhao},
  year={2022},
  booktitle={International Conference on Learning Representations},
  url={https://openreview.net/forum?id=k7-s5HSSPE5}
}

Our implementation follows the following paper closely, where some code come from:

@inproceedings{tachetdescombes2020domainadaptation,
  title={Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift},
  author={Tachet des Combes, Remi and Zhao, Han and Wang, Yu-Xiang and Gordon, Geoff},
  year={2020},
  booktitle={Advances in Neural Information Processing Systems}
}

About

Code for importance-weighted domain alignment, and the paper “Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations”.

Topics

Resources

License

Stars

Watchers

Forks