SemanticRL

SemanticRL is designed to preserve the semantic information instead of strictly securing the bit-level precision. It enables a general-purpose, large-scale, wireless, and semantic communication framework.

Note that this is a research project and by definition is unstable. Please write to us if you find something not correct or strange. We are sharing the codes under the condition that reproducing full or part of codes must cite the paper.

@article{lu2021reinforcement,
  title={Reinforcement learning-powered semantic communication via semantic similarity},
  author={Lu, Kun and Li, Rongpeng and Chen, Xianfu and Zhao, Zhifeng and Zhang, Honggang},
  journal={arXiv preprint arXiv:2108.12121},
  url={https://arxiv.org/abs/2108.12121},
  year={2021}
}

Features

• A schematic shift from bit-precision to semantic consistency.
• Compatible with any (non-differenable) semantic similarity metric as the objective function.
• RL-based end-to-end optimization on non-differentiable and unknown wireless channel with high-dimensional action/semantic space.

Requirements

pip install -r requirements.txt

Dataset Preparation

cd $your_data_root
wget https://www.statmt.org/europarl/v7/fr-en.tgz
tar -zxvf fr-en.tgz    
python preprocess_captions.py --data_root $your_data_root

Training

Training SemanticRL-JSCC

# AWGN-CE-Stage1
python Trainng_SemanticRL.py --training_config ./config/config_AWGN_CE.yaml --dataset_path $your_data_root

# AWGN-CE-Stage2
python Trainng_SemanticRL.py --training_config ./config/config_AWGN_CE_Stage2.yaml --dataset_path $your_data_root

# AWGN-RL-Stage2
python Trainng_SemanticRL.py --training_config ./config/config_AWGN_RL.yaml --dataset_path $your_data_root

You can change the type of random channel to trian and test in different scenarios. For more details, run python Trainng_SemanticRL.py --help.

Training SemanticRL-SCSIU

python Trainng_SemanticRL.py --training_config ./config/config_AWGN_RL_SCSIU.yaml --dataset_path $your_data_root

Inference

Download the pretrained model. Place them into the root directory.

Baidu Netdisk extraction code: fp3t

SemanticRL
├── ckpt_AWGN_CE_Stage2
│   └── all checkpoints
├── ckpt_AWGN_RL 			  (the second version, see arXiv. trained with shuffle=True)
├── ckpt_AWGN_RL_SemanticRLv1 (the first version, see arXiv. trained with shuffle=False)
├── Evaluation
│   └── Inference_Given_Input.py
│   └── Run_Inference_Checkpoints.py
│   └── ...
├── Trainng_SemanticRL.py
├── ...

Reproducing Quantitative Results

# Step1. load checkpoint and run inference on test set. 
# Output dir: ./Evaluation/InferenceResutls/$EXP_NAME/xxx.json (output tokens inside)
python Evaluation/Run_Inference_Checkpoints.py --path $CKPT_PATH --name $EXP_NAME --data_root $your_data_root
# Step2. calculate metrics like BLEU, CIDER etc.
# Output dir: ./Evaluation/EvalResults/$EXP_NAME/xxx.json (scores inside)
python Evaluation/CaptionMetrics-master/Eval_Metric_Checkpoints.py --path Evaluation/InferenceResutls/$EXP_NAME --name $EXP_NAME

Reproducing Visual Results

# Output dir: std output (i.e., your screen) (sentences of type str)
python Evaluation/Inference_Given_Input.py

Your trained model may behave a little different from ours, but they should be similar.

Integrating SemanticRL with your own framework

Besides LSTM backbone, we provide a Transformer backbone to facilitate further researches. You can rewrite methods in model.py to customize your own framework. SemanticRL is model-agnostic. You may also design any semantic similarity metric to build a customed communication system.

Thanks

This repository is largely inspired by ruotianluo's excellent captioning work.