RIS-MISO-DRL

DRL-based RIS Configuration in RIS-assisted MU-MISO mmWave Systems for Min-Max MSE Optimization under HWI of Phase errors, Phase-dependent amplitude response model, and Imperfect CSI

Paper

Rejected by IEEE VTC2024-Spring
Working paper on IEEE Wireless Communication Letters
Rejected by IEEE Globecom 2024
Working paper on IEEE Wireless Communications and Networking Conference

Note

I'll upload the code once I graduate or the paper gets accepted Never mind.
The code is a mess btw.

Installing

• Install Anaconda
• Import the environment

  conda env create --file sb3.yaml --name sb3
  

Meeting Notes

Spring 2024

• meeting 03/12 Current Progress
  • psi-to-MSE
• meeting 03/05 Current Progress
  • MSE-Matrix vs Signal-Tx
  • Nk-to-MSE
    • PPO-[3, 6, 8, 10]-16-16
  • Nt-to-MSE
    • PPO-2-[8, 16, 32, 64]-16
  • Ns-to-MSE
    • PPO-2-16-[16, 36, 64, 100]
  • beta_min-to-MSE
    • PPO-2-16-36
  • psi-to-MSE
    • PPO-2-16-16
• meeting 02/26 Current Progress
  • Bugs fixing
  • Validate self-identity
  • Ns-to-MSE
• meeting 02/20 Current Progress
  • Ns-to-MSE
• meeting 01/23 Current Progress
  • Nk-to-MSE
• meeting 01/09 Current Progress
  • Baseline method
    • Dominant Eigenvector Matching (DEM) heuristic for RIS Configuration
      • Performance: SDR > DEM > Power method
      • Speed: DEM > Power method > SDR
    • Max Ratio Transmission (MRT) for Precoder Design
  • Appendix
    • Validate MSE values with compute_raw_MSE()
  • Reference
    • N. K. Kundu and M. R. McKay, "RIS-Assisted MISO Communication: Optimal Beamformers and Performance Analysis," 2020 IEEE Globecom Workshops (GC Wkshps, Taipei, Taiwan, 2020, pp. 1-6. (Cited by 13)
    • S. Ragi, E. K. P. Chong and H. D. Mittelmann, "Polynomial-Time Methods to Solve Unimodular Quadratic Programs With Performance Guarantees," in IEEE Transactions on Aerospace and Electronic Systems, vol. 55, no. 5, pp. 2118-2127, Oct. 2019. (Cited by 6)
    • J. Gao, C. Zhong, X. Chen, H. Lin and Z. Zhang, "Unsupervised Learning for Passive Beamforming," in IEEE Communications Letters, vol. 24, no. 5, pp. 1052-1056, May 2020.
• meeting 01/02 Current Progress
  • Inference result: PPO-2-16-[4, 16, 36, 64, 100]
  • Confidence Interval: Random vs. Agent

Fall 2023

• meeting 12/19 Current Progress
  • PPO-2-16-[4, 9, 16, 25, 36, 64]
  • Comparison of different settings
• meeting 12/14 Current Progress
  • M. -A. Badiu and J. P. Coon, "Communication Through a Large Reflecting Surface With Phase Errors," in IEEE Wireless Communications Letters, vol. 9, no. 2, pp. 184-188, Feb. 2020.
  • R. Kozlica, S. Wegenkittl and S. Hiränder, "Deep Q-Learning versus Proximal Policy Optimization: Performance Comparison in a Material Sorting Task," 2023 IEEE 32nd International Symposium on Industrial Electronics (ISIE), Helsinki, Finland, 2023, pp. 1-6.
• meeting 12/13 Current Progress
  • PPO-2-16-[4, 36]
  • PPO-2-16-9
  • PPO-2-16-25
  • PPO-[2, 4, 6, 8, 10]-16-16
  • PPO-10-16-36
• meeting 12/05 Current Progress
  • System validation: Brute force check
    • Try every possible combination of actions
    • Plot the Sum-Rate for every possible actions
  • Update Max Ratio Transmission (MRT)
    • J. Gao, C. Zhong, X. Chen, H. Lin and Z. Zhang, "Unsupervised Learning for Passive Beamforming," in IEEE Communications Letters, vol. 24, no. 5, pp. 1052-1056, May 2020.
    • D. Tse and P. Viswanath, Fundamentals of Wireless Communication, Cambridge, U.K.:Cambridge Univ. Press, 2005.
  • Training results
  • Inference results
  • Plotting functions
  • Future works
    • Adding more neurons in each layer
    • Deepen the network architecture
    • PPO default network architecture is [64, 64] for both actor and critic networks
• meeting 11/28 Current Progress
  • New feature: seed_everything()
  • Bug fixing
  • Training results
    • PPO (1-4-4 to 4-4-4, and 4-16-16)
    • A2C (1-4-4 to 4-4-4)
  • Training of more complex settings with PPO (4-16-16)
  • Training of more episodes with PPO (1000 episodes)
  • Comparison of all continuous agents (TD3, DDPG, A2C, PPO, SAC)
• meeting 11/21 Current Progress
  • Training results
    • Scaling rewards doesn't actually work
  • Channel model
    • General Communication Systems
  • Problem formulations
    • Max-min downlink rate
    • Sum-Rate Maximization
  • Future works
    • Go back to Box discrete
• meeting 11/16 Summary
  • System model
    • Downlink RIS-aided MU-MISO System
  • Channel model
    • mmWave Systems
    • General Communication Systems
  • Steering vectors
    • ULA, UPA, USPA
    • Array response implementations in torch
  • Problem formulations
    • Min-max MSE
    • Max-min downlink rate
    • Sum-Rate Maximization
• meeting 11/14 Channel model - mmWave Systems
  • P. Wang, J. Fang, L. Dai and H. Li, "Joint Transceiver and Large Intelligent Surface Design for Massive MIMO mmWave Systems," in IEEE Transactions on Wireless Communications, vol. 20, no. 2, pp. 1052-1064, Feb. 2021. (Cited by 80)
  • K. Ying, Z. Gao, S. Lyu, Y. Wu, H. Wang and M. -S. Alouini, "GMD-Based Hybrid Beamforming for Large Reconfigurable Intelligent Surface Assisted Millimeter-Wave Massive MIMO," in IEEE Access, vol. 8, pp. 19530-19539, 2020. (Cited by 91)
• meeting 11/07 Steering vectors
  • K. Ying, Z. Gao, S. Lyu, Y. Wu, H. Wang and M. -S. Alouini, "GMD-Based Hybrid Beamforming for Large Reconfigurable Intelligent Surface Assisted Millimeter-Wave Massive MIMO," in IEEE Access, vol. 8, pp. 19530-19539, 2020. (Cited by 91)
  • J. Yuan, Y. -C. Liang, J. Joung, G. Feng and E. G. Larsson, "Intelligent Reflecting Surface-Assisted Cognitive Radio System," in IEEE Transactions on Communications, vol. 69, no. 1, pp. 675-687, Jan. 2021. (Cited by 130)
• meeting 10/31 Random action rewards
  • Random action rewards
  • TODO list
    • Inference
    • more anttenas do help
    • more bits don't actually help
• meeting 10/30 Current Progress
  • Training results
  • Inference results
• meeting 10/24 Current Progress
  • Bugs fixing
  • Training results
    • PPO, A2C DQN
    • Compare differenct models with their best performance
    • Compare different numbers of users
    • Compare the complexity of different settings
• meeting 10/17 Current Progress
  • True Discrete action space version
  • Normalize Box action space
  • Apply GPU acceleration
  • Learn and Save
  • Load and Predict
• meeting 10/03 Custom Gym Environment
  • Environment built
  • Able to train
  • Future works
• meeting 09/26 MU-MISO system model
  • System model
  • Problem formulation
  • MSE derivation
• meeting 09/14 MU-MIMO system model and possible methods
• meeting 09/12 MSE derivation
  • K. -Y. Chen, H. -Y. Chang, R. Y. Chang and W. -H. Chung, "Hybrid Beamforming in mmWave MIMO-OFDM Systems via Deep Unfolding," 2022 IEEE 95th Vehicular Technology Conference: (VTC2022-Spring), Helsinki, Finland, 2022, pp. 1-7.
  • X. Zhao, T. Lin, Y. Zhu and J. Zhang, "Partially-Connected Hybrid Beamforming for Spectral Efficiency Maximization via a Weighted MMSE Equivalence," in IEEE Transactions on Wireless Communications, vol. 20, no. 12, pp. 8218-8232, Dec. 2021.
• meeting 09/05 Paper reading
  • W. -Y. Chen, C. -Y. Wang, R. -H. Hwang, W. -T. Chen and S. -Y. Huang, "Impact of Hardware Impairment on the Joint Reconfigurable Intelligent Surface and Robust Transceiver Design in MU-MIMO System," in IEEE Transactions on Mobile Computing.
• meeting 08/29 Paper reading
  • C. Huang, R. Mo and C. Yuen, "Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning," in IEEE Journal on Selected Areas in Communications, vol. 38, no. 8, pp. 1839-1850, Aug. 2020. (Cited by 397)
• meeting 08/22 Paper reading
  • Saglam Baturay, Doga Gurgunoglu, and Suleyman S. Kozat. "Deep Reinforcement Learning Based Joint Downlink Beamforming and RIS Configuration in RIS-aided MU-MISO Systems Under Hardware Impairments and Imperfect CSI." arXiv preprint arXiv:2211.09702 (2022).
    • which was accepted to 2023 IEEE International Conference on Communications the 5th Workshop on Data Driven Intelligence for Networks and Systems (DDINS).