BeamCLIP: Multimodal Alignment For Mmwave Beam Pattern Learning

Beam-Clip: Multimodal Alignment For Mmwave Beam Pattern Learning

This is the official codebase for the BeamCLIP models library, providing tools for multimodal alignment and mmWave beam pattern learning.

Pre-training Beam-CLIP

Beam-CLIP Retrieval Approach

---

Dataset and Modalities

BeamCLIP handles multiple varying modalities to align spatial and contextual data into a shared embedding space.

mmWave Raw Polar Plot

The target prediction labels are raw mmWave beam patterns showcasing multi-path propagation and diverse signal lobes.

---

Project Structure

beamclip
├── src
├── pyproject.toml
└── README.md

Installation

1. Clone the Repository

First, clone the repository to your local machine:

git clone https://github.com/Ommos92/BeamCLIP.git
cd BeamCLIP

2. Set Up a Virtual Environment (Optional but Recommended)

Create and activate a virtual environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install Dependencies

Install the required dependencies listed in the pyproject.toml:

# Install core library
pip install .

# Install with training dependencies
pip install ".[trainer]"

# Install in edit mode for development
pip install -e .
pip install -e ".[trainer]"

Downloading the Dataset

BeamCLIP relies on raw multimodal sensor data collected from testbeds. You can automatically download and extract sample scenarios using the provided bash script:

chmod +x download_scenarios.sh
./download_scenarios.sh

By default, the script pulls from the links provided in download_scenarios.sh. You can edit this file to comment out or uncomment specific scenario zip files based on your needs. Data will be automatically extracted into the data/ directory.

Training Your Own Beam-CLIP Model

To train your own Beam-CLIP model, you can use the pre-training script provided in the library. Ensure you have installed the training dependencies:

pip install -e ".[trainer]"

Run the pre-training script with your specified data scenarios:

python src/beamclip/train/pretrain_beam_clip.py \
    --scenario-paths data/Scenario32 data/Scenario33 \
    --batch-size 32 \
    --num-epochs 50 \
    --output-dir results/my_model

You can customize the training process with various arguments such as --learning-rate, --contrastive-weight, and using --wandb for Weights & Biases logging. Run python src/beamclip/train/pretrain_beam_clip.py --help for a full list of options.

Evaluation Metrics

BeamCLIP tracks a comprehensive suite of metrics designed to evaluate both the performance of beam power predictions and the cross-modal alignment across sensors.

Beam Metrics

These metrics measure the precision, spatial consistency, and realistic usability of the simulated beam prediction.

• Accuracy & Ranking: Top-K Accuracy, Distance-based Accuracy (DBA), Top-K1/K2, Normalized Discounted Cumulative Gain (nDCG), and Power Capture Ratio (PCR).
• Basic Error Metrics: MAE, MSE, RMSE, and Cosine Similarity.
• Power & Spread: Power Ratio (PR), Beam Spread (spread error/ratio, coverage error), Peak Characteristics, and Power Distribution Quality (concentration, KL divergence, efficiency, balance).
• Practical & Robustness: Beam Pattern Similarity (correlation, overlap, angular similarity), Practical Beam Selection (accuracy within specific tolerances), and Robustness Metrics (stability, worst-case accuracy, outlier resistance).

CLIP Metrics

These metrics measure the quality of alignment across the different modalities (e.g., LiDAR, GPS, Camera, Radar, mmWave).

• Modality Alignment: Cross-Modal Similarity, Centered Kernel Alignment (CKA), and Singular Vector Canonical Correlation Analysis (SVCCA).
• Retrieval Performance: Top-K Accuracy, Recall@K, and Mean Average Precision (mAP) for generalized feature retrieval.
• Global Alignment: Global Correlation, Global Distance, and Alignment Consistency.
• Visualizations: Generating feature correlation heatmaps, PCA, and t-SNE clustering projections.

Contributing

Contributions are welcome! If you have any ideas, suggestions, or bug reports, please open an issue or submit a pull request.

License

This project is licensed under the Apache License.

Citation

If you use this codebase in your research, please cite:

@article{el2025beam,
  title={Beam-Clip: Multimodal Alignment For mmWave Beam Pattern Learning},
  author={El Kommos, Andrew P},
  year={2025}
}