SNAG

In this work, we introduce a Unified Multi-Modal Knowledge Graph (MMKG) Representation Framework that incorporates tailored training objectives for Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Our approach achieves SOTA performance across a comprehensive suite of ten datasets, including three for MKGC and seven for MMEA, demonstrating the framework's effectiveness and versatility in diverse multi-modal contexts.

🔔 News

• 2024-02 We preprint our Survey Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [Repo].

🔬 Dependencies

pip install -r requirement.txt

Details

• Python (>= 3.7)
• PyTorch (>= 1.6.0)
• numpy (>= 1.19.2)
• Transformers (== 4.21.3)
• easydict (>= 1.10)
• unidecode (>= 1.3.6)
• tensorboard (>= 2.11.0)

🚀 MKGC Train

• Quick start: Using script file (run.sh)

>> cd SNAG_MKGC
>> bash run.sh

• Optional: Using the bash command:

# Command Details:
#  GPU | DATA | num_proj | use_intermediate | joint_way | noise ratio | mask ratio | noise-level | num_hidden_layers | num_attention_heads | Exp ID

# # DATA=DB15K / MKG-W / MKG-Y
# num_proj: 1 / 2
# use_intermediate: 0 / 1
# joint_way: "Mformer_hd_mean" / "Mformer_hd_graph" / "Mformer_weight" / "atten_weight" / "learnable_weight"
# noise ratio: 0 ~ 1
# mask ratio: 0 ~ 1
# noise-level: epoch / step

>> bash scripts/run_base.sh 0 DB15K 2 0 Mformer_hd_graph 0.2 0.7 epoch 1 2 K001
>> bash scripts/run_base.sh 0 MKG-Y 1 0 Mformer_hd_mean 0.2 0.7 epoch 1 2 Y001
>> bash scripts/run_base.sh 0 MKG-W 1 0 Mformer_hd_mean 0.2 0.7 epoch 1 2 W001

❗Tips: you can open the run.sh file for parameter or training target modification.

• Optional: Modifying the basic parameters:
  • you can open the scripts/run_base.sh file for parameter or training target modification
    • Make NOISE = 0 to abandon the gauss modality noise masking.
    • EPOCH can be set to 8000 as early stopping is employed by default.
    • The noise_level or noise_update parameter determines whether the Noise mask is updated at every step or every epoch. Through experimentation, we have found that updating at the epoch level is sufficient.
    • The use_pool flag indicates that pooling operations are applied to all pre-extracted visual/text features for dimensionality reduction and uniformity in dimensions.

EMB_DIM=128
NUM_BATCH=1024
MARGIN=12
LR=1e-4
LRG=1e-4
NEG_NUM=32
EPOCH=8000
NOISE=1
POOL=1

🚀 MMEA Train

• Quick start: Using script file (run.sh)

>> cd SNAG_MMEA
>> bash run.sh 0

• Optional: Using the bash command

# Command Details:
# bash file | GPU | Dataset | data split | R_{sa} | random seed | use_surface | R_{img} | noise ratio | mask ratio |
# Begin:
>> bash run_snag.sh 0 DBP15K ja_en 0.3 3408 0 1.0 0.2 0.7
>> bash run_snag.sh 0 DBP15K ja_en 0.3 3408 0 0.6 0.2 0.7
>> bash run_snag.sh 0 DBP15K ja_en 0.3 3408 0 0.4 0.2 0.7

❗Tips: you can open the run_XXX.sh file for parameter or training target modification.

📚 Dataset

❗MMEA: From UMAEA Repo

❗MKGC: Download from Here

🤝 Cite:

Please condiser citing this paper if you use the code or data from our work. Thanks a lot :)

@article{chen2024power,
  title={The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework},
  author={Chen, Zhuo and Fang, Yin and Zhang, Yichi and Guo, Lingbing and Chen, Jiaoyan and Chen, Huajun and Zhang, Wen},
  journal={arXiv preprint arXiv:2403.06832},
  year={2024}
}