Segmentation fault when running SimAI_simulator

[youknowgjb]

Hi,

The program throws "Segmentation fault" when running SimAI_simulator. I am using the Mixtral_8*7 MoE workload file generated from AICB and the provided topo Spectrum-X_128g_8gps_100Gbps_A100, and the standard conf SimAI.conf. The program throws segfault after processing layer 7.

chunk size is: 8192 , size is: 8192 , layer_num is: 6 , node: 0
info: all-gather forward pass collective issued for layer: mlp_moelayer, involved dimensions:  1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: mlp_moelayer is finished************
chunk size is: 8388608 , size is: 8388608 , layer_num is: 7 , node: 0
info: all-to-all forward pass collective issued for layer: mlp_moelayer, involved dimensions:  1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Segmentation fault (core dumped)

Any help is appreciated!

Workload file ~10 lines

HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 4 ep: 1 pp: 1 vpp: 32 ga: 8 all_gpus: 512 checkpoints: 0 checkpoint_initiates: 0 pp_comm: 0
2338
grad_norm       -1      1       ALLGATHER       3505913856      1       NONE    0       1       REDUCESCATTER   7011827712      100
moe_grad_norm1  -1      1       NONE    0       1       NONE    0       1       ALLGATHER_DP_EP 1879048192      100
moe_grad_norm2  -1      1       NONE    0       1       NONE    0       1       REDUCESCATTER_DP_EP     3758096384      100
embedding_layer -1      1       NONE    33554432        1       NONE    33554432        1       NONE    0       100
attention_column        -1      1       ALLGATHER       33554432        1       NONE    0       1       NONE    0       100
attention_row   -1      1       REDUCESCATTER   33554432        1       NONE    33554432        1       NONE    0       100
mlp_moelayer    -1      1       ALLGATHER       8192    1       ALLGATHER       8192    1       NONE    0       100
mlp_moelayer    -1      1       ALLTOALL        8388608 1       ALLTOALL        8388608 1       NONE    0       100
mlp_moelayer    -1      1       ALLTOALL_EP     8388608 1       ALLTOALL_EP     8388608 1       NONE    0       100
mlp_moelayer    -1      1       ALLGATHER       8388608 1       ALLGATHER       33554432        1       NONE    0       100
mlp_moelayer    -1      1       REDUCESCATTER   8388608 1       ALLGATHER       8388608 1       NONE    0       100

[youknowgjb]

After using gdb I'm seeing segfault at

    at /app/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/applications/astra-sim/system/MockNcclChannel.cc:68
#1  0x0000774c7c1d097b in AstraSim::Sys::get_nccl_Info (this=0x615eae200160, comm_ps=AstraSim::EP, data_size=8388608,
    collective_type=AstraSim::ComType::All_to_All)
    at /app/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/applications/astra-sim/system/Sys.cc:1401

(gdb) print mock_nccl_comms
$4 = std::map with 4 elements = {[AstraSim::TP] = 0x615eae201440, [AstraSim::DP] = 0x615eae3fab80, [AstraSim::EP] = 0x0, [AstraSim::DP_EP] = 0x615eae3ffe40}

It looks like mock_nccl_comms map does not contains [AstraSim::EP] entry, because in mock_nccl_comms_init() the logic only inits the entry when EP > 1, however the workload file has EP = 1, is this expected?

[MXtremist]

I think that's indeed a bug in AICB. In previous versions, the EP communication size was incorrectly generated even when EP=1, which should result in zero communication (for instance, in your case with layer_num=8, size of ALLTOALL_EP should be 0).

This issue has been resolved in AICB 2.0. The latest version includes updated Megatron-MoE communication logic that correctly handles this case. We highly recommend upgrading to the latest version, which should solve the problem.

Btw, I noticed a potential configuration mismatch in your setup. Your workload configuration specifies a 512-GPU topology, but the provided topology file corresponds to 128 GPUs. Aligning these is necessary for correct results.

Let me know if there are any other problems:)