Inquiry About Dataset and Training Details for MoBA/Full Hybrid Experiment

[yukiy927]

Great work on your paper! I’m currently trying to reproduce the MoBA/full hybrid experiment and was wondering if you could share details about the dataset you used.

I noticed that the experiment involves training on 30B tokens with a context length of 32K. Does this mean that all training samples must be at least 32K tokens long? Or is it acceptable to construct training samples by concatenating shorter sequences?

I’m particularly concerned about how concatenation might affect the final loss curve. Since the position-wise loss would be computed over concatenated sequences, wouldn’t this lead to discontinuities in the loss curve, making it difficult to match the trend shown in Figure 5a? For example, if I only have 10K-token sequences, then the loss curve for a single 32K sequence would essentially be composed of three distinct segments, each corresponding to a different concatenated sample. Would this impact the outcomes?

That will be so great if you could provide some dataset info.

[whitelez]

Thank you for your interest in our work!

Regarding your questions about the dataset details:

1. Our data mainly consists of internally processed datasets, which unfortunately cannot be made public at this time.

2. For training with 32K context length, it is indeed necessary to have data samples that approach or reach 32K tokens in one sample. If you only use 10K-length data for training, the effective activation length of the model will only be around 10K. Even if shorter sequences are forcibly concatenated to create 32K-length inputs, the training results would be suboptimal. This is because simple concatenation introduces unnatural transitions at sequence boundaries, making it difficult for the model to learn true long-range dependencies.