[Model] DeepseekV2 Support

[saurabhkoshatwar]

Summary

Resolves #129 Add monkeypatch to support deepseepV2 model.

Details

Ops patched:

• rms_norm
• swiglu
• cross_entropy
• fused_linear_cross_entropy

Testing Done

• Hardware Type: NVIDIA A100-SXM4-40GB
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[saurabhkoshatwar]

@ByronHsu @yundai424 @Tcc0403 @qingquansong
As discussed in the issue, the rope implementation is different in DeepSeek.

deepseek:

    cos = cos[position_ids].unsqueeze(unsqueeze_dim)
    sin = sin[position_ids].unsqueeze(unsqueeze_dim)

    b, h, s, d = q.shape
    q = q.view(b, h, s, d // 2, 2).transpose(4, 3).reshape(b, h, s, d)

    b, h, s, d = k.shape
    k = k.view(b, h, s, d // 2, 2).transpose(4, 3).reshape(b, h, s, d)

    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)
    return q_embed, k_embed

llama:

    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)
    return q_embed, k_embed`

I will create a separate PR to implement the DeepSeek rope.

[tyler-romero]

Do deepseek and deepseek-v3 share the same architecture? If so, perhaps this function should be called apply_liger_kernel_to_deepseek, if not, perhaps we should strengthen this check.

[tyler-romero]

Why the change to create the model before applying the patch?

[shivam15s]

Don't see a reason for the change as well unless you want to experiment that patching still works post model init :)

[tyler-romero]

Can you explain why this is necessary? Its it because the model cannot be run without trust_remote_code? As is, this default opts-in anyone who runs these unit tests into running remote code on their machine, which is a red flag.

I think a preferable path would be to add deepseekv2 to the transformers library, then add it to Liger, so that trust_remote_code is not necessary.

This also has the benefit of making it easier to follow changes that are made to the underlying model, which is a common source of bugs in Liger.

[tyler-romero]

It looks like support for deepseekv2 is underway (maybe stalled though): huggingface/transformers#31976

[shivam15s]

We ideally don't want tests to require internet access. Can we keep the remote code in tests/resources folder for the time being. This would also make us omit the trust_remote_code issues.

[shivam15s]

Don't see a reason for the change as well unless you want to experiment that patching still works post model init :)

[shivam15s]

see my previous comment

[shivam15s]

see previous