Any plans to support phi-2 finetuning?

[asmith26]

Hi there,

Just thought I'd ask if there were any plans to support phi-2 finetuning, https://huggingface.co/microsoft/phi-2?

Many thanks for any help, and this amazing lib!

[danielhanchen]

@asmith26 On it! :)

[imrankh46]

Hi there,

Just thought I'd ask if there were any plans to support phi-2 finetuning, https://huggingface.co/microsoft/phi-2?

Many thanks for any help, and this amazing lib!

I tried, fine tuning of phi-2. Which is not good.
I just use hugging face transformer code not with unsloth..

[danielhanchen]

@imrankh46 Ye smaller models can sometimes not follow instructions well. Could you approx give an example maybe to show Phi not following instructions?

[imrankh46]

@imrankh46 Ye smaller models can sometimes not follow instructions well. Could you approx give an example maybe to show Phi not following instructions?

here is the code.

https://colab.research.google.com/drive/1a7rL3UzWfo5I7OPyVmTEnR6_tRqIOblg?usp=sharing

[cm2435]

@danielhanchen I'm also interesting in contributing this, let me know if you have space on your PR for another helping hand.

[danielhanchen]

@cm2435 Oh more than happy to collab if you're into it!! I actually took a look at Phi the other day:

1. Phi seems to not use swiglu but just general f(gate) @ down. Swiglu is (f(gate) * up) @ down. This does mean the Triton kernels backprop has to be derived.
2. Phi is using gelu instead of swish, and I think the first step is to derive the gradient for gelu. Gelu is f(x) = 0.5 * x* (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x+ 0.044715 * torch.pow(x, 3.0)))) Maybe using Desmos / hand derived
3. Phi uses a general Layernom, not a RMS Layernorm this means again new kernels have to be written - although I think Triton's tutorials has this already done for us: https://triton-lang.org/main/getting-started/tutorials/05-layer-norm.html
4. There is some dropout (0.1) after each attention and MLP module I think. This shouldn't be hard to implement hopefully.
5. I don't know what partial_rotary_factor is in https://huggingface.co/microsoft/phi-2/blob/main/config.json. Need to investigate.

In general Phi is possible, just there are a few blockers - esp (2) and (3). But again - if you want to take a crack at this @cm2435 - I'll be super grateful + I'll collab with you! :) Possibly taking a stab at some steps like (2) finding the derivative might be better as a first step :)

[cm2435]

@danielhanchen Started a fork and opened a staging PR to work off. I can start by adding some unit test coverage around the kernels if we need to test their accuracy, something simple like

assert torch.allclose(triton_out, torch_out)

or I can try to contribute some of the other steps you mentioned- what's your preference?

[danielhanchen]

@cm2435 Oh lmao just noticed i never responded on this thread WHOOPS! Well I responded on your other threads already I guess - sorry!!!