Support peft LoRA adapters

[artek0chumak]

No description provided.

[borzunov]

Can you please replace this with a smaller pair of adapters? The current ones are too large to download in CI, this slows down tests by ~30%.

[artek0chumak]

Create smaller peft weights for bloom-560m

[justheuristic]

Review / comments:

1. It seems that the test adapter is initialized at zeros; therefore, the test for exact match with adapters does not fully test exact match.

   • can you please update safe peft to make sure that they affect outputs in a measurable way?
   • we can use any dummy weights s.t. not allclose(outputs_with_peft, outputs_without_peft)

2. Currently we are using tensor_parallel with adapters naively, as though it was non-parallel

• this may cause problems in case when adapters are non-zero
• unless it is super-easy to fix, consider simply assert not adapters or not tensor_parallel for now
• naive, but correct solution for later: initialize shards[0] with correct adapters; shards[1,2,...] with no adapters
• optimal, but complicated solution for later: split adapter evenly along the low-rank dimension; this can be done using tensor_parallel's config

3. I may have screwed up some of the tests

• there's one change i made that seemed kinda sus: bf498a4

4. Load balancing in presence of adapters

• the current load-balancing algorithm ignores adapters altogether
• this is fine if peers are allowed to download missing adapters on demand (hopefully this is what we're gonna do)
• if not, we will have to somehow adapt load-balancing to solve cases where
  a. the bottleneck for the vanilla model is in one spot
  b. but if we consider only layers that have these two adapters, the bottleneck is in this other spot
  c. and that one small rarely used adapter doesn't even have enough servers for a full chain
• in either case, let's create an issue

5. adapter dtypes

• not necessarily in this PR, but it would be nice to make sure that the "safe" guanaco is cast to FP16 to save gpu memory

Note to @borzunov : the tests do indeed take longer now; the reason behind this is that we run 2 more "heavy" tests that verify full model exact match with adapters. We can probably skip them if this is a problem.

[artek0chumak]

1. It seems that the test adapter is initialized at zeros; therefore, the test for exact match with adapters does not fully test exact match.

Thank you for mentioning this! Change LoRA weights in hub, currently they have non-zero parameters.