adding finetuned model setup #62838

hitesh-1997 · 2024-05-22T06:05:20Z

Context

Adds support for finetuned FIM models hosted on Fireworks. This PR adds the 4 variants and the traffic is routed to the appropriate model.
Client side pull request: adding FIM finetuned model hosted on fireworks cody#4245

Test plan

curl -vS -X POST http://localhost:9992/v1/completions/fireworks -H 'Authorization: bearer <SGD_TOKEN>' -d '{"stream":false,"max_tokens":50, "model": "fim-fine-tuned-model-variant-1", "stop_sequences": ["\n\n"], "prompt": "const value = ", "stream":false}' -H 'X-sourcegraph-feature: code_completions'

curl -vS -X POST http://localhost:9992/v1/completions/fireworks -H 'Authorization: bearer <SGD_TOKEN>' -d '{"stream":false,"max_tokens":50, "model": "fim-fine-tuned-model-variant-2", "stop_sequences": ["\n\n"], "prompt": "const value = ", "stream":false}' -H 'X-sourcegraph-feature: code_completions'

curl -vS -X POST http://localhost:9992/v1/completions/fireworks -H 'Authorization: bearer <SGD_TOKEN>' -d '{"stream":false,"max_tokens":50, "model": "fim-fine-tuned-model-variant-3", "stop_sequences": ["\n\n"], "prompt": "const value = ", "stream":false}' -H 'X-sourcegraph-feature: code_completions'

curl -vS -X POST http://localhost:9992/v1/completions/fireworks -H 'Authorization: bearer <SGD_TOKEN>' -d '{"stream":false,"max_tokens":50, "model": "fim-fine-tuned-model-variant-4", "stop_sequences": ["\n\n"], "prompt": "const value = ", "stream":false}' -H 'X-sourcegraph-feature: code_completions'

hitesh-1997 · 2024-05-22T06:07:05Z

internal/completions/client/fireworks/fireworks.go

@@ -170,7 +188,7 @@ func (c *fireworksClient) makeRequest(ctx context.Context, feature types.Complet
 		}

 		payload := fireworksRequest{
-			Model:       requestParams.Model,
+			Model:       c.updateModelStringIfFinetunedModelId(requestParams.Model),


Hi @rafax
My local cody gateway setup is not working, can you please verify this re-routing logic for the fine tuning model string.

rafax · 2024-05-22T07:28:05Z

curl -vS -X POST http://localhost:9992/v1/completions/fireworks -H 'Authorization: bearer <SGD_TOKEN>' -d '{"stream":false,"max_tokens":50, "model": "cody-autocomplete-fim-finetuned-model-variant1", "stop_sequences": ["\n\n"], "prompt": "const value = ", "stream":false}' -H 'X-sourcegraph-feature: code_completions'

You want fim-fine-tuned-model-variant-1 as model name (no cody-autocomplete, with a - in fine-tuned, with a - before 1).

hitesh-1997 · 2024-05-22T07:41:41Z

curl -vS -X POST http://localhost:9992/v1/completions/fireworks -H 'Authorization: bearer <SGD_TOKEN>' -d '{"stream":false,"max_tokens":50, "model": "cody-autocomplete-fim-finetuned-model-variant1", "stop_sequences": ["\n\n"], "prompt": "const value = ", "stream":false}' -H 'X-sourcegraph-feature: code_completions'

You want fim-fine-tuned-model-variant-1 as model name (no cody-autocomplete, with a - in fine-tuned, with a - before 1).

Ah yes, sorry I did not run it since my local gateway is not working, I updated the commands in the description

hitesh-1997 · 2024-05-22T08:22:22Z

internal/completions/types/types.go

@@ -65,6 +65,7 @@ type CompletionRequestParameters struct {
 	Model             string    `json:"model,omitempty"`
 	Stream            *bool     `json:"stream,omitempty"`
 	Logprobs          *uint8    `json:"logprobs"`
+	LanguageId        string    `json:"languageId,omitempty"`


Hi @rafax
I added this new parameter for language level routing to fireworks models. Although it should empty by default, could you confirm this shouldn't break any existing functionality ?

@rafax do we still need changes here after shifting them to fireworks ?

rafax · 2024-05-22T08:44:33Z

internal/completions/client/fireworks/fireworks.go

@@ -152,6 +155,50 @@ func (c *fireworksClient) Stream(
 	return dec.Err()
 }

+func (c *fireworksClient) updateModelStringIfFinetunedModelId(languageId string, model string) string {


Note that this code is in Sourcegraph backend (no cody-gateway in the path), so it will not be executed for PLG requests (which bypass sourcegraph.com). If you want this to apply to PLG, you need to modify cmd/cody-gateway/internal/httpapi/completions/fireworks.go (ex. pickStarCoderModel function)

Hi @rafax do we need this from here ?

rafax

Move code to Cody Gateway

Move code for rewriting fine-tuned models

hitesh-1997 · 2024-05-22T17:00:25Z

cmd/cody-gateway/internal/httpapi/completions/fireworks.go


 	body.Model = pickStarCoderModel(body.Model, f.config)
+	body.Model = pickFineTunedModel(body.Model, body.LanguageID)


Hi @rafax
did not notice this before merging the PR, but doing this body.LanguageID = "" in line 142 for the condition body.LanguageID will always make languageId empty. I tested it locally and it was always returning the all language completions. Adding a minor fix for this, which I tested locally.

Right, I updated the E2E tests to catch similar issues (verifying we resolve the right model)

adding finetuned model setup

12c9ac5

cla-bot bot added the cla-signed label May 22, 2024

hitesh-1997 requested a review from rafax May 22, 2024 06:05

hitesh-1997 commented May 22, 2024

View reviewed changes

adding language level routing for llama and mixtral

3ed96ae

hitesh-1997 commented May 22, 2024

View reviewed changes

rafax approved these changes May 22, 2024

View reviewed changes

rafax requested changes May 22, 2024

View reviewed changes

Merge branch 'main' into hitesh/finetuned-fim-model-integration

4c31893

rafax mentioned this pull request May 22, 2024

Move code to Cody Gateway + allowlist Fireworks models #62856

Merged

rafax and others added 2 commits May 22, 2024 22:03

Move code to Cody Gateway + allowlist Fireworks models (#62856)

98077fa

Move code for rewriting fine-tuned models

Merge branch 'main' into hitesh/finetuned-fim-model-integration

01fdea9

hitesh-1997 commented May 22, 2024

View reviewed changes

fix lanuage id for fine-tuning variants

746393c

hitesh-1997 requested review from janhartman and rishabhmehrotra May 22, 2024 18:00

rafax added 4 commits May 22, 2024 21:09

Revert Pro tier change

9218b57

Remove .com code

abee80e

Remove unneeded param

8fd6609

Verify resolved model

69e1f82

rafax approved these changes May 22, 2024

View reviewed changes

hitesh-1997 merged commit f12393d into main May 22, 2024
11 checks passed

hitesh-1997 deleted the hitesh/finetuned-fim-model-integration branch May 22, 2024 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding finetuned model setup #62838

adding finetuned model setup #62838

hitesh-1997 commented May 22, 2024 •

edited

Loading

hitesh-1997 May 22, 2024

rafax commented May 22, 2024 •

edited

Loading

hitesh-1997 commented May 22, 2024 •

edited

Loading

hitesh-1997 May 22, 2024

hitesh-1997 May 22, 2024

rafax May 22, 2024

rafax May 22, 2024

hitesh-1997 May 22, 2024

rafax May 22, 2024

rafax left a comment

hitesh-1997 May 22, 2024

rafax May 22, 2024


		body.Model = pickStarCoderModel(body.Model, f.config)
		body.Model = pickFineTunedModel(body.Model, body.LanguageID)

adding finetuned model setup #62838

adding finetuned model setup #62838

Conversation

hitesh-1997 commented May 22, 2024 • edited Loading

Context

Test plan

Choose a reason for hiding this comment

rafax commented May 22, 2024 • edited Loading

hitesh-1997 commented May 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rafax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hitesh-1997 commented May 22, 2024 •

edited

Loading

rafax commented May 22, 2024 •

edited

Loading

hitesh-1997 commented May 22, 2024 •

edited

Loading