Extend api/show and ollama show to return more model info #4881

royjhan · 2024-06-06T21:48:30Z

Building off of #3899
Resolves #3570, #2732, #3899

mxyng

You should look at the comments in the linked PR

mxyng · 2024-06-06T22:40:07Z

server/routes.go

+	var keys []string
+	for k := range kv {
+		keys = append(keys, k)
+	}


Suggested change

var keys []string

for k := range kv {

keys = append(keys, k)

}

keys := maps.Keys(kv)

mxyng · 2024-06-06T22:40:50Z

server/routes.go

+		keys = append(keys, k)
+	}
+
+	kvMap := make(map[string]any)


I don't see a reason to copy the KVs

mxyng · 2024-06-06T22:50:57Z

server/routes.go

+	ggmlMap["kv"] = kvMap
+	ggmlMap["tensors"] = ggml.Tensors()
+
+	ggmlJson, err := json.Marshal(ggmlMap)


Suggested change

ggmlJson, err := json.Marshal(ggmlMap)

return json.Marshal(ggmlMap)

Co-Authored-By: Patrick Devine <pdevine@sonic.net>

royjhan · 2024-06-06T23:06:33Z

You should look at the comments in the linked PR

My bad I just pushed a copy of Patrick's code as a draft for reference, didn't mean to put it up for review yet

royjhan · 2024-06-06T23:29:00Z

Tokens and token_type are empty slices right now due to the < 5 condition, since llama3 overpopulates these like crazy. We can decide what direction we wanna go with this, maybe could add a flag to add them and remove them from ModelInfo for instance

mxyng · 2024-06-06T23:43:44Z

server/routes.go

+		return nil, err
+	}
+
+	f.Close()


The best way to close a file is to defer right after open succeeds, e.g.

f, err := os.Open(p) if err != nil { return nil, err } defer f.Close()

This ensure the file is closed even when the function returns early, e.g. DecodeGGML errors

mxyng · 2024-06-06T23:47:21Z

server/routes.go

+		}
+	}
+
+	kv["embedding_model"] = model.IsEmbedding()


I'm not sure about adding this as a KV since it's not part of the model itself. Ideally this can be captured as a top-level capabilities

mxyng · 2024-06-06T23:50:23Z

server/routes.go

+		switch v := kv[k].(type) {
+		case []interface{}:
+			if len(v) > 5 {
+				kv[k] = []string{}
+			}
+		}


I mentioned this in the other PR but showing an empty list is very confusing. Instead, this should show how many items are in the list. TBH this doesn't even need a length check, just always output the count

if t, ok := kv[k].([]any); ok { kv[k] = fmt.Sprintf("(%d items)", len(t)) }

mxyng · 2024-06-06T23:52:16Z

server/routes.go

 	return resp, nil
 }

+func getGGMLData(model *Model) ([]byte, error) {


Instead of returning []byte, this should return llm.KV

jmorganca · 2024-06-15T06:38:40Z

cmd/cmd.go

@@ -623,32 +625,144 @@ func ShowHandler(cmd *cobra.Command, args []string) error {
 		showType = "template"
 	}

-	if flagsSet > 1 {
+	switch flagsSet {


instead of a switch here, we can early return:

if len(flagsSet) > 1 { return errors.New("only one of '--license', '--modelfile', '--parameters', '--system', or '--template' can be specified") } if len(flagsSet) == 1 { ... do single flag stuff return nil } .. do default stuff return nil

jmorganca · 2024-06-15T06:41:40Z

cmd/cmd.go

@@ -579,8 +579,10 @@ func ShowHandler(cmd *cobra.Command, args []string) error {
 		return err
 	}

-	if len(args) != 1 {
+	if len(args) == 0 {


This is already handled by https://github.com/ollama/ollama/blob/main/cmd/cmd.go#L1119 so we can just remove this check

README.md

jmorganca · 2024-06-15T06:46:31Z

server/routes.go

@@ -720,9 +720,50 @@ func GetModelInfo(req api.ShowRequest) (*api.ShowResponse, error) {
 	fmt.Fprint(&sb, model.String())
 	resp.Modelfile = sb.String()

+	ggmlData, err := getGGMLData(model.ModelPath, req.Verbose)


I believe you can use llm.LoadModel:

ggml, err := llm.LoadModel(pending.model.ModelPath) kv := ggml.KV() ...

Ah I see we do some post processing for verbose below – above might still be useful for getGGMLData

jmorganca · 2024-06-16T19:37:24Z

cmd/cmd.go

+	return res
+}
+
+func handleParams(s string) string {


formatParams is a better name here

jmorganca · 2024-06-16T19:38:18Z

This is overall good! Let's add some tests to the handler, including:

basic test for model info
test projector case

royjhan · 2024-06-17T18:23:52Z

server/routes_test.go

@@ -212,6 +213,7 @@ func Test_Routes(t *testing.T) {
 					"top_p 0.9",
 				}
 				assert.Equal(t, expectedParams, params)
+				assert.InDelta(t, 0, showResp.ModelInfo["general.parameter_count"], 1e-9, "Parameter count should be 0")


this field gets set to 0 by default in gguf.go, but might not be a robust test

jmorganca · 2024-06-17T19:15:33Z

server/routes_show_test.go

+	"github.com/stretchr/testify/assert"
+)
+
+func TestShow(t *testing.T) {


Why not put this in routes_test.go?

Noting routes_create_test.go, routes_delete_test.go, routes_list_test.go, I thought show could be nice to separate out as well given the projector test. If we want to get rid of this file and just stick everything in routes_test, I'd change the create model handler to include model and projector KVs that can be retrieved in a show handler, but we might want to keep this stuff isolated. Lmk what you think

Oh yeah and if you mean literally just sticking this function in routes_test.go, yeah its just to be consistent with the existence of those three other files Mike created

Gotcha, sorry I missed those others ones. I'd start in routes.go and move it out when it gets large enough, just to avoid sprawl.

jmorganca · 2024-06-17T20:44:24Z

server/routes_show_test.go

+		t.Fatal(err)
+	}
+
+	assert.Equal(t, "test", resp.ModelInfo["general.architecture"])


Let's use t.Fatal instead of assert to stick to stdlib test functions

jmorganca · 2024-06-17T20:45:53Z

cmd/cmd.go

+		{renderSubTable(modelData, false)},
+	}
+
+	if slices.Contains(resp.Details.Families, "clip") {


Let's check for resp.ProjectorInfo instead of looking for clip in resp.Details.Families

jmorganca · 2024-06-19T20:03:03Z

README.md

@@ -182,6 +182,12 @@ $ ollama run llama3 "Summarize this file: $(cat README.md)"
 Ollama is a lightweight, extensible framework for building and running language models on the local machine. It provides a simple API for creating, running, and managing models, as well as a library of pre-built models that can be easily used in a variety of applications.
 ```

+### Show model information


Possible to make a separate PR for the docs change? We can merge it once this is released

jmorganca · 2024-06-19T20:03:37Z

docs/api.md

+  "model_info": {
+    "general.architecture": "llama",
+    "general.file_type": 2,
+    "general.name": "Meta-Llama-3-8B-Instruct",


Let's remove this field as well for now

jmorganca

A few small items otherwise LGTM!

* API Show Extended * Initial Draft of Information Co-Authored-By: Patrick Devine <pdevine@sonic.net> * Clean Up * Descriptive arg error messages and other fixes * Second Draft of Show with Projectors Included * Remove Chat Template * Touches * Prevent wrapping from files * Verbose functionality * Docs * Address Feedback * Lint * Resolve Conflicts * Function Name * Tests for api/show model info * Show Test File * Add Projector Test * Clean routes * Projector Check * Move Show Test * Touches * Doc update --------- Co-authored-by: Patrick Devine <pdevine@sonic.net>

API Show Extended

ccd624c

mxyng reviewed Jun 6, 2024

View reviewed changes

Initial Draft of Information

30f7064

Co-Authored-By: Patrick Devine <pdevine@sonic.net>

royjhan changed the title ~~API Show Extended (Resolves #3570)~~ API Show Extended Jun 6, 2024

Clean Up

cdbda76

royjhan marked this pull request as ready for review June 6, 2024 23:21

royjhan requested a review from pdevine June 6, 2024 23:22

mxyng reviewed Jun 6, 2024

View reviewed changes

royjhan marked this pull request as draft June 7, 2024 00:00

Descriptive arg error messages and other fixes

ebbaa8b

royjhan changed the title ~~API Show Extended~~ Extend api/show and ollama show to return more model info Jun 10, 2024

royjhan added 2 commits June 10, 2024 15:01

Second Draft of Show with Projectors Included

c453d52

Remove Chat Template

3ad14e1

royjhan self-assigned this Jun 10, 2024

royjhan added 4 commits June 10, 2024 16:32

Touches

78e1434

Prevent wrapping from files

c8e1bb5

Verbose functionality

b005bec

Docs

818074c

jmorganca mentioned this pull request Jun 15, 2024

cmd: initial functionality for ollama show <model> #4922

Closed

jmorganca reviewed Jun 15, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

jmorganca reviewed Jun 15, 2024

View reviewed changes

royjhan and others added 4 commits June 15, 2024 21:41

Address Feedback

3f9a020

Merge branch 'main' into royh-show

2a6d37a

Lint

3d4f859

Resolve Conflicts

5e4f5dc

royjhan marked this pull request as ready for review June 16, 2024 05:31

jmorganca reviewed Jun 16, 2024

View reviewed changes

cmd/cmd.go Outdated

return res

}

func handleParams(s string) string {

Copy link

Member

jmorganca Jun 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatParams is a better name here

royjhan added 2 commits June 17, 2024 09:38

Function Name

1d4bdee

Tests for api/show model info

1304e2e

royjhan commented Jun 17, 2024

View reviewed changes

Show Test File

917ab20

jmorganca reviewed Jun 17, 2024

View reviewed changes

royjhan added 2 commits June 17, 2024 13:00

Add Projector Test

81f301a

Clean routes

3fb4cec

jmorganca reviewed Jun 17, 2024

View reviewed changes

royjhan added 2 commits June 17, 2024 14:51

Projector Check

cc3bb98

Move Show Test

1b25a7b

jmorganca reviewed Jun 19, 2024

View reviewed changes

royjhan added 2 commits June 19, 2024 13:35

Touches

95c1775

Doc update

7a5069f

jmorganca approved these changes Jun 19, 2024

View reviewed changes

royjhan merged commit fedf716 into main Jun 19, 2024
12 checks passed

royjhan deleted the royh-show branch June 19, 2024 21:19

dhiltgen mentioned this pull request Jul 24, 2024

Provide model metadata with ollama show #1173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend api/show and ollama show to return more model info #4881

Extend api/show and ollama show to return more model info #4881

royjhan commented Jun 6, 2024 •

edited

Loading

mxyng left a comment

mxyng Jun 6, 2024

mxyng Jun 6, 2024

mxyng Jun 6, 2024

royjhan commented Jun 6, 2024

royjhan commented Jun 6, 2024

mxyng Jun 6, 2024

mxyng Jun 6, 2024

mxyng Jun 6, 2024

mxyng Jun 6, 2024

jmorganca Jun 15, 2024

jmorganca Jun 15, 2024

jmorganca Jun 15, 2024

jmorganca Jun 15, 2024

jmorganca Jun 16, 2024

jmorganca commented Jun 16, 2024

royjhan Jun 17, 2024

jmorganca Jun 17, 2024

royjhan Jun 17, 2024 •

edited

Loading

royjhan Jun 17, 2024

jmorganca Jun 17, 2024

jmorganca Jun 17, 2024

jmorganca Jun 17, 2024

jmorganca Jun 19, 2024

jmorganca Jun 19, 2024

jmorganca left a comment

	ggmlJson, err := json.Marshal(ggmlMap)
	return json.Marshal(ggmlMap)

Extend api/show and ollama show to return more model info #4881

Extend api/show and ollama show to return more model info #4881

Conversation

royjhan commented Jun 6, 2024 • edited Loading

mxyng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

royjhan commented Jun 6, 2024

royjhan commented Jun 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmorganca commented Jun 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

royjhan Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmorganca left a comment

Choose a reason for hiding this comment

royjhan commented Jun 6, 2024 •

edited

Loading

royjhan Jun 17, 2024 •

edited

Loading