Skip to content

Commit

Permalink
chore(example/llm): Use deepseek-coder-v2 for better usabilty demon…
Browse files Browse the repository at this point in the history 
…stration
  • Loading branch information
@shivaraj-bh
shivaraj-bh committed Jun 18, 2024
1 parent f9b3c6d commit 9c20952
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion example/llm/flake.nix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@
#
# You can also initialize this to empty list, and download the
# models manually in the UI.
models = [ "llama2-uncensored" ];
#
# Search for the models here: https://ollama.com/library
models = [ "deepseek-coder-v2" ];

This comment has been minimized.

Sign in to view

Copy link
@drupol

drupol Jun 20, 2024

Contributor

That makes 8.9GB to download before even running the UI. Is it really a good idea?

At home, that makes 24 minutes, at work probably twice or triple.

Compare time here: https://www.omnicalculator.com/other/download-time

This comment has been minimized.

Sign in to view

Copy link
@shivaraj-bh

shivaraj-bh Jun 20, 2024

Author Member

Agreed, it's a huge model. The main intention for this example has always been to demonstrate how Nix enables one to setup everything in one-click. It makes for a nice demo, like we showed here: https://x.com/nixos_asia/status/1803065244568244578. Using any smaller model would mean compromise on the response quality during the demo.

And since it's an example, users are not forced to just use this, they can have their own flake which doesn’t have these defaults.

This comment has been minimized.

Sign in to view

Copy link
@drupol

drupol Jun 20, 2024 •
edited
Loading

Contributor

I know I understand the intention behind this. I'm also aware that users are able to change that by themselves once they copied the files locally. It's just that, in my own conception of a demo, it should be obviously working but also very fast to spawn to not piss off a potential new user waiting 30 minutes before having something working.

For example, a pragmatic choice here would have been to use the phi3 model which works very well for practically all my use cases... and it's 2.4GB.

Try it :)

This comment has been minimized.

Sign in to view

Copy link
@shivaraj-bh

shivaraj-bh Jun 20, 2024

Author Member

For example, a pragmatic choice here would have been to use the phi3 model which works very well for practically all my use cases... and it's 2.4GB.

Even with programming tasks? Thanks for the suggestion, will check it out.

This comment has been minimized.

Sign in to view

Copy link
@srid

srid Jun 20, 2024

Collaborator

@drupol Zulip is more of an appropriate place to discuss this.

We already started a discussion there: https://nixos.zulipchat.com/#narrow/stream/414011-services-flake/topic/.E2.9C.94.20Require.20opinion.20on.20.60example.2Fllm.60
};

# Get ChatGPT like UI, but open-source, with Open WebUI
Expand Down

0 comments on commit 9c20952

Please sign in to comment.