Add Llama

[seanmor5]

Adds the popular Llama model. The model builds and the parameter map is correct, though the test is not passing. I'll look deeper into why the values differ tomorrow.

[seanmor5]

@jonatanklosko Seems there's something with the tokenizer we don't support?

Tokenizer(Error("data did not match any variant of untagged enum NormalizerWrapper", line: 49, column: 3))

[jonatanklosko]

@seanmor5 perhaps we need to bump rust tokenizers in elixir-nx/tokenizers?

[jonatanklosko]

@philss I tried bumping rust tokenizers to 0.13.3, but I got:

error[E0432]: unresolved import `onig`
 --> /Users/jonatanklosko/.asdf/installs/rust/nightly/registry/src/github.com-1ecc6299db9ec823/tokenizers-0.13.3/src/utils/onig.rs:3:5
  |
3 | use onig::Regex;
  |     ^^^^ help: a similar path exists: `super::onig`

error[E0433]: failed to resolve: use of undeclared crate or module `onig`
  --> /Users/jonatanklosko/.asdf/installs/rust/nightly/registry/src/github.com-1ecc6299db9ec823/tokenizers-0.13.3/src/utils/onig.rs:12:60
   |
12 |     pub fn find_iter<'r, 't>(&'r self, inside: &'t str) -> onig::FindMatches<'r, 't> {
   |                                                            ^^^^ use of undeclared crate or module `onig`

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.
error: could not compile `tokenizers` due to 2 previous errors

Are they are missing feature flags in tokenizers? Adding explicit dependency on onig didn't help either, but I don't know what I'm doing :)

[seanmor5]

@jonatanklosko Not sure, I got the same thing

[philss]

@jonatanklosko I think the problem is that they are declaring onig as an optional dependency here: https://github.com/huggingface/tokenizers/blob/d19bc63c6770a8ef5e59c816b653b68cca329cde/tokenizers/Cargo.toml#L44

But they are "requiring" in the code as it would be always available.
A quick-fix for your bumping is to declare in our crate (ex_tokenizers) the dependency with the default feature, that is adding onig:

tokenizers = { version = "0.13.3", default-features = false, features = ["default"]}

After that, run cargo update.

[philss]

@jonatanklosko another fix is to use the "unstable_wasm" feature instead of "default". It is what this issue suggests: huggingface/tokenizers#1104
I think it may be better for us, since it includes less dependencies 😬

tokenizers = { version = "0.13.3", default-features = false, features = ["unstable_wasm"]}

[seanmor5]

This one just needs a new tokenizers release now!

[philss]

This one just needs a new tokenizers release now!

@seanmor5 I'm working on that :)

[philss]

@seanmor5 @jonatanklosko version 0.3.2 of tokenizers was just released!

[jonatanklosko]

@philss all passing, thanks

[feynmanliang]

A thought on usability: how do commercial users easily discover that Llama (and its finetunes) are GPL v3 which is usually not license compatible, compared the the Apache 2.0 one usually expects in Elixir OSS?

[jonatanklosko]

A thought on usability: how do commercial users easily discover that Llama (and its finetunes) are GPL v3 which is usually not license compatible, compared the the Apache 2.0 one usually expects in Elixir OSS?

Fair point, though I'm not sure there's anything specific we should do. A commercial user should always check the huggingface repository for license and other details. The elixir codebase is one thing, but fetching a model from Hugging Face is clearly downloading external files and should be treated as such.

Technically if someone trained llama from scratch using HF transformers I believe it could be distributed under a more permissive license.