v0.9.0
The 0.9.0 release adds new models, hub integrations, and general usability improvements.
Summary
- Added the Gemma 1.1 release.
- Added the Llama 2, BLOOM and ELECTRA models.
- Expose new base classes. Allow
from_preset()
on base classes.keras_nlp.models.Backbone
keras_nlp.models.Task
keras_nlp.models.Classifier
keras_nlp.models.CausalLM
keras_nlp.models.Seq2SeqLM
keras_nlp.models.MaskedLM
- Some initial features for uploading to model hubs.
backbone.save_to_preset
,tokenizer.save_to_preset
,keras_nlp.upload_preset
.from_preset
andupload_preset
now work with the Hugging Face Models Hub.- More features (task saving, lora saving), and full documentation coming soon.
- Numerical fixes for the Gemma model at mixed_bfloat16 precision. Thanks unsloth for catching!
# Llama 2. Needs Kaggle consent and login, see https://github.com/Kaggle/kagglehub
causal_lm = keras_nlp.models.LlamaCausalLM.from_preset(
"llama2_7b_en",
dtype="bfloat16", # Run at half precision for inference.
)
causal_lm.generate("Keras is a", max_length=128)
# Base class usage.
keras_nlp.models.Classifier.from_preset("bert_base_en", num_classes=2)
keras_nlp.models.Tokenizer.from_preset("gemma_2b_en")
keras_nlp.models.CausalLM.from_preset("gpt2_base_en", dtype="mixed_bfloat16")
What's Changed
- Add dtype arg to Gemma HF conversion script by @nkovela1 in #1452
- Fix gemma testing import by @mattdangerw in #1462
- Add docstring for PyTorch conversion script install instructions by @nkovela1 in #1471
- Add an annotation to tests that need kaggle auth by @mattdangerw in #1470
- Fix Mistral memory consumption with JAX and default dtype bug by @tirthasheshpatel in #1460
- Bump the master version to 0.9 by @mattdangerw in #1473
- Pin to TF 2.16 RC0 by @sampathweb in #1478
- Fix gemma rms_normalization's use of epsilon by @cpsauer in #1472
- Add
FalconBackbone
by @SamanehSaadat in #1475 - CI - Add kaggle creds to pull model by @sampathweb in #1459
- bug in example for ReversibleEmbedding by @TheCrazyT in #1484
- doc fix for constrastive sampler by @mattdangerw in #1488
- Remove broken link to masking and padding guide by @mattdangerw in #1487
- Fix a typo in causal_lm_preprocessors by @SamanehSaadat in #1489
- Fix dtype accessors of tasks/backbones by @mattdangerw in #1486
- Auto-labels 'gemma' on 'gemma' issues/PRs. by @shmishra99 in #1490
- Add BloomCausalLM by @abuelnasr0 in #1467
- Remove the bert jupyter conversion notebooks by @mattdangerw in #1492
- Add
FalconTokenizer
by @SamanehSaadat in #1485 - Add
FalconPreprocessor
by @SamanehSaadat in #1498 - Rename 176B presets & Add other presets into bloom_presets.py by @abuelnasr0 in #1496
- Add bloom presets by @abuelnasr0 in #1501
- Create workflow for auto assignment of issues and for stale issues by @sachinprasadhs in #1495
- Update requirements to TF 2.16 by @sampathweb in #1503
- Expose Task and Backbone by @mattdangerw in #1506
- Clean up and add our gemma conversion script by @mattdangerw in #1493
- Don't auto-update JAX GPU by @sampathweb in #1507
- Keep rope at float32 precision by @grasskin in #1497
- Bump the python group with 2 updates by @dependabot in #1509
- Fixes for the LLaMA backbone + add dropout by @tirthasheshpatel in #1499
- Add
LlamaPreprocessor
andLlamaCausalLMPreprocessor
by @tirthasheshpatel in #1511 - Always run the rotary embedding layer in float32 by @tirthasheshpatel in #1508
- CI: Fix psutil - Remove install of Python 3.9 and alias of python3 by @sampathweb in #1514
- Update gemma_backbone.py for sharding config. by @qlzh727 in #1491
- Docs/modelling layers by @mykolaskrynnyk in #1502
- Standardize docstring by @sachinprasadhs in #1516
- Support tokenization of special tokens for word_piece_tokenizer by @abuelnasr0 in #1397
- Upload Model to Kaggle by @SamanehSaadat in #1512
- Add scoring mode to MistralCausalLM by @RyanMullins in #1521
- Add Mistral Instruct V0.2 preset by @tirthasheshpatel in #1520
- Add Tests for Kaggle Upload Validation by @SamanehSaadat in #1524
- Add presets for Electra and checkpoint conversion script by @pranavvp16 in #1384
- Allow saving / loading from Huggingface Hub preset by @Wauplin in #1510
- Stop on multiple end tokens by @grasskin in #1518
- Fix doc:
mistral_base_en
->mistral_7b_en
by @asmith26 in #1528 - Add lora example to GemmaCausalLM docstring by @SamanehSaadat in #1527
- Add LLaMA Causal LM with 7B presets by @tirthasheshpatel in #1526
- Add task base classes; support out of tree library extensions by @mattdangerw in #1517
- Doc fixes by @mattdangerw in #1530
- Run the LLaMA and Mistral RMS Layer Norm in float32 by @tirthasheshpatel in #1532
- Adds score API to GPT-2 by @RyanMullins in #1533
- increase pip timeout to 1000s to avoid connection resets by @sampathweb in #1535
- Adds the score API to LlamaCausalLM by @RyanMullins in #1534
- Implement compute_output_spec() for tokenizers with vocabulary. by @briango28 in #1523
- Remove staggler type annotiations by @mattdangerw in #1536
- Always run SiLU activation in float32 for LLaMA and Mistral by @tirthasheshpatel in #1540
- Bump the python group with 2 updates by @dependabot in #1538
- Disallow saving to preset from keras 2 by @SamanehSaadat in #1545
- Fix the rotary embedding computation in LLaMA by @tirthasheshpatel in #1544
- Fix re-compilation bugs by @mattdangerw in #1541
- Fix preprocessor from_preset bug by @mattdangerw in #1549
- Fix a strange issue with preprocessing layer output types by @mattdangerw in #1550
- Fix lowercase bug in wordpiece tokenizer by @abuelnasr0 in #1543
- Small docs updates by @mattdangerw in #1553
- Add a few new preset for gemma by @mattdangerw in #1556
- Remove the dev prefix for 0.9.0 release by @mattdangerw in #1557
New Contributors
- @cpsauer made their first contribution in #1472
- @SamanehSaadat made their first contribution in #1475
- @TheCrazyT made their first contribution in #1484
- @shmishra99 made their first contribution in #1490
- @sachinprasadhs made their first contribution in #1495
- @mykolaskrynnyk made their first contribution in #1502
- @RyanMullins made their first contribution in #1521
- @Wauplin made their first contribution in #1510
- @asmith26 made their first contribution in #1528
- @briango28 made their first contribution in #1523
Full Changelog: v0.8.2...v0.9.0