v0.2

Version 0.2 (#149)

* final evaluation notebook (#98)

* model specify + nextlogprobs load + tok specify

* Draft version of plotting

* get prompt examples

* add meeting todos

* grouped imports and cells

* redid typing and fixed bugs

* unique token check

* Remove token category in calculation function

* Remove token category from vis function

* removed token_group in token diff + calculate all loss

* vis_pos_map highlight + optimization

* fixes: tokenization, mask, typing

* use interact_manual for resampling

* update quantile function tests

* small update

* beartype fix

* rm comment

* var rename

* eval notebook updates

---------

Co-authored-by: Siwei Li <siwli@ea.com>
Co-authored-by: VICTOR ABIA <transcendingvictor@gmail.com>
Co-authored-by: Jett <jettjaniak@gmail.com>
Co-authored-by: JaiDhyani <jaiwithani@gmail.com>
Co-authored-by: Jai <jai@jai.one>

* remove stale notebooks, token map & labelling

* requirements revamp

* stale eval code purge

* remove constants

* use util in tokenize_dataset

* use load_dataset_split_* utils

* simpler structure: remove eval & dataset dirs

* src/delphi -> delphi

* remove stale test configs

* update HF cache version

* add notebooks deps to gh actions

* remove stale configs

* typo fix in train_tokenizer

* platformdirs dependency

* cosmetic changes in configs & args

* create HF repo once, outside try/except

* README: # Setup

* README: tokeniz*

* config help as comments

* dataset.name -> dataset.path

* README update

* README update

* validate_configs: overrides, init model

* README: out-repo typo

---------

Co-authored-by: Rai <62800649+menamerai@users.noreply.github.com>
Co-authored-by: Siwei Li <siwli@ea.com>
Co-authored-by: VICTOR ABIA <transcendingvictor@gmail.com>
Co-authored-by: JaiDhyani <jaiwithani@gmail.com>
Co-authored-by: Jai <jai@jai.one>

v0.1.1

update llama2 configs