Tessa/callibration script #937
Conversation
|
lgtm! I kinda hate checking in notebooks but I do think it's better than a script in this case. |
| @@ -0,0 +1,10 @@ | |||
| # Callibration | |||
There was a problem hiding this comment.
| # Callibration | |
| # Calibration |
| A good benchmark is one that clearly shows which models are better and which are worse. We test our benchmark tasks by using a series of progressively more advanced models to see if the benchmarks effectively differentiate between them, and at which number of shots they performed best at. | ||
|
|
||
| To run the code: | ||
| * Select an independant variable and a series of models that correspond to the settings of that variable |
There was a problem hiding this comment.
| * Select an independant variable and a series of models that correspond to the settings of that variable | |
| * Select an independent variable and a series of models that correspond to the settings of that variable |
throughout
There was a problem hiding this comment.
I suggest clearing the cell output before committing the notebook.
Easiest thing might be to just add the precommit hook from composer for this
- repo: https://github.com/kynan/nbstripout
rev: 0.5.0
hooks:
- id: nbstripout
types:
- "jupyter"
args:
# Strip all the metadata that vscode or colab may add to a notebook
- --strip-empty-cells
- --extra-keys
- >
metadata.colab metadata.interpreter metadata.accelerator
metadata.kernelspec metadata.language_info.version
cell.metadata.heading_collapsed metadata.name metadata.nbconvert_exporter
metadata.version metadata.vscode
| integrations: | ||
| - integration_type: git_repo | ||
| git_repo: mosaicml/llm-foundry | ||
| git_branch: main |
There was a problem hiding this comment.
should probably be pinned to a release.
There was a problem hiding this comment.
Lets move this to the eval/yamls folder
|
Would you mind adding the MCLI name of a test run you launched so I can go back and Additionally a screenshot of the resulting notebook would be good so that when I go back to this later I can confirm that I got the correct results? |
Here is code we use to test our benchmark tasks by using a series of progressively more advanced models to see if the benchmarks effectively differentiate between them, and at which number of shots they performed best at.
base_callibration.yamlto reflect the ones you want to seeanalyze_outputnotebook which collates the results from wandb