Skip to content

Commit

Permalink
SDK Evaluation Samples - datafiles removed (#2312)
Browse files Browse the repository at this point in the history
* sdk evaluation samples

* remove the samples flagged at smoke build
  • Loading branch information
esrao committed May 19, 2023
1 parent 3a33b44 commit 266ad1b
Show file tree
Hide file tree
Showing 24 changed files with 3,054 additions and 0 deletions.
15 changes: 15 additions & 0 deletions sdk/python/foundation-models/system/evaluation/fill-mask/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
## Fill Mask

### List of supported keyword arguments:

| Keyword Argument | Description | Type | Sample |
|:-----------------:|:-----------------------------------------------------------------------------------------------------------------|-----------|-----------------------------|
| metrics | List for subset of metrics to be computed. All supported metrics listed below. | list<str> | ["perplexities"] |
| model_id | Model used for calculating Perplexity. Perplexity can only be calculated for causal language models. | str | "gpt2", "bert-base-uncased" |
| batch_size | The batch size to run texts through the model | int | 16 |
| add_start_token | Boolean flag to add the start token to the texts so the perplexity can include the probability of the first word | boolean | true, false |
| custom_dimensions | Used to report telemetry data (can later be used to perform PII scrubbing) | dict | |

### List of supported metrics:

* perplexities
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"metrics": ["perplexities"],
"model_id": "gpt2",
"add_start_token": true,
"tokenizer_config": {
"truncation": true
}
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 266ad1b

Please sign in to comment.