-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
SDK Evaluation Samples - datafiles removed (#2312)
* sdk evaluation samples * remove the samples flagged at smoke build
- Loading branch information
Showing
24 changed files
with
3,054 additions
and
0 deletions.
There are no files selected for viewing
15 changes: 15 additions & 0 deletions
15
sdk/python/foundation-models/system/evaluation/fill-mask/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
## Fill Mask | ||
|
||
### List of supported keyword arguments: | ||
|
||
| Keyword Argument | Description | Type | Sample | | ||
|:-----------------:|:-----------------------------------------------------------------------------------------------------------------|-----------|-----------------------------| | ||
| metrics | List for subset of metrics to be computed. All supported metrics listed below. | list<str> | ["perplexities"] | | ||
| model_id | Model used for calculating Perplexity. Perplexity can only be calculated for causal language models. | str | "gpt2", "bert-base-uncased" | | ||
| batch_size | The batch size to run texts through the model | int | 16 | | ||
| add_start_token | Boolean flag to add the start token to the texts so the perplexity can include the probability of the first word | boolean | true, false | | ||
| custom_dimensions | Used to report telemetry data (can later be used to perform PII scrubbing) | dict | | | ||
|
||
### List of supported metrics: | ||
|
||
* perplexities |
8 changes: 8 additions & 0 deletions
8
sdk/python/foundation-models/system/evaluation/fill-mask/eval-config.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
{ | ||
"metrics": ["perplexities"], | ||
"model_id": "gpt2", | ||
"add_start_token": true, | ||
"tokenizer_config": { | ||
"truncation": true | ||
} | ||
} |
Binary file added
BIN
+225 KB
...thon/foundation-models/system/evaluation/fill-mask/fill-mask-eval-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.