Skip to content

Commit

Permalink
Mabables/foundation models (Azure#2182)
Browse files Browse the repository at this point in the history 
* foundation model samples - text classification - emotion detection `date`

* foundation model samples - text classification - emotion detection Wed Mar 15 23:03:38 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 15 23:13:42 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 11:15:27 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 17:03:46 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 18:43:05 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 21:56:19 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 22:02:51 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 23:02:34 PDT 2023

* foundation model samples - text classification - emotion detection Wed Mar 22 23:10:16 PDT 2023

* foundation model samples - text classification - emotion detection Thu Mar 23 11:09:07 PDT 2023

* summarization sample

* ft notebooks - fixes for eval and latest label

* inference sample (Azure#2159)

* emotion detectio cli sample

* cli sample emotion detection

* cli sample emotion detection

* cli sample emotion detection

* cli sample emotion detection

* fill mask inference cli sample

* fill mask inference cli sample

* Updating FT notebooks and CLI examples (Azure#2160)

* notebook samples added for translation and token classification

* Adding cli examples

---------

Co-authored-by: HrishikeshGeedMS <hgeed@microsoft.com>

* mlflow for metrics

* adding eval components

* FT notebook updates - added dependencies installation, command to find number of GPU's in SKU and Metrics (Azure#2162)

* Adding requirements install, cmd to find gpu number and metrics to notebooks

* Adding deployment supported SKU list to notebooks

* Removing version dependency for azure-ai-ml

* Updating gpus query to python code

* renaming compute_model_selector -> compute_model_import

* evaluation text classification sample

* renaming compute_model_selector -> compute_model_import

* evaluation text classification sample

* evaluation text classification sample

* evaluation text classification sample

* correcting the evaluation config extension from jsonl -> json

* updating the logic to calculate the gpu count

* evaluation text classification sample

* inference text classification sample

* Adding github workflows for FT notebooks

* model eval dashboard screenshot

* Inference samples for all tasks (Azure#2175)

* Sample scripts for fill-mask and translation

* Add generic inference scripts

* remove task-specific folders

* remove whisper files

* Update sample inputs

* use task specific scripts

* Sample scripts for fill-mask and translation

* Add generic inference scripts

* remove task-specific folders

* remove whisper files

* Update sample inputs

* use task specific scripts

* Sample scripts for fill-mask and translation

* Add generic inference scripts

* remove task-specific folders

* remove whisper files

* Update sample inputs

* use task specific scripts

* Add generic inference scripts

* remove task-specific folders

* remove whisper files

* Update sample inputs

* use task specific scripts

* inference samples for all tasks

* Add sample script for text-classification

* Remove generic samples

* remove token-classification samples

* delete extra dataset

* Add token-classification-samples (Azure#2176)

* Model Evaluation sample notebooks

* Model Evaluation sample notebooks - Removing cell outputs

* Model Evaluation sample notebooks - Cleaning up notebooks

* Model Evaluation sample notebooks - Adding Git actions

* Model Evaluation sample notebooks - Renaming cluster name

* Model Evaluation sample notebooks - Adding model evaluation dashboard screenshots

* Model Evaluation sample notebooks - Fixing black workflow runs

* add whisper files and update scripts (Azure#2183)

* Model Evaluation sample notebooks - Adding configs for text-gen and fill-mask tasks

* Model Evaluation sample notebooks - Removing multilabel Github workflow

* Model Evaluation sample notebooks - Modifying eval-configs for fill-mask and text-gen

* Model Evaluation sample notebooks - Fixing missing info for fill-mask and text-gen notebooks

* Add ground truth comparison, update sample scores (Azure#2186)

* Fixing github runners (Azure#2188)

* Try to read workspace details from config

* Reformatted versions with black

* Fixing text classification notebook issue

* Adding mlflow installation

* Update QnA metric

* Handeling runs with None

* Model Evaluation sample notebooks - Adding documentation for evaluation configs

* Model Evaluation sample notebooks - Fixing black check run

* added ymls for model-evaluation-subgraph cli (Azure#2189)

* added ymls for model-evaluation-subgraph cli

* changed type to pipeline in cli files

* updated model id

* Removing input_column_names from FT cli

* batch sample for text classification (Azure#2199)

* batch sample

* batch sample

* batch samples

* batch samples for text-classification

* batch samples for text-classification

* Updating FT notebooks to use latest NCD input format

* updating config for text and token classification. (Azure#2204)

Co-authored-by: Chandra Sekhar Gupta Aravpalli <caravapalli@microsoft.com>

* Model Evaluation sample notebooks - Modifying to Azure production registry

* Model Evaluation sample notebooks - Fixing black runs

* Updating registry_name to azureml-preview-test1 temporarily

* Model Evaluation sample notebooks - Fixing text-classification notebook

* Model Evaluation sample notebooks - Fixing text-classification notebook

* Model Evaluation sample notebooks - Changing registry for model fetching

* Model Evaluation sample notebooks - Changing registry for model fetching

* Model Evaluation sample notebooks - Fixing fill mask masks

* Model Evaluation sample notebooks - Finalising notebooks

* Model Evaluation sample notebooks - Fixing black runs

* task name typo (Azure#2207)

* Setting up FT notebooks cron to run at midnight daily

* Changing NCD compute

* Changing registry to azureml

* Aditisingh/update names (Azure#2212)

* renamed directories

* renamed cli directories

* updated data paths

* Updating FT notebooks inference compute

* updating notebooks for evaluation of base models (Azure#2216)

* updating evaluation text classification notebooks.

* addding notebook for evaluation of sentiment analysis models.

---------

Co-authored-by: Chandra Sekhar Gupta Aravpalli <caravapalli@microsoft.com>

* Add batch fill mask example. (Azure#2208)

Add fill mask batch endpoint example notebook.

* Add remaining batch examples for HF foundation models (Azure#2217)

adding batch endpoint example notebooks for all except summarization and asr, which did not pass.

* Pmanoj/foundational models cli issues (Azure#2223)

* fixing cli script bugs

* Changing inference sku

---------

Co-authored-by: Pavan Manoj Jonnalagadda <pavanmanojj@microsoft.com>

* Reformatting inference files

* Model Evaluation sample notebooks - Fixing Github workflows

---------

Co-authored-by: Manoj Bableshwar <mabables@microsoft.com>
Co-authored-by: skanakamedal <116672436+skanakamedal@users.noreply.github.com>
Co-authored-by: HrishikeshGeedMS <hgeed@microsoft.com>
Co-authored-by: Narayanan Madhu <naraymad@microsoft.com>
Co-authored-by: SitaRam Chaitanya Kanakamedala <skanakamedal@microsoft.com>
Co-authored-by: Pavan Manoj Jonnalagadda <pavanmanojj@microsoft.com>
Co-authored-by: Sumadhva Sridhar <109793745+susridhar@users.noreply.github.com>
Co-authored-by: Sarthak Singhal <sarsinghal@microsoft.com>
Co-authored-by: Aditi Singh <114134940+s-aditi@users.noreply.github.com>
Co-authored-by: Chandra Sekhar Gupta <38103118+guptha23@users.noreply.github.com>
Co-authored-by: Chandra Sekhar Gupta Aravpalli <caravapalli@microsoft.com>
Co-authored-by: amymsft <94562419+amymsft@users.noreply.github.com>
Co-authored-by: Haritha Pallavi Bendapudi <hbendapudi@microsoft.com>
  • Loading branch information
@sarthaks95 @ManojBableshwar
@skanakamedal @HrishikeshGeedMS @naraymad1 @jpmann @sumadhva30 @s-aditi @guptha23 @amymsft @hbendapudi-ms
14 people committed Apr 27, 2023
1 parent d18f656 commit d3bf478
Show file tree
Hide file tree
Showing 138 changed files with 16,118 additions and 0 deletions.
77 changes: 77 additions & 0 deletions .github/workflows/sdk-foundation-models-system-evaluation-fill-mask-fill-mask.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: sdk-foundation-models-system-evaluation-fill-mask-fill-mask
# This file is created by sdk/python/readme.py.
# Please do not edit directly.
on:
workflow_dispatch:
schedule:
- cron: "34 11/12 * * *"
pull_request:
branches:
- main
paths:
- sdk/python/foundation-models/system/evaluation/fill-mask/**
- .github/workflows/sdk-foundation-models-system-evaluation-fill-mask-fill-mask.yml
- sdk/python/dev-requirements.txt
- infra/**
- sdk/python/setup.sh
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: check out repo
uses: actions/checkout@v2
- name: setup python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: pip install notebook reqs
run: pip install -r sdk/python/dev-requirements.txt
- name: pip install mlflow reqs
run: pip install -r sdk/python/mlflow-requirements.txt
- name: azure login
uses: azure/login@v1
with:
creds: ${{secrets.AZUREML_CREDENTIALS}}
- name: bootstrap resources
run: |
echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
bash bootstrap.sh
working-directory: infra
continue-on-error: false
- name: setup SDK
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: sdk/python
continue-on-error: true
- name: setup-cli
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: cli
continue-on-error: true
- name: run foundation-models/system/evaluation/fill-mask/fill-mask.ipynb
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" replace_template_values "fill-mask.ipynb";
[ -f "../../.azureml/config" ] && cat "../../.azureml/config";
papermill -k python fill-mask.ipynb fill-mask.output.ipynb
working-directory: sdk/python/foundation-models/system/evaluation/fill-mask
- name: upload notebook's working folder as an artifact
if: ${{ always() }}
uses: actions/upload-artifact@v2
with:
name: fill-mask
path: sdk/python/foundation-models/system/evaluation/fill-mask
77 changes: 77 additions & 0 deletions ...rkflows/sdk-foundation-models-system-evaluation-question-answering-question-answering.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: sdk-foundation-models-system-evaluation-question-answering-question-answering
# This file is created by sdk/python/readme.py.
# Please do not edit directly.
on:
workflow_dispatch:
schedule:
- cron: "12 3/12 * * *"
pull_request:
branches:
- main
paths:
- sdk/python/foundation-models/system/evaluation/question-answering/**
- .github/workflows/sdk-foundation-models-system-evaluation-question-answering-question-answering.yml
- sdk/python/dev-requirements.txt
- infra/**
- sdk/python/setup.sh
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: check out repo
uses: actions/checkout@v2
- name: setup python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: pip install notebook reqs
run: pip install -r sdk/python/dev-requirements.txt
- name: pip install mlflow reqs
run: pip install -r sdk/python/mlflow-requirements.txt
- name: azure login
uses: azure/login@v1
with:
creds: ${{secrets.AZUREML_CREDENTIALS}}
- name: bootstrap resources
run: |
echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
bash bootstrap.sh
working-directory: infra
continue-on-error: false
- name: setup SDK
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: sdk/python
continue-on-error: true
- name: setup-cli
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: cli
continue-on-error: true
- name: run foundation-models/system/evaluation/question-answering/question-answering.ipynb
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" replace_template_values "question-answering.ipynb";
[ -f "../../.azureml/config" ] && cat "../../.azureml/config";
papermill -k python question-answering.ipynb question-answering.output.ipynb
working-directory: sdk/python/foundation-models/system/evaluation/question-answering
- name: upload notebook's working folder as an artifact
if: ${{ always() }}
uses: actions/upload-artifact@v2
with:
name: question-answering
path: sdk/python/foundation-models/system/evaluation/question-answering
77 changes: 77 additions & 0 deletions ...ation-models-system-evaluation-summarization-abstractive-and-extractive-summarization.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: sdk-foundation-models-system-evaluation-summarization-abstractive-and-extractive-summarization
# This file is created by sdk/python/readme.py.
# Please do not edit directly.
on:
workflow_dispatch:
schedule:
- cron: "28 9/12 * * *"
pull_request:
branches:
- main
paths:
- sdk/python/foundation-models/system/evaluation/summarization/**
- .github/workflows/sdk-foundation-models-system-evaluation-summarization-abstractive-and-extractive-summarization.yml
- sdk/python/dev-requirements.txt
- infra/**
- sdk/python/setup.sh
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: check out repo
uses: actions/checkout@v2
- name: setup python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: pip install notebook reqs
run: pip install -r sdk/python/dev-requirements.txt
- name: pip install mlflow reqs
run: pip install -r sdk/python/mlflow-requirements.txt
- name: azure login
uses: azure/login@v1
with:
creds: ${{secrets.AZUREML_CREDENTIALS}}
- name: bootstrap resources
run: |
echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
bash bootstrap.sh
working-directory: infra
continue-on-error: false
- name: setup SDK
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: sdk/python
continue-on-error: true
- name: setup-cli
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: cli
continue-on-error: true
- name: run foundation-models/system/evaluation/summarization/abstractive-and-extractive-summarization.ipynb
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" replace_template_values "abstractive-and-extractive-summarization.ipynb";
[ -f "../../.azureml/config" ] && cat "../../.azureml/config";
papermill -k python abstractive-and-extractive-summarization.ipynb abstractive-and-extractive-summarization.output.ipynb
working-directory: sdk/python/foundation-models/system/evaluation/summarization
- name: upload notebook's working folder as an artifact
if: ${{ always() }}
uses: actions/upload-artifact@v2
with:
name: abstractive-and-extractive-summarization
path: sdk/python/foundation-models/system/evaluation/summarization
77 changes: 77 additions & 0 deletions .../sdk-foundation-models-system-evaluation-text-classification-entailment-contradiction.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: sdk-foundation-models-system-evaluation-text-classification-entailment-contradiction
# This file is created by sdk/python/readme.py.
# Please do not edit directly.
on:
workflow_dispatch:
schedule:
- cron: "56 5/12 * * *"
pull_request:
branches:
- main
paths:
- sdk/python/foundation-models/system/evaluation/text-classification/**
- .github/workflows/sdk-foundation-models-system-evaluation-text-classification-entailment-contradiction.yml
- sdk/python/dev-requirements.txt
- infra/**
- sdk/python/setup.sh
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: check out repo
uses: actions/checkout@v2
- name: setup python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: pip install notebook reqs
run: pip install -r sdk/python/dev-requirements.txt
- name: pip install mlflow reqs
run: pip install -r sdk/python/mlflow-requirements.txt
- name: azure login
uses: azure/login@v1
with:
creds: ${{secrets.AZUREML_CREDENTIALS}}
- name: bootstrap resources
run: |
echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
bash bootstrap.sh
working-directory: infra
continue-on-error: false
- name: setup SDK
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: sdk/python
continue-on-error: true
- name: setup-cli
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: cli
continue-on-error: true
- name: run foundation-models/system/evaluation/text-classification/entailment-contradiction.ipynb
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" replace_template_values "entailment-contradiction.ipynb";
[ -f "../../.azureml/config" ] && cat "../../.azureml/config";
papermill -k python entailment-contradiction.ipynb entailment-contradiction.output.ipynb
working-directory: sdk/python/foundation-models/system/evaluation/text-classification
- name: upload notebook's working folder as an artifact
if: ${{ always() }}
uses: actions/upload-artifact@v2
with:
name: entailment-contradiction
path: sdk/python/foundation-models/system/evaluation/text-classification
77 changes: 77 additions & 0 deletions ...kflows/sdk-foundation-models-system-evaluation-text-classification-sentiment-analysis.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: sdk-foundation-models-system-evaluation-text-classification-sentiment-analysis
# This file is created by sdk/python/readme.py.
# Please do not edit directly.
on:
workflow_dispatch:
schedule:
- cron: "20 7/12 * * *"
pull_request:
branches:
- main
paths:
- sdk/python/foundation-models/system/evaluation/text-classification/**
- .github/workflows/sdk-foundation-models-system-evaluation-text-classification-sentiment-analysis.yml
- sdk/python/dev-requirements.txt
- infra/**
- sdk/python/setup.sh
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: check out repo
uses: actions/checkout@v2
- name: setup python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: pip install notebook reqs
run: pip install -r sdk/python/dev-requirements.txt
- name: pip install mlflow reqs
run: pip install -r sdk/python/mlflow-requirements.txt
- name: azure login
uses: azure/login@v1
with:
creds: ${{secrets.AZUREML_CREDENTIALS}}
- name: bootstrap resources
run: |
echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
bash bootstrap.sh
working-directory: infra
continue-on-error: false
- name: setup SDK
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: sdk/python
continue-on-error: true
- name: setup-cli
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash setup.sh
working-directory: cli
continue-on-error: true
- name: run foundation-models/system/evaluation/text-classification/sentiment-analysis.ipynb
run: |
source "${{ github.workspace }}/infra/sdk_helpers.sh";
source "${{ github.workspace }}/infra/init_environment.sh";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
bash "${{ github.workspace }}/infra/sdk_helpers.sh" replace_template_values "sentiment-analysis.ipynb";
[ -f "../../.azureml/config" ] && cat "../../.azureml/config";
papermill -k python sentiment-analysis.ipynb sentiment-analysis.output.ipynb
working-directory: sdk/python/foundation-models/system/evaluation/text-classification
- name: upload notebook's working folder as an artifact
if: ${{ always() }}
uses: actions/upload-artifact@v2
with:
name: sentiment-analysis
path: sdk/python/foundation-models/system/evaluation/text-classification
Loading

0 comments on commit d3bf478

Please sign in to comment.