test: Additional E2E testing infrastructure #1727
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hey @k-rush 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
...st/scala/com/microsoft/azure/synapse/ml/nbtest/SynapseExtension/SynapseExtensionsTests.scala
Outdated
Show resolved
Hide resolved
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1727 +/- ##
==========================================
+ Coverage 85.81% 85.85% +0.03%
==========================================
Files 278 278
Lines 14604 14604
Branches 765 765
==========================================
+ Hits 12533 12538 +5
+ Misses 2071 2066 -5
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
…sh/SynapseML into kyrush/synapseinternaltests
k-rush
changed the title
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
mhamilton723
added a commit
that referenced
this pull request
Dec 1, 2022
* docs: fix command to launch jupyter notebook (#1649) * docs: add aisample uplift modelling (#1640) * add uplift model sample * update uplift model notebook * Update notebooks/community/aisamples/AIsample - Uplift Modelling.ipynb Co-authored-by: weitian <weitian@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * chore: publish test jars * docs: improve example notebooks * docs: remove unused docs and fix links * docs: move magic command forward since it restarts interpreter * chore: added `synapse-internal` to platform detector function (#1651) * chore: added trident platform * chore: update naming * Update core/src/main/python/synapse/ml/core/platform/Platform.py * chore: update detection rules * chore: remove redundant check * docs: apply black formatter Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * fix: KernelSHAP throws error when the key type in the ZipMap output is LongType (#1656) Signed-off-by: Jason Wang <jasowang@microsoft.com> Signed-off-by: Jason Wang <jasowang@microsoft.com> * chore: turn off failing synapse tests temporarily (#1658) * docs: simplify data downloading and add mlflow to uplift modelling (#1659) * chore: simplify data downloading section update save dataframe with delta format * docs: add mlflow logging and loading * Update notebooks/community/aisamples/AIsample - Uplift Modelling.ipynb Co-authored-by: Li Jiang <lijiang1@microsoft.com> * fix: fix Uplift Modelling style * docs: improve error msg to make it clearer for users and fix typos (#1662) * docs: improve error msg to make it clearer for users. * fix: fix typos Co-authored-by: Li Jiang <lijiang1@microsoft.com> * docs: fix multiple typos and update error hintings in ai-samples-timeseries notebook (#1663) Co-authored-by: Serena Ruan <82044803+serena-ruan@users.noreply.github.com> * chore: clean up github workflows and add issue label remover (#1674) * chore: re open github issues after a comment (#1676) * chore: fix typo in issue reopen yaml * chore: fix reopen on comment workflow * chore: fix reopen comment action * Update reopen-issue-on-comment.yml * build: bump amannn/action-semantic-pull-request from 4 to 5.0.1 (#1680) Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 4 to 5.0.1. - [Release notes](https://github.com/amannn/action-semantic-pull-request/releases) - [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md) - [Commits](amannn/action-semantic-pull-request@v4...v5.0.1) --- updated-dependencies: - dependency-name: amannn/action-semantic-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: fix linked service on cog service base (#1685) * fix: fix pyarrow failure in deeplearning test (#1689) * build: bump amannn/action-semantic-pull-request from 5.0.1 to 5.0.2 (#1688) Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 5.0.1 to 5.0.2. - [Release notes](https://github.com/amannn/action-semantic-pull-request/releases) - [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md) - [Commits](amannn/action-semantic-pull-request@v5.0.1...v5.0.2) --- updated-dependencies: - dependency-name: amannn/action-semantic-pull-request dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * docs: update developer readme instruction on python env creation (#1693) * update developer readme instruction on python env creation * Update developer-readme.md * Update developer-readme.md * fix: don't throw on invalid columns in DropColumns (#1695) * remove unused imports * batch prompts * don't throw for invalid columns in DropColumn * remove now unused verify method from DropColumns * Remove trident.mlflow APIs. (#1687) Co-authored-by: Li Jiang <lijiang1@microsoft.com> * using predictionCol for isolation forest (#1686) fixing issue #1060 Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * fix: update isolation forest notebook (#1696) * fix: remove synapse E2E testing exclusion - cyber ml (#1699) * remove CyberML exclusion from Synapse test pipeline * black style Co-authored-by: Jessica Wang <jessiwang@microsoft.com> * chore: remove notebooks (#1703) * chore: fix ado integration (#1704) * chore: pin az and python versions (#1705) * chore: remove synapse test exclusions (#1698) * chore: update scalatest and scalactic (#1706) * remove unused imports * batch prompts * don't throw for invalid columns in DropColumn * remove now unused verify method from DropColumns * update scalatest and scalactic * strip unicode out of transliteration result * clean transliterate test * override assertDFEq in TransliterateSuite to strip out zero-width chars * fix: remove Vowpal Wabbit exclusion, add Interpretability exclusion (#1708) * remove Vowpal Wabbit exclusion, add Interpretability exclusion * exclude all Interpretability Notebooks from Synapse E2E Co-authored-by: Jessica Wang <jessiwang@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * feat: Remove CNTK functionality and replace with ONNX (#1593) * Switch usage of CNTK to use ONNX * style fix * small test fixes * resolved comments * some test fixes * test fixes and comment responses * more test fixes * fix dbx tests * fix notebooks * misc small fixes * Switch usage of CNTK to use ONNX * style fix * small test fixes * resolved comments * some test fixes * test fixes and comment responses * more test fixes * fix dbx tests * fix notebooks * misc small fixes * test fix * dotnet fix * add protobuf to sbt * notebook style fixes * add protobuf to databricks * revert dependency * dependency override * add shading * more shading * add maven shade plugin * add shade to subproject * switch to using separate onnx dependency * increase onnx version * add ignoreDecodingErrors * fix decoding errors param * fix style error * changed to sonatype package * add snapshots repo to tests - revert later * downgrade package to Java 8 * more snapshot repo refs * databricks onnx version fix * add error comment * centralize package constants * add debugging for image name * debugging image name * add conversion of black and white * style fix * fix to b&w usage * cherry pick python version fix * throw if only 1 channel * cli fix * cleanup of review 1 * fix typo * responded to comments * remove explicit dependency * responded to comments Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * build: bump loader-utils from 2.0.2 to 2.0.3 in /website (#1709) Bumps [loader-utils](https://github.com/webpack/loader-utils) from 2.0.2 to 2.0.3. - [Release notes](https://github.com/webpack/loader-utils/releases) - [Changelog](https://github.com/webpack/loader-utils/blob/v2.0.3/CHANGELOG.md) - [Commits](webpack/loader-utils@v2.0.2...v2.0.3) --- updated-dependencies: - dependency-name: loader-utils dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * chore: ScalaStyle fixes (#1716) * Style fixes * Style fixes * Style fixes in tests * test style fixes * make some warnings errors * make some warnings errors * make some warnings errors * make some warnings errors * fix vw * fixes from review pass * test: Improve ONNXtests reliability (#1713) * Improve ONNXtests reliability * fix constructor build * make new defaults * chore: Move new ImageFeaturizer to onnx namespace (#1711) * feat: Deprecate CNTK objects (#1712) * Deprecate CNTK objects * remove CNTK-based notebooks * ignore 1 failing test * deprecate more util classes * chore: add secret scanning infrastructure (#1724) * feat: Add SpeakerEmotionInference transformer for generating SSML t… (#1691) * feat: Add TextToSpeechSSMLGenerator transformer for generating SSML to augment TTS requestsfeat: Add TextToSpeechSSMLGenerator transformer for generating SSML to augment TTS requests. Adding support for SSML to the TTS endpoint. * Fixing style issues * More style issues * ok it finally passes scalastyle Co-authored-by: Brendan Walsh <brwals@outlook.com> * test: Additional E2E testing infrastructure (#1727) * Addition E2E testing infrastructure * Ignore cleanup test * Style fix * Update platform.py Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * chore: Making secrets optional and cached (#1726) * Making secrets optional and cached * do not cache normal secrets * set publishing for dotnet * more dotnet pipeline edits * testing dotnet no env vars * enable publish only Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * chore: autodelete old models (#1729) * autodelete old models * move cleanup calls to afterAll * fix style Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * clarify date comparisons when deleting old models/groups (#1733) * chore: automate clean-acr with github action workflow (#1735) * hello workflow * make pre-commit executable * clean acr * gh-set-secret * pat->github * echo * env * Create gh-set-secret.yml * Update gh-set-secret.yml * Update gh-set-secret.yml * Create manual.yml * Update manual.yml * Update manual.yml * Update manual.yml * Update manual.yml * use azurecli for keyvault access * remove pip cache * remove columns * fix indent * fix env var name * split off script file * azurecli@v1 * shorten path * lengthen path * add query option to az cmd * re-indent * re-indent again * echo * print * test maniehtestkv * back to azure kv task * back to mmlspark-keys * quot arg * typo * use popen for pipeline-run * run through deletions in whatif mode * print result code * delete result code * format changes and check result of transfer * remove tqdn * remove tqdn * restore actual deletion * formatize prints * delete cruft * switch from manual to cron * sundays at 1am * chmod pre-commit Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * feat: add simple deep learning text classifier (#1591) * refactor deep vision model params * add Text Classifier and tests * update text classifier * add default values for transformation edit fields and removed fields * add deep text classification notebook * update hovorod installation script * update environment * update env * add installing packages on dbx * fix python environment * add more tests * skip deep text model test * address comments * add learning_rate param * fix notebook style * fix _find_num_proc * update newtonsoft.json version in dotnet to resolve security issue * fix missing learning rate param * fix dataframe partition error and strange read output * fix failing test * fix param updates * update notebook to make it run faster * make train data size smaller * update models * remove minor version of python to avoid warnings in pipeline * ignore dl notebooks in Synapse tests * update mardown name * chore: fix style (#1736) * build: bump actions/checkout from 2 to 3 (#1737) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump version to 0.10.2 (#1738) * docs: removing beta tag from R * build: bump loader-utils from 2.0.3 to 2.0.4 in /website (#1719) Bumps [loader-utils](https://github.com/webpack/loader-utils) from 2.0.3 to 2.0.4. - [Release notes](https://github.com/webpack/loader-utils/releases) - [Changelog](https://github.com/webpack/loader-utils/blob/v2.0.4/CHANGELOG.md) - [Commits](webpack/loader-utils@v2.0.3...v2.0.4) --- updated-dependencies: - dependency-name: loader-utils dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> * chore: bump docusaurus (#1740) * chore: bump spark to 3.2.3 (#1744) * downgrade spark 3.1 * remove exlusion of org.json4s * fix json4s-ast version * resolve dependency conflicts * remove notebook * fix environment * fix onnxhub load method and add error message in platform.py * update platform.py Signed-off-by: Jason Wang <jasowang@microsoft.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: elswork <elswork@gmail.com> Co-authored-by: Tian Wei <12080578+whiskyboy@users.noreply.github.com> Co-authored-by: weitian <weitian@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com> Co-authored-by: lhrotk <lhrotk@gmail.com> Co-authored-by: Li Jiang <bnujli@gmail.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Jason Wang <jasowang@microsoft.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: niehaus59 <41398747+niehaus59@users.noreply.github.com> Co-authored-by: Markus Cozowicz <marcozo@microsoft.com> Co-authored-by: JessicaXYWang <108437381+JessicaXYWang@users.noreply.github.com> Co-authored-by: Jessica Wang <jessiwang@microsoft.com> Co-authored-by: Scott Votaw <svotaw@gmail.com> Co-authored-by: Brendan Walsh <37676373+BrendanWalsh@users.noreply.github.com> Co-authored-by: Brendan Walsh <brwals@outlook.com> Co-authored-by: Kyle Rush <kyrush@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Adds some more e2e test coverage to internal platform.
How is this patch tested
Tested locally during development
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?