chore: Downgrade spark3.1 #1747

serena-ruan · 2022-11-30T13:06:10Z

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Briefly describe the changes included in this Pull Request.

How is this patch tested?

I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change any dependencies?

No. You can skip this section.
Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

No. You can skip this section.
Yes. Make sure you have added samples following below steps.

Find the corresponding markdown file for your new feature in website/docs/documentation folder.
Make sure you choose the correct class estimators/transformers and namespace.
Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
Make sure the DocTable points to correct API link.
Navigate to website folder, and run yarn run start to make sure the website renders correctly.
Don't forget to add  before each python code blocks to enable auto-tests for python samples.
Make sure the WebsiteSamplesTests job pass in the pipeline.

* add uplift model sample * update uplift model notebook * Update notebooks/community/aisamples/AIsample - Uplift Modelling.ipynb Co-authored-by: weitian <weitian@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

…oft#1651) * chore: added trident platform * chore: update naming * Update core/src/main/python/synapse/ml/core/platform/Platform.py * chore: update detection rules * chore: remove redundant check * docs: apply black formatter Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

…s LongType (microsoft#1656) Signed-off-by: Jason Wang <jasowang@microsoft.com> Signed-off-by: Jason Wang <jasowang@microsoft.com>

…icrosoft#1659) * chore: simplify data downloading section update save dataframe with delta format * docs: add mlflow logging and loading * Update notebooks/community/aisamples/AIsample - Uplift Modelling.ipynb Co-authored-by: Li Jiang <lijiang1@microsoft.com>

…icrosoft#1662) * docs: improve error msg to make it clearer for users. * fix: fix typos Co-authored-by: Li Jiang <lijiang1@microsoft.com>

…series notebook (microsoft#1663) Co-authored-by: Serena Ruan <82044803+serena-ruan@users.noreply.github.com>

…ft#1674)

…osoft#1680) Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 4 to 5.0.1. - [Release notes](https://github.com/amannn/action-semantic-pull-request/releases) - [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md) - [Commits](amannn/action-semantic-pull-request@v4...v5.0.1) --- updated-dependencies: - dependency-name: amannn/action-semantic-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…icrosoft#1688) Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 5.0.1 to 5.0.2. - [Release notes](https://github.com/amannn/action-semantic-pull-request/releases) - [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md) - [Commits](amannn/action-semantic-pull-request@v5.0.1...v5.0.2) --- updated-dependencies: - dependency-name: amannn/action-semantic-pull-request dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

…rosoft#1693) * update developer readme instruction on python env creation * Update developer-readme.md * Update developer-readme.md

* remove unused imports * batch prompts * don't throw for invalid columns in DropColumn * remove now unused verify method from DropColumns

Co-authored-by: Li Jiang <lijiang1@microsoft.com>

fixing issue microsoft#1060 Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

* remove CyberML exclusion from Synapse test pipeline * black style Co-authored-by: Jessica Wang <jessiwang@microsoft.com>

github-actions · 2022-11-30T13:06:22Z

Hey @serena-ruan 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.
We appreciate your patience and contributions 💯!

serena-ruan · 2022-11-30T13:07:50Z

/azp run

azure-pipelines · 2022-11-30T13:08:03Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov-commenter · 2022-11-30T13:37:45Z

Codecov Report

Merging #1747 (ccec9c7) into spark3.1 (7abf839) will decrease coverage by 0.00%.
The diff coverage is 88.13%.

@@             Coverage Diff              @@
##           spark3.1    #1747      +/-   ##
============================================
- Coverage     85.83%   85.83%   -0.01%     
============================================
  Files           272      278       +6     
  Lines         14230    14608     +378     
  Branches        767      769       +2     
============================================
+ Hits          12215    12539     +324     
- Misses         2015     2069      +54

Impacted Files	Coverage Δ
...t/azure/synapse/ml/cognitive/BingImageSearch.scala	`89.28% <ø> (-0.90%)`	⬇️
...ft/azure/synapse/ml/cognitive/ComputerVision.scala	`90.37% <ø> (+0.37%)`	⬆️
.../azure/synapse/ml/cognitive/FormRecognizerV3.scala	`88.00% <ø> (ø)`
...oft/azure/synapse/ml/cognitive/SpeechSchemas.scala	`100.00% <ø> (ø)`
...oft/azure/synapse/ml/cognitive/TextAnalytics.scala	`83.73% <ø> (-2.78%)`	⬇️
...soft/azure/synapse/ml/automl/EvaluationUtils.scala	`48.00% <ø> (ø)`
...microsoft/azure/synapse/ml/codegen/PyCodegen.scala	`90.69% <ø> (ø)`
...soft/azure/synapse/ml/core/utils/ClusterUtil.scala	`65.75% <0.00%> (ø)`
...zure/synapse/ml/explainers/KernelSHAPSampler.scala	`97.05% <ø> (ø)`
...crosoft/azure/synapse/ml/explainers/LIMEBase.scala	`92.30% <ø> (ø)`
... and 104 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

serena-ruan · 2022-11-30T13:48:47Z

/azp run

azure-pipelines · 2022-11-30T13:49:00Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2022-11-30T14:56:13Z

/azp run

azure-pipelines · 2022-11-30T14:56:27Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2022-12-01T08:56:30Z

/azp run

azure-pipelines · 2022-12-01T08:56:44Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2022-12-01T10:55:27Z

/azp run

azure-pipelines · 2022-12-01T10:55:40Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2022-12-01T12:38:43Z

/azp run

azure-pipelines · 2022-12-01T12:38:56Z

Azure Pipelines successfully started running 1 pipeline(s).

serena-ruan · 2022-12-01T13:08:59Z

/azp run

azure-pipelines · 2022-12-01T13:09:13Z

Azure Pipelines successfully started running 1 pipeline(s).

elswork and others added 30 commits September 12, 2022 15:10

docs: fix command to launch jupyter notebook (microsoft#1649)

c8750ce

docs: add aisample uplift modelling (microsoft#1640)

b7c8cf1

* add uplift model sample * update uplift model notebook * Update notebooks/community/aisamples/AIsample - Uplift Modelling.ipynb Co-authored-by: weitian <weitian@microsoft.com> Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

chore: publish test jars

a7d097a

docs: improve example notebooks

d5a4997

docs: remove unused docs and fix links

3a160b3

docs: move magic command forward since it restarts interpreter

ca358e3

fix: KernelSHAP throws error when the key type in the ZipMap output i…

76d7382

…s LongType (microsoft#1656) Signed-off-by: Jason Wang <jasowang@microsoft.com> Signed-off-by: Jason Wang <jasowang@microsoft.com>

chore: turn off failing synapse tests temporarily (microsoft#1658)

95f451a

fix: fix Uplift Modelling style

dd9e5d2

docs: improve error msg to make it clearer for users and fix typos (m…

4fa7249

…icrosoft#1662) * docs: improve error msg to make it clearer for users. * fix: fix typos Co-authored-by: Li Jiang <lijiang1@microsoft.com>

docs: fix multiple typos and update error hintings in ai-samples-time…

965231a

…series notebook (microsoft#1663) Co-authored-by: Serena Ruan <82044803+serena-ruan@users.noreply.github.com>

chore: clean up github workflows and add issue label remover (microso…

8914750

…ft#1674)

chore: re open github issues after a comment (microsoft#1676)

a7c50de

chore: fix typo in issue reopen yaml

9fe273b

chore: fix reopen on comment workflow

b3a9ba9

chore: fix reopen comment action

c9278b5

Update reopen-issue-on-comment.yml

50ac0c8

fix: fix linked service on cog service base (microsoft#1685)

e857511

fix: fix pyarrow failure in deeplearning test (microsoft#1689)

302831f

docs: update developer readme instruction on python env creation (mic…

c531bbb

…rosoft#1693) * update developer readme instruction on python env creation * Update developer-readme.md * Update developer-readme.md

fix: don't throw on invalid columns in DropColumns (microsoft#1695)

f4af33f

* remove unused imports * batch prompts * don't throw for invalid columns in DropColumn * remove now unused verify method from DropColumns

Remove trident.mlflow APIs. (microsoft#1687)

448f6b7

Co-authored-by: Li Jiang <lijiang1@microsoft.com>

using predictionCol for isolation forest (microsoft#1686)

9120b05

fixing issue microsoft#1060 Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

fix: update isolation forest notebook (microsoft#1696)

b257c70

fix: remove synapse E2E testing exclusion - cyber ml (microsoft#1699)

a953780

* remove CyberML exclusion from Synapse test pipeline * black style Co-authored-by: Jessica Wang <jessiwang@microsoft.com>

chore: remove notebooks (microsoft#1703)

92d4095

serena-ruan requested review from memoryz, eisber, imatiach-msft and mhamilton723 as code owners November 30, 2022 13:06

Merge branch 'spark3.1' into downgradeSpark3.1

c0ad64e

mhamilton723 approved these changes Nov 30, 2022

View reviewed changes

remove exlusion of org.json4s

9b04f61

fix json4s-ast version

f6a3818

resolve dependency conflicts

a58b879

serena-ruan added 2 commits December 1, 2022 17:58

remove notebook

1f5e282

fix environment

15d457a

fix onnxhub load method and add error message in platform.py

9b79650

update platform.py

ccec9c7

mhamilton723 merged commit fe224d3 into microsoft:spark3.1 Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Downgrade spark3.1 #1747

chore: Downgrade spark3.1 #1747

serena-ruan commented Nov 30, 2022

github-actions bot commented Nov 30, 2022

serena-ruan commented Nov 30, 2022

azure-pipelines bot commented Nov 30, 2022

codecov-commenter commented Nov 30, 2022 •

edited

Loading

serena-ruan commented Nov 30, 2022

azure-pipelines bot commented Nov 30, 2022

serena-ruan commented Nov 30, 2022

azure-pipelines bot commented Nov 30, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

chore: Downgrade spark3.1 #1747

chore: Downgrade spark3.1 #1747

Conversation

serena-ruan commented Nov 30, 2022

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change any dependencies?

Does this PR add a new feature? If so, have you added samples on website?

github-actions bot commented Nov 30, 2022

serena-ruan commented Nov 30, 2022

azure-pipelines bot commented Nov 30, 2022

codecov-commenter commented Nov 30, 2022 • edited Loading

Codecov Report

serena-ruan commented Nov 30, 2022

azure-pipelines bot commented Nov 30, 2022

serena-ruan commented Nov 30, 2022

azure-pipelines bot commented Nov 30, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

serena-ruan commented Dec 1, 2022

azure-pipelines bot commented Dec 1, 2022

codecov-commenter commented Nov 30, 2022 •

edited

Loading