chore: Remove MVAD's dependence on hardwired credentials and azure SDKs #1629
Conversation
|
Hey @mhamilton723 👋! |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1629 +/- ##
=======================================
Coverage 82.81% 82.81%
=======================================
Files 286 286
Lines 15064 15064
Branches 745 745
=======================================
Hits 12476 12476
Misses 2588 2588 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
mhamilton723
force-pushed
the
remove-mvad-blob
branch
from
3c908f2
to
9cd485c
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| val intermediateSaveDir = new Param[String](this, "intermediateSaveDir", "Directory name " + | ||
| "of which you want to save the intermediate data produced while training.") | ||
| private def validateIntermediateSaveDir(dir: String): Boolean = { | ||
| assert(dir.startsWith("wasbs://"), "improper HDFS loacation. Please use a wasb path such as: \n" + |
There was a problem hiding this comment.
Is only wasbs accepted? What about abfss and say in Trident some other protocols?
There was a problem hiding this comment.
Added abfss too
| private def getStorageInfo: StorageInfo = { | ||
| val uri = new URI(getIntermediateSaveDir) | ||
| val account = uri.getHost.split(".".toCharArray).head | ||
| val config = s"fs.azure.account.key.$account.blob.core.windows.net" |
There was a problem hiding this comment.
Does this config also work on databricks?
| val errors = summary.errors.get.toJson.compactPrint | ||
| throw new RuntimeException(s"Caught errors during inference: $errors") | ||
| throw new RuntimeException(s"Could not get trained model: $errors") |
There was a problem hiding this comment.
This is not can't get trained model error I guess, it should be inference failure?
| @@ -397,6 +348,19 @@ trait MADBase extends HasOutputCol | |||
| |""".stripMargin | |||
| } | |||
|
|
|||
| protected def submitDataset(dataset: Dataset[_]): Map[String, JsValue] = { | |||
There was a problem hiding this comment.
Can we rename this method as it does both submitDataset and send request to MVAD?
mhamilton723
force-pushed
the
remove-mvad-blob
branch
from
9cd485c
to
20ea5e0
Compare
mhamilton723
force-pushed
the
remove-mvad-blob
branch
from
20ea5e0
to
58ca0e1
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Related Issues/PRs
#xxx
What changes are proposed in this pull request?
Briefly describe the changes included in this Pull Request.
How is this patch tested?
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?
website/docs/documentationfolder.Make sure you choose the correct class
estimators/transformersand namespace.DocTablepoints to correct API link.yarn run startto make sure the website renders correctly.<!--pytest-codeblocks:cont-->before each python code blocks to enable auto-tests for python samples.WebsiteSamplesTestsjob pass in the pipeline.AB#1944709