Fix order of precedence for AML workspace references in submit.py (#209)

* refactor components to use dpv2 + remove unnecessary environments

* working dpv2 pipeline

* refactor scripts with right inputs and outputs

* fix code path

* implement fake outputs

* fix paths

* fix imports

* fix args of aggregation script

* add note, fix component args

* add chekcpoint arg

* linting

* linting

* remove sdkv2 folder

* add argparse to submit script

* add docstring

* add docstring

* linting

* linting

* add staging branch to build

* rollback changes to build, leave it for another PR

* remove logging lien

* remove custom uuid

* linting

* add docstring to custom path function

* polish docstring

* rename model_silo_X to input_silo_X

* rename output

* rename agg output

* Improve auto-provisioning resources (#35) (#36)

* docker file stub

* move docker file, implement feedback

* login before setting subscription

* login before setting subscription

* use default k8s version

* pin latest version since default won't work

* remove executionpolicy part, other small updates

* clarify to change job file _in docker filesystem_

* login before setting subscription

* formatting

* \ -> /

* install azureml-core in docker file

* propagate changes to section 7

* fix dataset creation command

Co-authored-by: thomasp-ms <XXX@me.com>

Co-authored-by: thomasp-ms <XXX@me.com>

* Refactor folder structure (#37)

* `plan` -> `docs`

* 'plan' -> 'docs'

* 'automated_provisioning' -> 'mlops'

* 'fl_arc_k8s' -> 'examples'

Co-authored-by: thomasp-ms <XXX@me.com>

* auto provisioning - vanilla internal silos (#41)

* split internal and external provisioning

* adjust directories after internal/external split

* introduce overall mlops readme

* first stab

* remove useless comment and my alias

Co-authored-by: thomasp-ms <XXX@me.com>

* Perform real FL training on the MNIST dataset

Added component files customized for MNIST dataset. Set the setup for 3
silo having their own compute and datastore.
git config --global user.email "you@example.com"

* refine components and add logs

* maintain consistency b/w config files

* add requirement and env files

* add requirement and env files

* rmv redundant dependencies, rename conda envs

* Correct epoch default value

* point data asset instead of underlying URI

* beef up orchestrator cluster (#46)

Co-authored-by: thomasp-ms <XXX@me.com>

* Provision CPUs for silos (instead of GPUs) (#47)

* beef up orchestrator cluster

* gpu -> cpu

Co-authored-by: thomasp-ms <XXX@me.com>

* add preprocessing comp description, fix typo and correct default datastore name

* add integration validation test - build

* update readme file

* Move logger to the maion if block, add pytorch channel in the conda env
yaml and move readme to the docs folder

* code reformatting using black

* add documentation to run an FL experiment

* add more intuitive path for aggr output dir

* Merge changes

* fix order of precedence

* fix build

Co-authored-by: Jeff Omhover <jeomhove@microsoft.com>
Co-authored-by: Jeff Omhover <jf.omhover@gmail.com>
Co-authored-by: Thomas <7998422+thomasp-ms@users.noreply.github.com>
Co-authored-by: thomasp-ms <XXX@me.com>

examples/pipelines/ccfraud/config.yaml

@@ -2,9 +2,9 @@
 
 # using this to store references to Azure ML
 aml:
-  subscription_id: "<SUBSCRIPTION_ID>"
-  resource_group_name: "<RESOURCE_GROUP>"
-  workspace_name: "<AML_WORKSPACE_NAME>"
+  # subscription_id: "<SUBSCRIPTION_ID>"
+  # resource_group_name: "<RESOURCE_GROUP>"
+  # workspace_name: "<AML_WORKSPACE_NAME>"
 
 # federated learning parameters
 federated_learning:

examples/pipelines/ccfraud/submit.py

@@ -105,14 +105,6 @@ def connect_to_aml():
 
     # Get a handle to workspace
     try:
-        # tries to connect using local config.json
-        ML_CLIENT = MLClient.from_config(credential=credential)
-
-    except Exception as ex:
-        print(
-            "Could not find config.json, using config.yaml refs to Azure ML workspace instead."
-        )
-
         # tries to connect using cli args if provided else using config.yaml
         ML_CLIENT = MLClient(
             subscription_id=args.subscription_id or YAML_CONFIG.aml.subscription_id,
@@ -121,10 +113,14 @@ def connect_to_aml():
             workspace_name=args.workspace_name or YAML_CONFIG.aml.workspace_name,
             credential=credential,
         )
-    return ML_CLIENT
 
+    except Exception as ex:
+        print("Could not find either cli args or config.yaml.")
+        # tries to connect using local config.json
+        ML_CLIENT = MLClient.from_config(credential=credential)
+
+    return ML_CLIENT
 
-ML_CLIENT = connect_to_aml()
 
 ####################################
 ### LOAD THE PIPELINE COMPONENTS ###
@@ -335,6 +331,7 @@ def fl_ccfraud_basic():
 if args.submit:
     print("Submitting the pipeline job to your AzureML workspace...")
 
+    ML_CLIENT = connect_to_aml()
     pipeline_job = ML_CLIENT.jobs.create_or_update(
         pipeline_job, experiment_name="fl_demo_ccfraud"
     )

examples/pipelines/fl_cross_silo_factory/config.yaml

@@ -12,9 +12,9 @@
 
 # References to Azure ML workspace (use cli args to override)
 aml:
-  subscription_id: "<SUBSCRIPTION_ID>"
-  resource_group_name: "<RESOURCE_GROUP>"
-  workspace_name: "<AML_WORKSPACE_NAME>"
+  # subscription_id: "<SUBSCRIPTION_ID>"
+  # resource_group_name: "<RESOURCE_GROUP>"
+  # workspace_name: "<AML_WORKSPACE_NAME>"
 
 # Parameters to generate the FL graph
 federated_learning:

examples/pipelines/fl_cross_silo_factory/submit.py

@@ -144,14 +144,6 @@ def connect_to_aml():
 
     # Get a handle to workspace
     try:
-        # tries to connect using local config.json
-        ML_CLIENT = MLClient.from_config(credential=credential)
-
-    except Exception as ex:
-        print(
-            "Could not find config.json, using config.yaml refs to Azure ML workspace instead."
-        )
-
         # tries to connect using cli args if provided else using config.yaml
         ML_CLIENT = MLClient(
             subscription_id=args.subscription_id or YAML_CONFIG.aml.subscription_id,
@@ -160,10 +152,13 @@ def connect_to_aml():
             workspace_name=args.workspace_name or YAML_CONFIG.aml.workspace_name,
             credential=credential,
         )
-    return ML_CLIENT
 
+    except Exception as ex:
+        print("Could not find either cli args or config.yaml.")
+        # tries to connect using local config.json
+        ML_CLIENT = MLClient.from_config(credential=credential)
 
-ML_CLIENT = connect_to_aml()
+    return ML_CLIENT
 
 
 #######################################
@@ -376,7 +371,7 @@ def silo_scatter_subgraph(
 
 if args.submit:
     print("Submitting the pipeline job to your AzureML workspace...")
-
+    ML_CLIENT = connect_to_aml()
     pipeline_job = ML_CLIENT.jobs.create_or_update(
         pipeline_job, experiment_name="fl_dev"
     )

examples/pipelines/fl_cross_silo_literal/config.yaml

@@ -12,9 +12,9 @@
 
 # References to Azure ML workspace (use cli args to override)
 aml:
-  subscription_id: "<SUBSCRIPTION_ID>"
-  resource_group_name: "<RESOURCE_GROUP>"
-  workspace_name: "<AML_WORKSPACE_NAME>"
+  # subscription_id: "<SUBSCRIPTION_ID>"
+  # resource_group_name: "<RESOURCE_GROUP>"
+  # workspace_name: "<AML_WORKSPACE_NAME>"
 
 # Parameters to generate the FL graph
 federated_learning:

examples/pipelines/fl_cross_silo_literal/submit.py

@@ -115,14 +115,6 @@ def connect_to_aml():
 
     # Get a handle to workspace
     try:
-        # tries to connect using local config.json
-        ML_CLIENT = MLClient.from_config(credential=credential)
-
-    except Exception as ex:
-        print(
-            "Could not find config.json, using config.yaml refs to Azure ML workspace instead."
-        )
-
         # tries to connect using cli args if provided else using config.yaml
         ML_CLIENT = MLClient(
             subscription_id=args.subscription_id or YAML_CONFIG.aml.subscription_id,
@@ -131,10 +123,14 @@ def connect_to_aml():
             workspace_name=args.workspace_name or YAML_CONFIG.aml.workspace_name,
             credential=credential,
         )
-    return ML_CLIENT
 
+    except Exception as ex:
+        print("Could not find either cli args or config.yaml.")
+        # tries to connect using local config.json
+        ML_CLIENT = MLClient.from_config(credential=credential)
+
+    return ML_CLIENT
 
-ML_CLIENT = connect_to_aml()
 
 ####################################
 ### LOAD THE PIPELINE COMPONENTS ###
@@ -342,7 +338,7 @@ def fl_cross_silo_internal_basic():
 
 if args.submit:
     print("Submitting the pipeline job to your AzureML workspace...")
-
+    ML_CLIENT = connect_to_aml()
     pipeline_job = ML_CLIENT.jobs.create_or_update(
         pipeline_job, experiment_name="fl_dev"
     )

examples/pipelines/ner/config.yaml

@@ -12,9 +12,9 @@
 
 # References to Azure ML workspace (use cli args to override)
 aml:
-  subscription_id: "<SUBSCRIPTION_ID>"
-  resource_group_name: "<RESOURCE_GROUP>"
-  workspace_name: "<AML_WORKSPACE_NAME>"
+  # subscription_id: "<SUBSCRIPTION_ID>"
+  # resource_group_name: "<RESOURCE_GROUP>"
+  # workspace_name: "<AML_WORKSPACE_NAME>"
 
 # Parameters to generate the FL graph
 federated_learning:

examples/pipelines/ner/submit.py

@@ -105,14 +105,6 @@ def connect_to_aml():
 
     # Get a handle to workspace
     try:
-        # tries to connect using local config.json
-        ML_CLIENT = MLClient.from_config(credential=credential)
-
-    except Exception as ex:
-        print(
-            "Could not find config.json, using config.yaml refs to Azure ML workspace instead."
-        )
-
         # tries to connect using cli args if provided else using config.yaml
         ML_CLIENT = MLClient(
             subscription_id=args.subscription_id or YAML_CONFIG.aml.subscription_id,
@@ -121,10 +113,14 @@ def connect_to_aml():
             workspace_name=args.workspace_name or YAML_CONFIG.aml.workspace_name,
             credential=credential,
         )
-    return ML_CLIENT
 
+    except Exception as ex:
+        print("Could not find either cli args or config.yaml.")
+        # tries to connect using local config.json
+        ML_CLIENT = MLClient.from_config(credential=credential)
+
+    return ML_CLIENT
 
-ML_CLIENT = connect_to_aml()
 
 ####################################
 ### LOAD THE PIPELINE COMPONENTS ###
@@ -337,7 +333,7 @@ def fl_ner_basic():
 
 if args.submit:
     print("Submitting the pipeline job to your AzureML workspace...")
-
+    ML_CLIENT = connect_to_aml()
     pipeline_job = ML_CLIENT.jobs.create_or_update(
         pipeline_job, experiment_name="fl_demo_ner"
     )

examples/pipelines/pneumonia/config.yaml

@@ -2,9 +2,9 @@
 
 # using this to store references to Azure ML
 aml:
-  subscription_id: "<SUBSCRIPTION_ID>"
-  resource_group_name: "<RESOURCE_GROUP>"
-  workspace_name: "<AML_WORKSPACE_NAME>"
+  # subscription_id: "<SUBSCRIPTION_ID>"
+  # resource_group_name: "<RESOURCE_GROUP>"
+  # workspace_name: "<AML_WORKSPACE_NAME>"
 
 # federated learning parameters
 federated_learning:

examples/pipelines/pneumonia/submit.py

@@ -105,14 +105,6 @@ def connect_to_aml():
 
     # Get a handle to workspace
     try:
-        # tries to connect using local config.json
-        ML_CLIENT = MLClient.from_config(credential=credential)
-
-    except Exception as ex:
-        print(
-            "Could not find config.json, using config.yaml refs to Azure ML workspace instead."
-        )
-
         # tries to connect using cli args if provided else using config.yaml
         ML_CLIENT = MLClient(
             subscription_id=args.subscription_id or YAML_CONFIG.aml.subscription_id,
@@ -121,10 +113,14 @@ def connect_to_aml():
             workspace_name=args.workspace_name or YAML_CONFIG.aml.workspace_name,
             credential=credential,
         )
-    return ML_CLIENT
 
+    except Exception as ex:
+        print("Could not find either cli args or config.yaml.")
+        # tries to connect using local config.json
+        ML_CLIENT = MLClient.from_config(credential=credential)
+
+    return ML_CLIENT
 
-ML_CLIENT = connect_to_aml()
 
 ####################################
 ### LOAD THE PIPELINE COMPONENTS ###
@@ -274,7 +270,7 @@ def fl_pneumonia_basic():
 
 if args.submit:
     print("Submitting the pipeline job to your AzureML workspace...")
-
+    ML_CLIENT = connect_to_aml()
     pipeline_job = ML_CLIENT.jobs.create_or_update(
         pipeline_job, experiment_name="fl_demo_pneumonia"
     )

examples/pipelines/utils/upload_data/config.yaml

@@ -2,9 +2,9 @@
 
 # References to Azure ML workspace (use cli args to override)
 aml:
-  subscription_id: "<SUBSCRIPTION_ID>"
-  resource_group_name: "<RESOURCE_GROUP>"
-  workspace_name: "<AML_WORKSPACE_NAME>"
+  # subscription_id: "<SUBSCRIPTION_ID>"
+  # resource_group_name: "<RESOURCE_GROUP>"
+  # workspace_name: "<AML_WORKSPACE_NAME>"
 
 # Parameters to generate the FL graph
 federated_learning:

examples/pipelines/utils/upload_data/submit.py

@@ -107,14 +107,6 @@ def connect_to_aml():
 
     # Get a handle to workspace
     try:
-        # tries to connect using local config.json
-        ML_CLIENT = MLClient.from_config(credential=credential)
-
-    except Exception as ex:
-        print(
-            "Could not find config.json, using config.yaml refs to Azure ML workspace instead."
-        )
-
         # tries to connect using cli args if provided else using config.yaml
         ML_CLIENT = MLClient(
             subscription_id=args.subscription_id or YAML_CONFIG.aml.subscription_id,
@@ -123,10 +115,14 @@ def connect_to_aml():
             workspace_name=args.workspace_name or YAML_CONFIG.aml.workspace_name,
             credential=credential,
         )
-    return ML_CLIENT
 
+    except Exception as ex:
+        print("Could not find either cli args or config.yaml.")
+        # tries to connect using local config.json
+        ML_CLIENT = MLClient.from_config(credential=credential)
+
+    return ML_CLIENT
 
-ML_CLIENT = connect_to_aml()
 
 ####################################
 ### LOAD THE PIPELINE COMPONENTS ###
@@ -206,7 +202,7 @@ def fl_cross_silo_upload_data():
 
 if args.submit:
     print("Submitting the pipeline job to your AzureML workspace...")
-
+    ML_CLIENT = connect_to_aml()
     pipeline_job = ML_CLIENT.jobs.create_or_update(
         pipeline_job, experiment_name="fl_demo_upload_data"
     )