This repository contains a set of scripts that will create CDP minimal assets for demo in one wrapper script, including:
- Cloud pre-requisites (bucket, policies, roles, network)
- Cloud CDP Environmment
- CDP Data Lake
- Any CDP Data Hub cluster definition
- AWS CLI (Instructions)
- You must run
aws configure
after install, and ensure your region is set
- You must run
- AWS ssh key (Instructions); Alternatively, you can use the
field
or_field
keys setup in our AWS SE accounts
- Azure CLI (Instructions
- Use
az login
after install to login
- Use
- ssh Key: you will need to paste the public key into your parameters file
Note: Azure does not support CML yet, so don't add it in your parameters file :)
-
CDP CLI (Instructions)
-
CDP Credential (Instructions)
{ Required parameters: "required": { Prefix used for cdp assets creation: "prefix": "pvi", Name of credential to use: "credential": "pvidal-aws-se-credential", Region to use (should also be the default region of your cloud provider cli profile): "region": "us-east-1", ssh key to use for cdp instances setup: "key": "field", Workload password to use in CDP: "workload_pwd": "cdpw0rksh0p", Array of datahub to setup (can be empty): "datahub_list": [ Element 1: { Definition from cdp-cluster-definitions folder: "definition": "data-mart.json", Custom script from cdp-dh-custom-scripts folder: "custom_script": "" }, Element 2: { Definition from cdp-cluster-definitions folder: "definition": "cdp-mod-workshop.json", Custom script from cdp-dh-custom-scripts folder: "custom_script": "cdp_mod_wkp.sh" }, ], Array of ml workspaces to setup (can be empty): "ml_workspace_list": [ Element 1: { Definition from cml-workspace-definitions folder: "definition": "small_workspace.json" } ] }, Optional (defaulted) parameters (can be empty): "optional": { Cloud provider (default: aws, possible values: aws, az): "cloud_provider": "aws", Cloud provider cli profile (only supported in AWS) (default: default): "cloud_profile": "default", CDP cli profile (default: default): "cdp_profile": "default", Flag to create cdp credential or not (default: no, possible values: yes, no) "generate_credential": "no", External ID for cross account role creation, can be found in register environment screen (only needed for AWS) "external_id": "YOUR_EXTERNAL_ID", NOT SUPPORTED YET Flag to generate minimal cross account role policy or not (default: no, possible values: yes, no) "generate_minimal_cross_account": "no", Flag to create network in cloud provider or not (only supported in AWS) (default: no, possible values: yes, no) "create_network": "no", Tags to setup: "tags": { End date tag (default: today's date +3 days): "end_date": "04102020", End date tag (default: [prefix]_one_click_project): "project": "pvi_test" }, } }
See parameters_sample
folder
Run the source target wrapper script:
cdp_create_all_the_things.sh <your_param_file>
Run the deletion script:
cdp_delete_all_the_things.sh <your_param_file>
cdp_aws_pre_reqs.sh.sh <your_param_file>
cdp_aws_sdx.sh <your_param_file> [<network_file>]
cdp_az_pre_reqs.sh.sh <your_param_file>
cdp_az_sdx.sh <your_param_file>
cdp_create_datahub_things.sh <your_param_file>
cdp_create_ml_things.sh <your_param_file>
cdp_stop_all_the_things.sh <your_param_file>
cdp_start_all_the_things.sh <your_param_file>
- Add support for Azure ML
- Add support for minimal set of policies for AWS
- Add dynamic definition updates
- Create a nifi flow wrapper?
Paul Vidal - LinkedIn
Dan Chaffelson - LinkedIn
Chris Perro - LinkedIn
André Araújo - LinkedIn
Nathan Anthony - LinkedIn
Steffen Maerkl - LinkedIn
Mike Riggs - LinkedIn