Use NetApp Cloud API. This python scripts show how to work with NetApp Cloud API.
Thes scripts need to be used with python 3
The following python modules must be installed (Linux/Windows):
python3 -m pip install pyopenssl
python3 -m pip install urllib3
python3 -m pip install requests
python3 -m pip install requests-toolbelt
# cd <Installation Directory>
# git clone https://github.com/jbnetapp/demo-netapp-cloud-api
echo "alias cloudaccount='python3 $PWD/demo-netapp-cloud-api/cloudaccount.py'" >> $HOME/.bashrc
echo "alias cloudsync='python3 $PWD/demo-netapp-cloud-api/cloudsync.py'" >> $HOME/.bashrc
echo "alias occm='python3 $PWD/demo-netapp-cloud-api/occm.py'" >> $HOME/.bashrc
source $HOME/.bashrc
NetApp Cloud Central user can be Federated or Non-Federated:
- For Non-Federated users (Regular Access) you need to enter username (email) and password:
- Create the configuration file for Regular Access using cloudaccount.py script:
# cloudaccount --setup Is your NetApp Cloud Central use federated users ? [y/n] : n NetApp Cloud Central Email : <YOUR EMAIL NETAPP CLOUD CENTRAL> NetApp Cloud Central Password : <YOUR PASSWORD>
- Create the configuration file for Regular Access using cloudaccount.py script:
- For Federated user (ADFS, Microsfot AD, or SAML ) associate with your corporate email
- If your user is federated user, you must use a refresh token Access to Acquire a JWT access token from the OAuth token endpoint.
- To get your refresh token Access login to the https://services.cloud.netapp.com/refresh-token
- Click on Generate Refresh Token or click on Revoke Token(s)
Copy your token and save it in to the api.conf in section header [API_LOGIN] in variable refresh_token, as shown in the following example.
- Click on Generate Refresh Token or click on Revoke Token(s)
- Create the Configuration File for federated users using cloudaccount.py script:
# cloudaccount --setup Is your NetApp Cloud Central use federated users ? [y/n] : y To get your Refresh Token please go to : https://services.cloud.netapp.com/refresh-token NetApp Cloud Central Refresh Token : <YOUR_REFRESH_TOKEN>
Configuration File:
- For Linux configuration file is: $HOME/NetAppCloud/api.conf
- For Windows configuration file is: %homedrive%%homepath%\NetAppCloud\api.conf
When using cloudaccount.py script with option --setup to create the configuration file the JWT access token is automatically saved in the section header [API_TOKEN]. To Check if your JWT access token is valid you can use --token-check option.
# cloudaccount --token-check
Access Token is valid
The JWT access token is limited in time and will expire after few days. If JWT access token as expired a Code error 401 with message TokenExpiredError: jwt expired is return by the API:
# cloudaccount --token-check
ERROR: {"code":401,"message":"TokenExpiredError: jwt expired"}
To Get a new JWT access token :
# cloudaccount --token-get-new
Check if your new JWT access token is valid and saved in your private api.conf configuration file.
# cloudsync --token-check
Access Token is valid
The script occm.py show how to work with NetApp Cloud Manager API. The Cloud Manager API documentation is available here : https://cloudmanager.netapp.com/api-docs/
# occm --cvo-list
Name:[cvoaz01][Azure] id:[VsaWorkingEnvironment-tNz1RNLH] HA:[False]
Name:[cvoaz02][Azure] id:[VsaWorkingEnvironment-7HRP4PQL] HA:[True]
Name:[cvoaw03][Amazon] id:[VsaWorkingEnvironment-yBHqMbb3] HA:[False]
# occm --cvo-list --status
Name:[cvoaz01][Azure] id:[VsaWorkingEnvironment-tNz1RNLH] HA:[False] status[ON] mgmt:[172.16.1.23]
Name:[cvoaz02][Azure] id:[VsaWorkingEnvironment-7HRP4PQL] HA:[True] status[ON] mgmt:[172.16.1.5]
Name:[cvoaw03][Amazon] id:[VsaWorkingEnvironment-yBHqMbb3] HA:[False] status[ON] mgmt:[172.20.1.20]
# occm --cvo-list --status
Name:[cvoaz01][Azure] node1:[cvoaz01-01][90920130000001027571][2886730007]
Name:[cvoaz01][Azure] node1_lif:[Data][172.16.1.27][['iscsi']]
Name:[cvoaz01][Azure] node1_lif:[Node Management][172.16.1.23]
Name:[cvoaz01][Azure] node1_lif:[Cluster Management][172.16.1.24]
Name:[cvoaz01][Azure] node1_lif:[Intercluster][172.16.1.25]
Name:[cvoaz01][Azure] node1_lif:[Data][172.16.1.26][['nfs', 'cifs']]
Name:[cvoaz01][Azure] ontap_version:[9.12.1RC1P1]
Name:[cvoaz01][Azure] license_name:[Freemium]
Name:[cvoaz01][Azure] mgmt_ip:[172.16.1.24]
Name:[cvoaz01][Azure] svm_name:[svm_cvoaz01]
Name:[cvoaz01][Azure] status:[ON]
Name:[cvoaz02][Azure] node1:[cvoaz02-01][90920140000001043426][2886729997]
Name:[cvoaz02][Azure] node1_lif:[Intercluster][172.16.1.14]
Name:[cvoaz02][Azure] node1_lif:[Cluster Management][172.16.1.5]
Name:[cvoaz02][Azure] node1_lif:[Data][172.16.1.15][['iscsi']]
Name:[cvoaz02][Azure] node1_lif:[Data][172.16.1.6][['nfs', 'cifs']]
Name:[cvoaz02][Azure] node1_lif:[SVM Management][172.16.1.8]
Name:[cvoaz02][Azure] node1_lif:[Node Management][172.16.1.13]
Name:[cvoaz02][Azure] node1_lif:[Cluster][172.16.1.10]
Name:[cvoaz02][Azure] node2:[cvoaz02-02][90920140000001043427][2886730001]
Name:[cvoaz02][Azure] node2_lif:[Cluster][172.16.1.12]
Name:[cvoaz02][Azure] node2_lif:[Node Management][172.16.1.17]
Name:[cvoaz02][Azure] node2_lif:[Intercluster][172.16.1.18]
Name:[cvoaz02][Azure] node2_lif:[Data][172.16.1.19][['iscsi']]
Name:[cvoaz02][Azure] node2_lif:[Data][172.16.1.7][['nfs', 'cifs']]
Name:[cvoaz02][Azure] ontap_version:[9.12.1RC1]
Name:[cvoaz02][Azure] license_name:[Freemium]
Name:[cvoaz02][Azure] mgmt_ip:[172.16.1.5]
Name:[cvoaz02][Azure] svm_name:[svm_cvoaz02]
Name:[cvoaz02][Azure] status:[ON]
Name:[cvoaz02][Azure] HA:[True]
Name:[cvoaz02][Azure] loadBalancer:[RT1927389_CVOHA-lb]
Name:[cvoaz02][Azure] resourceGroup:[RT1927389_CVOHA]
Name:[cvoaz02][Azure] multiZone:[False]
Name:[cvoaw03][Amazon] node1:[cvoaw03-01][90920130000001027620][2886992148]
Name:[cvoaw03][Amazon] node1_lif:[Data][172.20.1.111][['nfs', 'cifs']]
Name:[cvoaw03][Amazon] node1_lif:[Data][172.20.1.193][['iscsi']]
Name:[cvoaw03][Amazon] node1_lif:[Node Management][172.20.1.20]
Name:[cvoaw03][Amazon] node1_lif:[SVM Management][172.20.1.168]
Name:[cvoaw03][Amazon] node1_lif:[Cluster Management][172.20.1.32]
Name:[cvoaw03][Amazon] node1_lif:[Intercluster][172.20.1.200]
Name:[cvoaw03][Amazon] ontap_version:[9.12.1RC1]
Name:[cvoaw03][Amazon] license_name:[Freemium]
Name:[cvoaw03][Amazon] mgmt_ip:[172.20.1.32]
Name:[cvoaw03][Amazon] svm_name:[svm_cvoaw03]
Name:[cvoaw03][Amazon] status:[ON]
# occm --cvo-get VsaWorkingEnvironment-tNz1RNLH
Name:[cvoaz01][Azure] HA:[False] svm:[svm_cvoaz01] status[ON] mgmt[172.30.24.207]
# occm --cvo-get cvoaz01 --json > /tmp/cvoaz01.json
# occm --cvo-get-creation-parameters cvoaz01 > /tmp/cvoaz01-config-parameters.json
Recreate a new Azure Cloud Volume ONTAP Using JSON file create with otpion --cvo-get-creation-parameters
# occm --cvo-az-create /tmp/cvoaz01-config-parameters.json
Creates a new Azure Cloud Volumes ONTAP working environment
Name:[cvoaz01b] id:[VsaWorkingEnvironment-4DEHFQMN] HA:[False] svm:[svm_cvoaz01b] provider[Azure]
# occm --cvo-az-create ./new-cvo-azure-single-cvoaz01-ws1.json
Creates a new Azure Cloud Volumes ONTAP working environment
Name:[cvoaz01] id:[VsaWorkingEnvironment-4DEHFQMN] HA:[False] svm:[svm_cvoaz01b] provider[Azure]
# occm --cvo-az-create-ha ./new-cvo-azure-ha-cvoaz02-ws1.json
Creates a new Azure HA Cloud Volumes ONTAP working environment
Name:[cvoaz02] id:[VsaWorkingEnvironment-7HRP4PQL] HA:[True] svm:[svm_cvoaz02] provider[Azure]
# occm --cvo-aw-create ./new-cvo-aws-single-cvoaw03-ws1.json
Creates a new AWS Cloud Volumes ONTAP working environment
Name:[cvoaw03] id:[VsaWorkingEnvironment-DPKWwhPC] HA:[False] svm:[svm_cvoaw03] provider[Amazon]
# occm --cvo-aw-create-ha ./new-cvo-aws-ha-cvoaw02-single-az-ws1.json
Creates a new AWS HA Cloud Volumes ONTAP working environment
Name:[cvoaw02] id:[VsaWorkingEnvironment-1uqJIydY] HA:[True] svm:[svm_cvoaw02] provider[Amazon]
# occm --cvo-stop cvoaz01b
Stop Cloud volumes ONTAP working environment ID: VsaWorkingEnvironment-4DEHFQMN
Name:[cvoaz01b] id:[VsaWorkingEnvironment-4DEHFQMN] HA:[False] status:[ON] provider[Azure]
WARNING: do you want to stop CVO [cvoaz01b] ? [y/n] : y
CVO Name:[cvoaz01b] stopped
# occm --cvo-start cvoaz01b
Start Cloud volumes ONTAP working environment ID: VsaWorkingEnvironment-4DEHFQMN
Name:[cvoaz01b] id:[VsaWorkingEnvironment-4DEHFQMN] HA:[False] status:[ON] provider[Azure]
WARNING: do you want to stop CVO [cvoaz01b] ? [y/n] : y
CVO Name:[cvoaz01b] started
# occm --cvo-delete cvoaz01b
Delete cloud volumes ONTAP working environment ID: VsaWorkingEnvironment-4DEHFQMN
Name:[cvoaz01b] id:[VsaWorkingEnvironment-4DEHFQMN] HA:[False] svm:[svm_cvoaz01b] provider[Azure]
WARNING: do you want to delete the CVO [cvoaz01b] ? [y/n] :
WARNING: do you want to delete the CVO [cvoaz01b] ? [y/n] : y
CVO Name:[cvoaz01b] deleted
# occm --help
options:
-h, --help show this help message and exit
-d, --debug debug mode
-v, --verbose verbose mode
-s, --status display cvo status
--account-id ACCOUNT_ID
select account ID
--agent-id AGENT_ID select cloud manager Connector
-j, --json print in json format
--account-list print accounts list
--cloud-account-list print cloud accounts list
--agent-list print cloud manager connectors list
--agent-switch SWITCH_AGENT_ID
switch to a connector using it agent_id
--we-list list all working environments
--cvo-list list all Cloud Volumes ONTAP working environments
--cvo-get GET_CVO_ID get an existing Cloud Volumes ONTAP working environment details
--cvo-get-creation-parameters GET_CVO_ID_CREATION_PARAMETERS
get an existing Cloud Volumes ONTAP creation parameters
--cvo-start START_CVO_ID
start an existing Cloud Volumes ONTAP working environment
--cvo-stop STOP_CVO_ID
stop an existing Cloud Volumes ONTAP working environment
--cvo-delete DELETE_CVO_ID
delete an existing Cloud Volumes ONTAP working environment
--cvo-az-create CREATE_CVO_AZ_FILE
create a new Cloud Volumes ONTAP in Azure
--cvo-az-create-ha CREATE_CVO_AZ_HA_FILE
create a new Cloud Volumes ONTAP HA in Azure
--cvo-aws-create CREATE_CVO_AWS_FILE
create a new Cloud Volumes ONTAP in AWS
--cvo-aws-create-ha CREATE_CVO_AWS_HA_FILE
create a new Cloud Volumes ONTAP HA in AWS
--cvo-gcp-create CREATE_CVO_GCP_FILE
create a new Cloud Volumes ONTAP in GCP
--cvo-gcp-create-ha CREATE_CVO_GCP_HA_FILE
create a new Cloud Volumes ONTAP HA in GCP
--token-check check NetApp Cloud access token
--token-get-new get a new access token
--version print release
The script cloudsync.py show how to work with NetApp Cloud API Sync API. The Cloud Sync API documentation is available here : https://api.cloudsync.netapp.com/docs/
# cloudsync --account-list
Print NetApp Account list:
Blanchet account_id: [account-yX7cS8vU] Current:[X]
Demo_SIM account_id: [account-j3aZttuL]
NetAppHCL account_id: [account-U0dbRcKS]
# cloudaccount --account-switch account-U0dbRcKS
# cloudaccount --account-list
Print NetApp Account list:
Blanchet account_id: [account-yX7cS8vU]
Demo_SIM account_id: [account-j3aZttuL]
NetAppHCL account_id: [account-U0dbRcKS] Current:[X]
# cloudsync --data-broker-list
Print NetApp data-borkers list:
Name:[vmazjb1] ID:[616d9c8d48301b1e6cdfe1df] PrivateIP:[172.30.31.5] TransferRate:[1753.016465568962] Status:[COMPLETE]
Example using the local JSON file example file from this git repository to create a Cloud Sync relation between two Azure blobs.
# cloudsync --create-relation ./Example/new-cloudsync-relation-blob-to-blob-example.json
New cloud Sync relationship successfully created
- Remarque to use the JSON example file you must:
- Change the dataBrokerId with your one of your dataBorkerId. To get your dataBrokerID you can:
- Use th script cloudsny.py with the option --data-borker-list (see example above)
- Go to cloudmanager -> sync -> Manager Data Broker -> Select your Data Broker and click on button (>)
- The source jbblob and the target jblobcopy Blobs must exist in your Azure storage account
- The storage account name jbblobazure must also exist.
- Change the dataBrokerId with your one of your dataBorkerId. To get your dataBrokerID you can:
- For more information about the JSON syntax used: https://api.cloudsync.netapp.com/docs/
# cloudsync --print
Print cloudsync relations:
id: 61c2e054b10e1f362ede48e1
account: 5e528f4504a9a4d63d6962de
dataBroker: 616d9c8d48301b1e6cdfe1df
source: {'protocol': 'azure', 'azure': {'storageAccountName': 'jbdemostorageaccount', 'container': 'jbblob', 'prefix': 'DIR', 'tags': [], 'provider': 'azure'}}
target: {'protocol': 'azure', 'azure': {'storageAccountName': 'jbdemostorageaccount', 'container': 'jbblobcopy', 'prefix': '', 'tags': [], 'blobTier': 'HOT', 'provider': 'azure'}}
type: Sync
status: DONE
# cloudsync --sync 61c2e054b10e1f362ede48e1
Sync cloudsync relation ID: 61c2e054b10e1f362ede48e1
Verify if the relation is in RUNNING state Print CloudSync Relation again:
# cloudsync --print
Print cloud sync relations:
id: 61c2e054b10e1f362ede48e1
account: 5e528f4504a9a4d63d6962de
dataBroker: 616d9c8d48301b1e6cdfe1df
source: {'protocol': 'azure', 'azure': {'storageAccountName': 'jbdemostorageaccount', 'container': 'jbblob', 'prefix': 'DIR1', 'tags': [], 'provider': 'azure'}}
target: {'protocol': 'azure', 'azure': {'storageAccountName': 'jbdemostorageaccount', 'container': 'jbblobcopy', 'prefix': '', 'tags': [], 'blobTier': 'HOT', 'provider': 'azure'}}
type: Sync
status: RUNNING
if you Sync agin the Cloud Sync relation during an existing sync action the script will display error message :
# cloudsync --sync 61c2e054b10e1f362ede48e1
Sync cloudsync relation ID: 61c2e054b10e1f362ede48e1
ERROR: {"code":400,"message":"A sync action is running right now, can not run another action"}
# cloudsync --delete 61c2e054b10e1f362ede48e1 -d
Delete cloudsync relation ID: 61c2e054b10e1f362ede48e1
# cloudsync --print --json
[
{
"account": 5e528f4504a9a4d63d6962de
"dataBroker": "616d9c8d48301b1e6cdfe1df",
"source": {
"protocol": "azure",
"azure": {
"storageAccountName": "jbblobazure",
"container": "jbblob",
"prefix": "DIR1",
"tags": [],
"provider": "azure"
}
},
"target": {
"protocol": "azure",
"azure": {
"storageAccountName": "jbblobazure",
"container": "jbblobcopy",
"prefix": "DST1",
"tags": [],
"blobTier": "COOL",
"provider": "azure"
}
},
"settings": {
"gracePeriod": 30,
"deleteOnSource": false,
"deleteOnTarget": false,
"objectTagging": false,
"retries": 3,
"copyAcl": false,
"files": {
"excludeExtensions": [],
"maxSize": 9007199254740991,
"minSize": 0,
"minDate": "1970-01-01",
"maxDate": null,
"minCreationDate": "1970-01-01",
"maxCreationDate": null
},
"fileTypes": {
"files": true,
"directories": true,
"symlinks": true
},
"compareBy": {
"uid": false,
"gid": false,
"mode": false,
"mtime": true
},
"schedule": {
"syncInDays": 0,
"syncInHours": 1,
"syncInMinutes": 0,
"nextTime": "2021-12-24T09:00:00.000Z",
"isEnabled": true,
"syncWhenCreated": true
},
"copyProperties": {
"metadata": false,
"tags": false
}
},
"isQstack": false,
"isCm": true,
"phase": "Initial Copy",
"group": "616d9c8dcec9f8eef6a411fa",
"startTime": "2021-12-23T09:20:12.875Z",
"createdAt": 1640251212876,
"endTime": "2021-12-23T09:22:59.304Z",
"id": "61c43f4c86f2ee48a82a0813",
"relationshipId": "61c43f4c86f2ee48a82a0813",
"activity": {
"type": "Initial Copy",
"status": "DONE",
"errors": [],
"failureMessage": "",
"executionTime": 166429,
"startTime": "2021-12-23T09:20:12.875Z",
"endTime": "2021-12-23T09:22:59.304Z",
"bytesMarkedForCopy": 1243560,
"filesMarkedForCopy": 1,
"dirsMarkedForCopy": 0,
"filesCopied": 0,
"bytesCopied": 0,
"dirsCopied": 0,
"filesFailed": 1,
"bytesFailed": 1243560,
"dirsFailed": 0,
"filesMarkedForRemove": 0,
"bytesMarkedForRemove": 0,
"dirsMarkedForRemove": 0,
"filesRemoved": 0,
"bytesRemoved": 0,
"dirsRemoved": 0,
"bytesRemovedFailed": 0,
"filesRemovedFailed": 0,
"filesMarkedForIgnore": 0,
"dirsScanned": 0,
"filesScanned": 1,
"dirsFailedToScan": 0,
"bytesScanned": 1243560,
"progress": 100,
"lastMessageTime": "2021-12-23T09:22:59.307Z",
"topFiveMostCommonRelationshipErrors": [
{
"step": "TRANSFER",
"errorCode": "invalid_credentials",
"counter": 1,
"description": "source connection-string should be SAS Connection-string and not regular connection-string"
}
]
}
}
]
# cloudsync --account-id account-U0dbRcKS --account-list --debug
DEBUG: [DEFAULT: <Section: DEFAULT> ]
DEBUG: [API_LOGIN: <Section: API_LOGIN> ]
DEBUG: [API_TOKEN: <Section: API_TOKEN> ]
DEBUG: [API Configuration File: /home/blanchet/NetAppCloud/api.conf]
DEBUG: [access_token: ******************************************************************************************]
DEBUG: [url: https://api.cloudsync.netapp.com/api/accounts ]
DEBUG: [status_code: 200]
DEBUG: [text: [{"accountId":"account-yX7cS8vU","name":"Blanchet"},{"accountId":"account-j3aZttuL","name":"Demo_SIM"},{"accountId":"account-U0dbRcKS","name":"NetAppHCL"}]]
DEBUG: [content: b'[{"accountId":"account-yX7cS8vU","name":"Blanchet"},{"accountId":"account-j3aZttuL","name":"Demo_SIM"},{"accountId":"account-U0dbRcKS","name":"NetAppHCL"}]']
DEBUG: [reason: OK]
DEBUG: [access_token : *****************************************************************************************]
Print NetApp Account list:
DEBUG: [url: https://api.cloudsync.netapp.com/api/accounts ]
DEBUG: [status_code: 200]
DEBUG: [text: [{"accountId":"account-yX7cS8vU","name":"Blanchet"},{"accountId":"account-j3aZttuL","name":"Demo_SIM"},{"accountId":"account-U0dbRcKS","name":"NetAppHCL"}]]
DEBUG: [content: b'[{"accountId":"account-yX7cS8vU","name":"Blanchet"},{"accountId":"account-j3aZttuL","name":"Demo_SIM"},{"accountId":"account-U0dbRcKS","name":"NetAppHCL"}]']
DEBUG: [reason: OK]
DEBUG: [{'status': 'success', 'message': 'ok', 'accounts': '[{"accountId":"account-yX7cS8vU","name":"Blanchet"},{"accountId":"account-j3aZttuL","name":"Demo_SIM"},{"accountId":"account-U0dbRcKS","name":"NetAppHCL"}]', 'token': '****************']
Blanchet account_id: [account-yX7cS8vU]
Demo_SIM account_id: [account-j3aZttuL]
NetAppHCL account_id: [account-U0dbRcKS]
# cloudsync --help
usage: cloudsync.py [-h] [-d] [--account-id ACCOUNT_ID] [-j]
(--account-list | --data-broker-list | --create-relation CREATE_RELATION_FILE | --delete-relation DELETE_RELATION_ID | --sync-relation SYNC_RELATION_ID | --print-relations | --token-check | --token-get-new)
optional arguments:
-h, --help show this help message and exit
-d, --debug debug mode
--account-id ACCOUNT_ID
select NetApp Cloud account ID
-j, --json print in json format
--account-list print cloudsnyc accounts
--data-broker-list print cloudsync data-brokers
--create-relation CREATE_RELATION_FILE
create new cloudsync relation from json file
--delete-relation DELETE_RELATION_ID
delete a cloudsync relation
--sync-relation SYNC_RELATION_ID
sync a cloudsync relation
--print-relations print cloudsnyc relations list
--token-check check NetApp Cloud access token
--token-get-new get a new access token