export policy on qtree broken with ontap 9.8

[sweeneyben]

We have code that adds IP's to existing NFS export policies.

This has been running on ontap 9.7 with the 9.7 python library

We upgraded ontap to 9.8, and the API broke. We tried running it with the 9.7, 9.8.1 and 9.9 python libraries. Please take a look into why this API broke with the upgrade to OnTap 9.8

Code:
def getQtreeExportID(qtreeName):
print("Getting QTree Export Policy ID")
print("QTree Name: ", qtreeName)
qTreeList = list(Qtree.get_collection(fields="export_policy"))
for qTree in qTreeList:
if(qTree.name == qtreeName):
return qTree.export_policy.id

Output:
Getting QTree Export Policy ID
QTree Name: dev-qtree
Traceback (most recent call last):
File "./mount_nfs_v2.py", line 136, in
main(sys.argv[1:])
File "./mount_nfs_v2.py", line 129, in main
exportPolicyQtreeID = getQtreeExportID(qtreeName)
File "./mount_nfs_v2.py", line 32, in getQtreeExportID
return qTree.export_policy.id
File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 195, in getattribute
) from None
AttributeError: The 'id' field has not been set on the ExportPolicy. Try refreshing the object by calling get().

[github-actions]

Thank you for reporting an issue! If you haven't already joined our Slack community,
then we invite you to do so. This is a great place to get help and ask questions from our community.

[RobertBlackhart]

I did some testing and was able to verify the same. I checked both 9.8 and 9.9.1 and the issue was as you described. I filed a bug
report for the team that owns the /api/storage/qtrees endpoint to look into the issue. As the team makes any updates, you'll be able
to track it here: https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1424170 (note: it may take up to a day
for that link to become active)

In the meantime, as a workaround, you can change your code like this:

def getQtreeExportID(qtreeName):
    print("Getting QTree Export Policy ID")
    print("QTree Name: ", qtreeName)
    qTreeList = list(Qtree.get_collection(fields="export_policy"))
    for qTree in qTreeList:
        if(qTree.name == qtreeName):
            return ExportPolicy.find(name=qTree.export_policy.name).id

This change requires an extra API call to the /api/protocols/nfs/export-policies endpoint, but should serve as a temporary workaround.

[sweeneyben]

Robert, any ETA on when a patch with this is planned? We have this on multiple places, and have to make the decision to manually update all our code, or wait for the patch. This would be dependent on when you think it would be fixed by.

[RobertBlackhart]

No, I don't have that information. There's not yet an ETA or decided patch release. I will ask the developers to update this thread when they make that decision.

[asish-prabhakar]

It might take a couple of months for the fix to be available in a patch, so please use the above mentioned workaround.

[sudodevnull]

I tried using the work around:
def getQtreeExportID(qtreeName): print("Getting QTree Export Policy ID") print("QTree Name: ", qtreeName) qTreeList = list(Qtree.get_collection(fields="export_policy")) for qTree in qTreeList: if(qTree.name == qtreeName): return ExportPolicy.find(name=qTree.export_policy.name).id

I'm using netapp-ontap==9.9.1
and I get this error:

Traceback (most recent call last): File "./mount_nfs_v2_workaround.py", line 136, in <module> main(sys.argv[1:]) File "./mount_nfs_v2_workaround.py", line 131, in main setExportPolicy(exportPolicyRootID, ipAddresses) File "./mount_nfs_v2_workaround.py", line 56, in setExportPolicy exportRule = list(ExportRule.get_collection(exportPolicyID)) File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 623, in _get_collection url = sample.get_collection_url(connection=connection) File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 582, in get_collection_url return "%s%s" % (self.get_connection().origin, self._location) File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 270, in __getattribute__ value = super().__getattribute__(name) File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 402, in _location raise NetAppRestError(message=msg, cause=exc) from None netapp_ontap.error.NetAppRestError: Could not compute the location of the ExportRule collection. Values for ['policy.id'] are required. Caused by TypeError('quote_from_bytes() expected bytes',)

[RobertBlackhart]

You appear to be running into bug 1378458: http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1378458

To work around that, on line 56 of your mount_nfs_v2_workaround.py script, you can change list(ExportRule.get_collection(exportPolicyId)) to list(ExportRule.get_collection(str(exportPolicyId)))

[sudodevnull]

I tried that... no luck:

[root@i-0d03741cd10a2d9ce ~]# cat /root/mount.log
Script Starting
Getting Basic Auth Username and Password
Getting Volume Export Policy ID
Getting Volume Export Policy ID
Getting QTree Export Policy ID
QTree Name: dev-jenkins-oc
Getting ASG IPs
IP(s) that are being added
10.217.62.149,10.217.56.216
Updating Export Policy Client IPs
8589934594
Traceback (most recent call last):
File "./mount_nfs_v2_workaround.py", line 136, in
main(sys.argv[1:])
File "./mount_nfs_v2_workaround.py", line 131, in main
setExportPolicy(exportPolicyRootID, ipAddresses)
File "./mount_nfs_v2_workaround.py", line 56, in setExportPolicy
exportRule = list(ExportRule.get_collection(str(exportPolicyId)))
NameError: name 'exportPolicyId' is not defined
[root@i-0d03741cd10a2d9ce ~]#

Here is my script:

import json
import os
import boto3
from botocore.exceptions import ClientError
from netapp_ontap import config, HostConnection
from netapp_ontap.resources import Volume, ExportPolicy, ExportClient, ExportRule, Qtree
import sys, getopt

def establishConnection(region):
  print("Getting Basic Auth Username and Password")
  ssm = boto3.client('ssm', region_name=region)
  parameter = ssm.get_parameter(Name="/netapp/username")
  user=parameter['Parameter']['Value']
  parameter = ssm.get_parameter(Name="/netapp/password", WithDecryption=True)
  password=parameter['Parameter']['Value']
  return user, password

def getVolumeExportID(volName):
  print("Getting Volume Export Policy ID")
  volumesList = list(Volume.get_collection(fields="nas"))
  for volume in volumesList:
    if(volume.name == volName):
      return volume.nas.export_policy.id

def getQtreeExportID(qtreeName):
    print("Getting QTree Export Policy ID")
    print("QTree Name: ", qtreeName)
    qTreeList = list(Qtree.get_collection(fields="export_policy"))
    for qTree in qTreeList:
        if(qTree.name == qtreeName):
            return ExportPolicy.find(name=qTree.export_policy.name).id

def getAsgIps(region, asgName):
  print("Getting ASG IPs")
  asg_client = boto3.client('autoscaling', region_name=region)
  asg=asg_client.describe_auto_scaling_groups(AutoScalingGroupNames=[asgName])
  instanceIds = []
  for a in asg['AutoScalingGroups'][0]['Instances']:
    instanceIds.append(a['InstanceId'])

  ipAddresses = []
  ec2_client = boto3.client('ec2', region_name=region)
  instances=ec2_client.describe_instances(InstanceIds=instanceIds)
  for i in instances['Reservations']:
    ipAddresses.append(i['Instances'][0]['PrivateIpAddress'])
  ipAddressesAsString = ','.join(ipAddresses)
  print("IP(s) that are being added")
  print(ipAddressesAsString)
  return ipAddressesAsString


def setExportPolicy(exportPolicyID, ipAddresses):
  print("Updating Export Policy Client IPs")
  print(exportPolicyID)
  exportRule = list(ExportRule.get_collection(str(exportPolicyId)))
  exportRuleIndex = exportRule[0].index
  resource = ExportClient(exportPolicyID, exportRuleIndex)
  resource.match = ipAddresses
  resource.post()


def main(argv):
  try:
    opts, args = getopt.getopt(argv,"hb:r:v:a:n:q:",["baseurl=","region=","rootvolname=","volname=","asg=","qtreename=","help"])
  except getopt.GetoptError as error:
    print(error + '  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit(2)
  for opt, arg in opts:
    if opt in ("-h", "--help"):
      print('test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
      sys.exit()
    elif opt in ("-b", "--baseurl"):
      baseUrl = arg
    elif opt in ("-r", "--region"):
      region = arg
    elif opt in ("-n", "--rootvolname"):
      rootVolName = arg
    elif opt in ("-v", "--volname"):
      volName = arg
    elif opt in ("-a", "--asg"):
      asgName = arg
    elif opt in ("-q", "--qtreename"):
      qtreeName = arg

  try:
    region
  except NameError:
    print('Missing -r, --region.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    baseUrl
  except NameError:
    print('Missing -b --baseurl.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    rootVolName
  except NameError:
    print('Missing -n --rootvolname.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    volName
  except NameError:
    print('Missing -v --volname.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    asgName
  except NameError:
    print('Missing -a --asg.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    qtreeName
  except NameError:
    print('Missing -q --qtreename.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  print("Script Starting")

  user, password = establishConnection(region)
  os.environ["no_proxy"] = baseUrl
  with HostConnection(baseUrl, username=user, password=password, verify=False):
    exportPolicyRootID = getVolumeExportID(rootVolName)
    exportPolicyVolumeID = getVolumeExportID(volName)
    exportPolicyQtreeID = getQtreeExportID(qtreeName)
    ipAddresses = getAsgIps(region, asgName)
    setExportPolicy(exportPolicyRootID, ipAddresses)
    setExportPolicy(exportPolicyVolumeID, ipAddresses)
    setExportPolicy(exportPolicyQtreeID, ipAddresses)

if __name__ == "__main__":
  main(sys.argv[1:])

[RobertBlackhart]

Sorry, I think my answer just had a typo. Your script has exportPolicyID in it, not exportPolicyId (the D should be capitalized).

[sudodevnull]

Hey no worries... here's the next error:

Traceback (most recent call last):
File "./mount_nfs_v2_workaround.py", line 136, in
main(sys.argv[1:])
File "./mount_nfs_v2_workaround.py", line 131, in main
setExportPolicy(exportPolicyRootID, ipAddresses)
File "./mount_nfs_v2_workaround.py", line 60, in setExportPolicy
resource.post()
File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resources/export_client.py", line 222, in post
poll_timeout=poll_timeout, **kwargs
File "/usr/local/lib/python3.6/site-packages/netapp_ontap/utils.py", line 51, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 987, in _post
url = "%s%s" % (self.get_connection().origin, self._location)
File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 270, in getattribute
value = super().getattribute(name)
File "/usr/local/lib/python3.6/site-packages/netapp_ontap/resource.py", line 402, in _location
raise NetAppRestError(message=msg, cause=exc) from None
netapp_ontap.error.NetAppRestError: Could not compute the location of the ExportClient collection. Values for ['policy.id', 'export_rule.index'] are required. Caused by TypeError('quote_from_bytes() expected bytes',)

And code:

import json
import os
import boto3
from botocore.exceptions import ClientError
from netapp_ontap import config, HostConnection
from netapp_ontap.resources import Volume, ExportPolicy, ExportClient, ExportRule, Qtree
import sys, getopt

def establishConnection(region):
  print("Getting Basic Auth Username and Password")
  ssm = boto3.client('ssm', region_name=region)
  parameter = ssm.get_parameter(Name="/netapp/username")
  user=parameter['Parameter']['Value']
  parameter = ssm.get_parameter(Name="/netapp/password", WithDecryption=True)
  password=parameter['Parameter']['Value']
  return user, password

def getVolumeExportID(volName):
  print("Getting Volume Export Policy ID")
  volumesList = list(Volume.get_collection(fields="nas"))
  for volume in volumesList:
    if(volume.name == volName):
      return volume.nas.export_policy.id

def getQtreeExportID(qtreeName):
    print("Getting QTree Export Policy ID")
    print("QTree Name: ", qtreeName)
    qTreeList = list(Qtree.get_collection(fields="export_policy"))
    for qTree in qTreeList:
        if(qTree.name == qtreeName):
            return ExportPolicy.find(name=qTree.export_policy.name).id

def getAsgIps(region, asgName):
  print("Getting ASG IPs")
  asg_client = boto3.client('autoscaling', region_name=region)
  asg=asg_client.describe_auto_scaling_groups(AutoScalingGroupNames=[asgName])
  instanceIds = []
  for a in asg['AutoScalingGroups'][0]['Instances']:
    instanceIds.append(a['InstanceId'])

  ipAddresses = []
  ec2_client = boto3.client('ec2', region_name=region)
  instances=ec2_client.describe_instances(InstanceIds=instanceIds)
  for i in instances['Reservations']:
    ipAddresses.append(i['Instances'][0]['PrivateIpAddress'])
  ipAddressesAsString = ','.join(ipAddresses)
  print("IP(s) that are being added")
  print(ipAddressesAsString)
  return ipAddressesAsString


def setExportPolicy(exportPolicyID, ipAddresses):
  print("Updating Export Policy Client IPs")
  print(exportPolicyID)
  exportRule = list(ExportRule.get_collection(str(exportPolicyID)))
  exportRuleIndex = exportRule[0].index
  resource = ExportClient(exportPolicyID, exportRuleIndex)
  resource.match = ipAddresses
  resource.post()


def main(argv):
  try:
    opts, args = getopt.getopt(argv,"hb:r:v:a:n:q:",["baseurl=","region=","rootvolname=","volname=","asg=","qtreename=","help"])
  except getopt.GetoptError as error:
    print(error + '  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit(2)
  for opt, arg in opts:
    if opt in ("-h", "--help"):
      print('test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
      sys.exit()
    elif opt in ("-b", "--baseurl"):
      baseUrl = arg
    elif opt in ("-r", "--region"):
      region = arg
    elif opt in ("-n", "--rootvolname"):
      rootVolName = arg
    elif opt in ("-v", "--volname"):
      volName = arg
    elif opt in ("-a", "--asg"):
      asgName = arg
    elif opt in ("-q", "--qtreename"):
      qtreeName = arg

  try:
    region
  except NameError:
    print('Missing -r, --region.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    baseUrl
  except NameError:
    print('Missing -b --baseurl.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    rootVolName
  except NameError:
    print('Missing -n --rootvolname.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    volName
  except NameError:
    print('Missing -v --volname.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    asgName
  except NameError:
    print('Missing -a --asg.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  try:
    qtreeName
  except NameError:
    print('Missing -q --qtreename.  test.py -b <baseurl> -r <region> -n <rootvolname> -v <volname> -a <autoscaling_group> -q <qtreename>')
    sys.exit()

  print("Script Starting")

  user, password = establishConnection(region)
  os.environ["no_proxy"] = baseUrl
  with HostConnection(baseUrl, username=user, password=password, verify=False):
    exportPolicyRootID = getVolumeExportID(rootVolName)
    exportPolicyVolumeID = getVolumeExportID(volName)
    exportPolicyQtreeID = getQtreeExportID(qtreeName)
    ipAddresses = getAsgIps(region, asgName)
    setExportPolicy(exportPolicyRootID, ipAddresses)
    setExportPolicy(exportPolicyVolumeID, ipAddresses)
    setExportPolicy(exportPolicyQtreeID, ipAddresses)

if __name__ == "__main__":
  main(sys.argv[1:])

[RobertBlackhart]

Same issue. You need to change line 60 from resource = ExportClient(exportPolicyID, exportRuleIndex) to resource = ExportClient(str(exportPolicyID), str(exportRuleIndex))

[sweeneyben]

Any update on if this has been fixed in any version yet?

[RobertBlackhart]

Yes, the mentioned bug was fixed. You'll want to grab version 9.10.1.0 of the library which was released in January: https://pypi.org/project/netapp-ontap/9.10.1.0/

[sweeneyben]

Great, any idea if this would work n-2. Our system is still on 9.8

[RobertBlackhart]

Yes, the library is backwards compatible with prior versions of ONTAP. This is because the REST API is itself backwards compatible. My recommendation would to always use the latest version of the library.

[noorbuchi]

Cleaning up stale issues