[Cost Management] [API Management] Azure Cost Management Package just throws first 1000 lines and not managing the paging from the cost management rest api

[victorsalesa]

• Package Name: azure-mgmt-costmanagement
• Package Version: 3.0.0
• Operating System: Windows
• Python Version: 3.9.6

Describe the bug
When you call the Usage method in python: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/costmanagement/azure-mgmt-costmanagement/azure/mgmt/costmanagement/operations/_query_operations.py
that is related to this REST api endpoint here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage
the python api is not looping to get next page using the nextLink attribute in the response described here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage#queryresult
so we get a 1000 rows maximum per execution and no way to manage the paging with this library

To Reproduce

from azure.mgmt.costmanagement        import CostManagementClient
from azure.mgmt.costmanagement.models import QueryAggregation,QueryGrouping,QueryDataset,QueryDefinition,QueryTimePeriod,QueryFilter,QueryComparisonExpression
from azure.mgmt.resource              import ResourceManagementClient
from azure.identity                   import DefaultAzureCredential
from IPython.display                  import display, HTML
from typing                           import ContextManager

import json
import pandas as pd
import datetime as dt
import calendar
import numpy as np


thedate = dt.datetime.combine(dt.date.today(), dt.time())
first = thedate.replace(day=1)
last = thedate.replace(day = calendar.monthrange(thedate.year, thedate.month)[1])

credential = DefaultAzureCredential()
subscription_id = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

scope = '/subscriptions/{}'.format(subscription_id)

client = ResourceManagementClient(credential, subscription_id)

cmgmtc = CostManagementClient(credential = credential)

"""
'ResourceGroup','ResourceGroupName','ResourceLocation',
'ConsumedService','ResourceType','ResourceId',
'MeterId','BillingMonth','MeterCategory',
'MeterSubcategory','Meter','AccountName',
'DepartmentName','SubscriptionId','SubscriptionName',
'ServiceName','ServiceTier','EnrollmentAccountName',
'BillingAccountId','ResourceGuid','BillingPeriod',
'InvoiceNumber','ChargeType','PublisherType',
'ReservationId','ReservationName','Frequency',
'PartNumber','CostAllocationRuleName','MarkupRuleName',
'PricingModel','BenefitId','BenefitName',''
"""


query_template = ( 
  QueryDefinition( 
     type      = "ActualCost"
    , timeframe = "ThisMonth" 
   , dataset   = 
     QueryDataset(
        granularity  = "Monthly"
      , aggregation  = { 
        "totalCost": QueryAggregation(name = "Cost", function = "Sum")
        ,"totalCostUSD": QueryAggregation(name = "CostUSD", function = "Sum") 
        }
      , grouping     = [
             QueryGrouping(name = "ResourceGroupName", type = "Dimension")
            ,QueryGrouping(name = "ResourceId"       , type = "Dimension")
            ,QueryGrouping(name = "ResourceType"     , type = "Dimension")
        ]
      , filter =  
        QueryFilter(
          dimensions = 
            QueryComparisonExpression(
                name = "ResourceGroupName"
              , operator = "In"
              , values = ["RESOURCE_GROUP"]
            )
        )
     )
  )
)



replaced_query = (
  query_template.deserialize(
    json.loads(
      json.dumps(
        query_template.serialize()
      ).replace('RESOURCE_GROUP','destination_rg')
    )
  )
)

result = cmgmtc.query.usage( scope = scope, parameters = replaced_query)

data = pd.DataFrame(result.rows, columns = list(map(lambda col: col.name, result.columns)))
 
data_sorted = data.sort_values(by='CostUSD' ,ascending = False)

data_filtered = data_sorted

pd.set_option('display.max_rows', data_filtered.shape[0]+1)

display(HTML(data_filtered.to_html())) 

Expected behavior
As a python developer using this package I would expect the result to be an iterable so I can get all result pages not just the first one

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'ARM:0.1822567,Cosmos:0.06052991,API Management:0.02198743'

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @ccmixpdevs.

Issue Details

---

• Package Name: azure-mgmt-costmanagement
• Package Version: 3.0.0
• Operating System: Windows
• Python Version: 3.9.6

Describe the bug
When you call the Usage method in python: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/costmanagement/azure-mgmt-costmanagement/azure/mgmt/costmanagement/operations/_query_operations.py
that is related to this REST api endpoint here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage
the python api is not looping to get next page using the nextLink attribute in the response described here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage#queryresult
so we get a 1000 rows maximum per execution and no way to manage the paging with this library

To Reproduce

from azure.mgmt.costmanagement        import CostManagementClient
from azure.mgmt.costmanagement.models import QueryAggregation,QueryGrouping,QueryDataset,QueryDefinition,QueryTimePeriod,QueryFilter,QueryComparisonExpression
from azure.mgmt.resource              import ResourceManagementClient
from azure.identity                   import DefaultAzureCredential
from IPython.display                  import display, HTML
from typing                           import ContextManager

import json
import pandas as pd
import datetime as dt
import calendar
import numpy as np


thedate = dt.datetime.combine(dt.date.today(), dt.time())
first = thedate.replace(day=1)
last = thedate.replace(day = calendar.monthrange(thedate.year, thedate.month)[1])

credential = DefaultAzureCredential()
subscription_id = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

scope = '/subscriptions/{}'.format(subscription_id)

client = ResourceManagementClient(credential, subscription_id)

cmgmtc = CostManagementClient(credential = credential)

"""
'ResourceGroup','ResourceGroupName','ResourceLocation',
'ConsumedService','ResourceType','ResourceId',
'MeterId','BillingMonth','MeterCategory',
'MeterSubcategory','Meter','AccountName',
'DepartmentName','SubscriptionId','SubscriptionName',
'ServiceName','ServiceTier','EnrollmentAccountName',
'BillingAccountId','ResourceGuid','BillingPeriod',
'InvoiceNumber','ChargeType','PublisherType',
'ReservationId','ReservationName','Frequency',
'PartNumber','CostAllocationRuleName','MarkupRuleName',
'PricingModel','BenefitId','BenefitName',''
"""


query_template = ( 
  QueryDefinition( 
     type      = "ActualCost"
    , timeframe = "ThisMonth" 
   , dataset   = 
     QueryDataset(
        granularity  = "Monthly"
      , aggregation  = { 
        "totalCost": QueryAggregation(name = "Cost", function = "Sum")
        ,"totalCostUSD": QueryAggregation(name = "CostUSD", function = "Sum") 
        }
      , grouping     = [
             QueryGrouping(name = "ResourceGroupName", type = "Dimension")
            ,QueryGrouping(name = "ResourceId"       , type = "Dimension")
            ,QueryGrouping(name = "ResourceType"     , type = "Dimension")
        ]
      , filter =  
        QueryFilter(
          dimensions = 
            QueryComparisonExpression(
                name = "ResourceGroupName"
              , operator = "In"
              , values = ["RESOURCE_GROUP"]
            )
        )
     )
  )
)



replaced_query = (
  query_template.deserialize(
    json.loads(
      json.dumps(
        query_template.serialize()
      ).replace('RESOURCE_GROUP','destination_rg')
    )
  )
)

result = cmgmtc.query.usage( scope = scope, parameters = replaced_query)

data = pd.DataFrame(result.rows, columns = list(map(lambda col: col.name, result.columns)))
 
data_sorted = data.sort_values(by='CostUSD' ,ascending = False)

data_filtered = data_sorted

pd.set_option('display.max_rows', data_filtered.shape[0]+1)

display(HTML(data_filtered.to_html())) 

Expected behavior
As a python developer using this package I would expect the result to be an iterable so I can get all result pages not just the first one

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Author:	victorsalesa
Assignees:	-
Labels:	question, Service Attention, Mgmt, customer-reported, needs-triage, Cost Management - Query
Milestone:	-

[annatisch]

Thanks @victorsalesa, you're correct - it does not look like this operation has been identified as pageable in the specification that is used to generate the SDK:
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cost-management/resource-manager/Microsoft.CostManagement/stable/2021-10-01/costmanagement.json#L950

I have tagged the Cost Management team to hopefully get this resolved ASAP :)

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @ccmixpdevs.

Issue Details

---

• Package Name: azure-mgmt-costmanagement
• Package Version: 3.0.0
• Operating System: Windows
• Python Version: 3.9.6

Describe the bug
When you call the Usage method in python: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/costmanagement/azure-mgmt-costmanagement/azure/mgmt/costmanagement/operations/_query_operations.py
that is related to this REST api endpoint here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage
the python api is not looping to get next page using the nextLink attribute in the response described here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage#queryresult
so we get a 1000 rows maximum per execution and no way to manage the paging with this library

To Reproduce

from azure.mgmt.costmanagement        import CostManagementClient
from azure.mgmt.costmanagement.models import QueryAggregation,QueryGrouping,QueryDataset,QueryDefinition,QueryTimePeriod,QueryFilter,QueryComparisonExpression
from azure.mgmt.resource              import ResourceManagementClient
from azure.identity                   import DefaultAzureCredential
from IPython.display                  import display, HTML
from typing                           import ContextManager

import json
import pandas as pd
import datetime as dt
import calendar
import numpy as np


thedate = dt.datetime.combine(dt.date.today(), dt.time())
first = thedate.replace(day=1)
last = thedate.replace(day = calendar.monthrange(thedate.year, thedate.month)[1])

credential = DefaultAzureCredential()
subscription_id = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

scope = '/subscriptions/{}'.format(subscription_id)

client = ResourceManagementClient(credential, subscription_id)

cmgmtc = CostManagementClient(credential = credential)

"""
'ResourceGroup','ResourceGroupName','ResourceLocation',
'ConsumedService','ResourceType','ResourceId',
'MeterId','BillingMonth','MeterCategory',
'MeterSubcategory','Meter','AccountName',
'DepartmentName','SubscriptionId','SubscriptionName',
'ServiceName','ServiceTier','EnrollmentAccountName',
'BillingAccountId','ResourceGuid','BillingPeriod',
'InvoiceNumber','ChargeType','PublisherType',
'ReservationId','ReservationName','Frequency',
'PartNumber','CostAllocationRuleName','MarkupRuleName',
'PricingModel','BenefitId','BenefitName',''
"""


query_template = ( 
  QueryDefinition( 
     type      = "ActualCost"
    , timeframe = "ThisMonth" 
   , dataset   = 
     QueryDataset(
        granularity  = "Monthly"
      , aggregation  = { 
        "totalCost": QueryAggregation(name = "Cost", function = "Sum")
        ,"totalCostUSD": QueryAggregation(name = "CostUSD", function = "Sum") 
        }
      , grouping     = [
             QueryGrouping(name = "ResourceGroupName", type = "Dimension")
            ,QueryGrouping(name = "ResourceId"       , type = "Dimension")
            ,QueryGrouping(name = "ResourceType"     , type = "Dimension")
        ]
      , filter =  
        QueryFilter(
          dimensions = 
            QueryComparisonExpression(
                name = "ResourceGroupName"
              , operator = "In"
              , values = ["RESOURCE_GROUP"]
            )
        )
     )
  )
)



replaced_query = (
  query_template.deserialize(
    json.loads(
      json.dumps(
        query_template.serialize()
      ).replace('RESOURCE_GROUP','destination_rg')
    )
  )
)

result = cmgmtc.query.usage( scope = scope, parameters = replaced_query)

data = pd.DataFrame(result.rows, columns = list(map(lambda col: col.name, result.columns)))
 
data_sorted = data.sort_values(by='CostUSD' ,ascending = False)

data_filtered = data_sorted

pd.set_option('display.max_rows', data_filtered.shape[0]+1)

display(HTML(data_filtered.to_html())) 

Expected behavior
As a python developer using this package I would expect the result to be an iterable so I can get all result pages not just the first one

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Author:	victorsalesa
Assignees:	-
Labels:	question, Service Attention, Mgmt, customer-reported, needs-team-attention, needs-team-triage, Cost Management - Query
Milestone:	-

[omrodrig]

Sorry for the inconvenience. but we have a backlog item to fix this issue.

[victorsalesa]

Sorry for the inconvenience. but we have a backlog item to fix this issue.

Hi @omrodrig . Am I able to access that backlog item to track the issue status?. Could you please provide a link?
Thanks

[msyyc]

Hi @victorsalesa , python SDK is generated with swagger. Just like #23175 (comment), swagger doesn't define the operation as pageable so SDK will not treat it as pageable. If you want the feature, it is better to make issue in https://github.com/Azure/azure-rest-api-specs/issues. Thanks!

[ghost]

Hi @victorsalesa. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @victorsalesa, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

[sgiovetti]

Sorry, i'm having the same problem, but from the thread i don't understand if the issue will be fixed or not. Thanks in advance for your help. In the meantime did you find some workaround ?

[victorsalesa]

@sgiovetti I was the one notificating that and copying that. It seems that the python api is generated with azurerest tool. I guess that it will be simply changing the swagger definition for the next release to include the iteration capability but no answer yet

[tanbyte]

Hi folks - was this issue ever resolved? If so, what package version is it fixed on?