Pagination for microsoft graph module

[Drakonian]

Summary

Pagination for MicrosoftGraph module

I will provide more examples and tests soon

Work Item(s)

Fixes #3926

[pri-kise]

I like the design so far.

Following ideas for test cases:

• Request two items (no NextLink) -> Verify that HasMorePages returns false + NextLink is empty
• Request two items (with NextLink) -> Verify that HasMorePages returns true + NextLink contains a value.

[JesperSchulz]

No objections to the code in its current shape. It's coming along nicely!

[Drakonian]

I’ve completed development and testing, including validation on a real tenant. The code behaves as expected.

Additionally, I’ve prepared test codeunits for the new pagination methods and codeunits.

I’ve also written a README with usage instructions for these procedures (see below).

@JesperSchulz @pri-kise

[Drakonian]

Microsoft Graph API Pagination Guide

This guide explains how to use the new pagination functionality in the Microsoft Graph module for Business Central. Pagination is essential when working with large datasets from Microsoft Graph API.

Table of Contents

• Microsoft Graph API Pagination Guide
  • Table of Contents
  • Overview
  • Key Components
    • GraphPaginationData Codeunit
    • New Graph Client Methods
  • Usage Examples
    • Prerequisites
    • 1. Simple Automatic Pagination
    • 2. Manual Page-by-Page Processing
    • 3. Filtered Pagination
    • 4. Getting Total Count
  • Advanced Usage
    • Combining with Skip Parameter
    • Progress Dialog for Large Datasets
  • Best Practices
  • API Reference
    • Graph Client Methods
      • GetWithPagination
      • GetNextPage
      • GetAllPages
    • GraphPaginationData Methods
  • See Also

Overview

Microsoft Graph API returns data in pages to improve performance. When querying large datasets (e.g., all users in a tenant), the API returns a subset of results along with a @odata.nextLink URL to fetch the next page.

The new pagination functionality provides three main approaches:

1. Automatic pagination - Fetch all pages at once
2. Manual pagination - Process page by page with full control
3. Hybrid approach - Combine with filters and custom processing

Key Components

GraphPaginationData Codeunit

Manages pagination state including:

• Next page URL (@odata.nextLink)
• Page size configuration
• Pagination status

New Graph Client Methods

• GetWithPagination() - Fetches first page with pagination support
• GetNextPage() - Retrieves the next page of results
• GetAllPages() - Automatically fetches all pages and combines results

Usage Examples

Prerequisites

All examples assume you have initialized the Graph Client with proper authentication:

var
    GraphClient: Codeunit "Graph Client";
    GraphAuthorization: Codeunit "Graph Authorization";
    GraphAuthInterface: Interface "Graph Authorization";
begin
    // Initialize with your authentication method
    GraphAuthInterface := GraphAuthorization.CreateAuthorizationWithClientCredentials(
        TenantId,
        ClientId,
        ClientSecret,
        'https://graph.microsoft.com/.default'
    );
    
    GraphClient.Initialize(Enum::"Graph API Version"::"v1.0", GraphAuthInterface);
end;

1. Simple Automatic Pagination

The easiest way to get all results - let the module handle pagination automatically:

procedure GetAllUsersSimple()
var
    GraphClient: Codeunit "Graph Client";
    GraphOptionalParameters: Codeunit "Graph Optional Parameters";
    HttpResponseMessage: Codeunit "Http Response Message";
    AllUsers: JsonArray;
begin
    // Get all users automatically - the module handles pagination internally
    if GraphClient.GetAllPages('users', GraphOptionalParameters, HttpResponseMessage, AllUsers) then
        Message('Retrieved %1 users', AllUsers.Count())
    else
        Error('Failed to retrieve users: %1', HttpResponseMessage.GetReasonPhrase());
end;

2. Manual Page-by-Page Processing

For better control and memory management with large datasets:

procedure ProcessUsersPageByPage()
var
    GraphClient: Codeunit "Graph Client";
    GraphPaginationData: Codeunit "Graph Pagination Data";
    GraphOptionalParameters: Codeunit "Graph Optional Parameters";
    HttpResponseMessage: Codeunit "Http Response Message";
    PageNumber: Integer;
    TotalUsers: Integer;
begin
    // Configure pagination
    GraphPaginationData.SetPageSize(25); // 25 items per page
    
    // Optional: Select specific fields to reduce payload
    GraphOptionalParameters.SetODataQueryParameter(
        Enum::"Graph OData Query Parameter"::select,
        'id,displayName,mail'
    );

    // Get first page
    if not GraphClient.GetWithPagination('users', GraphOptionalParameters, GraphPaginationData, HttpResponseMessage) then
        Error('Failed to retrieve first page');

    PageNumber := 1;
    ProcessPage(HttpResponseMessage, PageNumber, TotalUsers);

    // Process remaining pages
    while GraphPaginationData.HasMorePages() do begin
        if not GraphClient.GetNextPage(GraphPaginationData, HttpResponseMessage) then
            Error('Failed to retrieve page %1', PageNumber + 1);

        PageNumber += 1;
        ProcessPage(HttpResponseMessage, PageNumber, TotalUsers);
    end;

    Message('Processed %1 users across %2 pages', TotalUsers, PageNumber);
end;

local procedure ProcessPage(HttpResponseMessage: Codeunit "Http Response Message"; PageNumber: Integer; var TotalUsers: Integer)
var
    ResponseJson: JsonObject;
    ValueArray: JsonArray;
    JsonToken: JsonToken;
begin
    if ResponseJson.ReadFrom(HttpResponseMessage.GetContent().AsText()) then
        if ResponseJson.Get('value', JsonToken) then begin
            ValueArray := JsonToken.AsArray();
            TotalUsers += ValueArray.Count();
            
            // Process each item in the current page
            // Your business logic here...
        end;
end;

3. Filtered Pagination

Combine pagination with OData filters for efficient data retrieval:

procedure GetFilteredUsers()
var
    GraphClient: Codeunit "Graph Client";
    GraphOptionalParameters: Codeunit "Graph Optional Parameters";
    HttpResponseMessage: Codeunit "Http Response Message";
    FilteredUsers: JsonArray;
begin
    // Filter for enabled users whose display name starts with 'A'
    GraphOptionalParameters.SetODataQueryParameter(
        Enum::"Graph OData Query Parameter"::filter,
        'accountEnabled eq true and startswith(displayName, ''A'')'
    );

    // Select specific fields
    GraphOptionalParameters.SetODataQueryParameter(
        Enum::"Graph OData Query Parameter"::select,
        'id,displayName,mail,accountEnabled,createdDateTime'
    );

    // For complex queries, might need consistency level
    GraphOptionalParameters.SetConsistencyLevelRequestHeader('eventual');

    // Get all filtered results
    if GraphClient.GetAllPages('users', GraphOptionalParameters, HttpResponseMessage, FilteredUsers) then
        Message('Found %1 filtered users', FilteredUsers.Count())
    else
        Error('Failed to retrieve filtered users');
end;

4. Getting Total Count

Get the total count before processing to show progress or confirm with user:

procedure GetUsersWithCount()
var
    GraphClient: Codeunit "Graph Client";
    GraphPaginationData: Codeunit "Graph Pagination Data";
    GraphOptionalParameters: Codeunit "Graph Optional Parameters";
    HttpResponseMessage: Codeunit "Http Response Message";
    ResponseJson: JsonObject;
    JsonToken: JsonToken;
    TotalCount: Integer;
begin
    // Request count
    GraphOptionalParameters.SetODataQueryParameter(
        Enum::"Graph OData Query Parameter"::count, 
        'true'
    );
    GraphOptionalParameters.SetConsistencyLevelRequestHeader('eventual');

    // Set page size
    GraphPaginationData.SetPageSize(50);

    // Get first page with count
    if GraphClient.GetWithPagination('users', GraphOptionalParameters, GraphPaginationData, HttpResponseMessage) then begin
        if ResponseJson.ReadFrom(HttpResponseMessage.GetContent().AsText()) then
            if ResponseJson.Get('@odata.count', JsonToken) then begin
                TotalCount := JsonToken.AsValue().AsInteger();
                Message('Total users in tenant: %1', TotalCount);

                // Decide whether to continue based on count
                if TotalCount > 1000 then
                    if not Confirm('There are %1 users. Continue?', false, TotalCount) then
                        exit;

                // Process remaining pages...
            end;
    end;
end;

Advanced Usage

Combining with Skip Parameter

Skip a certain number of records before starting pagination:

// Skip first 100 users, then paginate
GraphOptionalParameters.SetODataQueryParameter(
    Enum::"Graph OData Query Parameter"::skip, 
    '100'
);
GraphPaginationData.SetPageSize(25);

GraphClient.GetWithPagination('users', GraphOptionalParameters, GraphPaginationData, HttpResponseMessage);

Progress Dialog for Large Datasets

Show progress when processing many pages:

procedure ProcessWithProgress(var GraphClient: Codeunit "Graph Client"; var GraphPaginationData: Codeunit "Graph Pagination Data"; TotalCount: Integer)
var
    ProgressDialog: Dialog;
    HttpResponseMessage: Codeunit "Http Response Message";
    ProcessedCount: Integer;
begin
    ProgressDialog.Open('Processing users\@1@@@@@@@@@@@@@@@@@@@@', ProcessedCount);

    while GraphPaginationData.HasMorePages() do
        if GraphClient.GetNextPage(GraphPaginationData, HttpResponseMessage) then begin
            ProcessedCount += GetPageItemCount(HttpResponseMessage);
            ProgressDialog.Update(1, Round(ProcessedCount / TotalCount * 10000, 1));
        end;

    ProgressDialog.Close();
end;

Best Practices

1. Choose the Right Method

   • Use GetAllPages() for small to medium datasets
   • Use manual pagination for large datasets or when you need processing control
   • Always set a reasonable page size (default is 100)

2. Performance Optimization

   • Use $select to retrieve only needed fields
   • Apply filters to reduce the dataset size
   • Consider using $skip for resumable operations

3. Page Size Guidelines

   • Default: 100 items per page
   • Maximum: 999 items per page
   • For complex objects, use smaller page sizes (25-50)

4. Safety Limits

   • The module enforces a maximum of 1000 iterations to prevent infinite loops
   • Always check HasMorePages() before calling GetNextPage()

5. Memory Management

   • Process pages individually for large datasets
   • Clear processed data from memory when no longer needed
   • Consider batching operations

API Reference

Graph Client Methods

GetWithPagination

procedure GetWithPagination(
    RelativeUriToResource: Text; 
    GraphOptionalParameters: Codeunit "Graph Optional Parameters"; 
    var GraphPaginationData: Codeunit "Graph Pagination Data"; 
    var HttpResponseMessage: Codeunit "Http Response Message"
): Boolean

Fetches the first page of results with pagination support.

GetNextPage

procedure GetNextPage(
    var GraphPaginationData: Codeunit "Graph Pagination Data"; 
    var HttpResponseMessage: Codeunit "Http Response Message"
): Boolean

Retrieves the next page using the stored pagination data.

GetAllPages

procedure GetAllPages(
    RelativeUriToResource: Text; 
    GraphOptionalParameters: Codeunit "Graph Optional Parameters"; 
    var HttpResponseMessage: Codeunit "Http Response Message"; 
    var JsonResults: JsonArray
): Boolean

Automatically fetches all pages and combines results into a single JsonArray.

GraphPaginationData Methods

• SetPageSize(NewPageSize: Integer) - Set items per page (1-999)
• GetPageSize(): Integer - Get current page size
• HasMorePages(): Boolean - Check if more pages exist
• GetNextLink(): Text - Get the URL for the next page
• Reset() - Reset pagination state

See Also

• Microsoft Graph REST API Reference
• OData Query Parameters
• Microsoft Graph Throttling

[pri-kise]

The example responses could be moved to the ressources files.
I'm not sure what's the best practice there.

[Drakonian]

I prefer to leave it all as it is, it's purposely done exactly the same way as the Graph Client Test, I think code in the same style is important

[pri-kise]

I'm not sure if it's a good pattern to have multiple test cases in one test procedure.

[Drakonian]

I think it depends on the context, of course there are recommendations to test one event per test. But I don't think that's always a good idea and is a directly inviolable rule.

I prefer to follow common sense, for example, you can blindly follow the principles from the book “Clean Code” by Robert Martin, which will make the cyclomatic complexity of your code much worse. Or, on the contrary, you may write functions for 500+ lines, mixing everything, which will make your code unsupported.

The truth is somewhere in the middle from my point of view :)

That's why in this particular place, I think it's quite normal to have three checks of one function.