Skip to content

Add IAsyncEnumerable pagination support (#58)#70

Merged
daviburg merged 7 commits into
mainfrom
feature/pagination-support
Apr 30, 2026
Merged

Add IAsyncEnumerable pagination support (#58)#70
daviburg merged 7 commits into
mainfrom
feature/pagination-support

Conversation

@daviburg
Copy link
Copy Markdown
Member

Summary

Implements IAsyncEnumerable<T> auto-pagination for paginated connector operations, following the Azure SDK pagination standard.

Closes #58

Breaking Changes

This is a pre-release breaking change (no customer previews shipped yet):

Method Before After
OnedriveforbusinessClient.ListFolderAsync Task<BlobMetadataPage> ConnectorPageable<BlobMetadataPage, BlobMetadata>
TeamsClient.GetMessagesFromChannelAsync Task<GetMessagesFromConversationResponse> ConnectorPageable<GetMessagesFromConversationResponse, ChatMessage>
TeamsClient.GetMessagesFromChatAsync Task<GetMessagesFromConversationResponse> ConnectorPageable<GetMessagesFromConversationResponse, ChatMessage>

What Changed

New Infrastructure

  • IPageable<T> — Interface for page types with Value + NextLink
  • ConnectorPageable<TPage, TItem>IAsyncEnumerable<TItem> that follows NextLink automatically, with AsPages() for page-level access

Paginated Methods (Breaking)

  • ListFolderAsync, GetMessagesFromChannelAsync, GetMessagesFromChatAsync now return ConnectorPageable instead of Task<Page>

Bug Fix (All Connectors)

  • CallConnectorAsync in all 6 connectors now supports absolute NextLink URLs via Uri.IsWellFormedUriString — previously would incorrectly prepend connectionRuntimeUrl to absolute URLs

Tests

  • 8 tests for ConnectorPageable (empty/null pages, single/multi-page, AsPages, cancellation, constructor validation)
  • 4 tests for OneDrive pagination (single-page, multi-page, AsPages, absolute URL verification)

Usage

// Automatic item-level iteration (no NextLink handling needed)
await foreach (var file in client.ListFolderAsync("root"))
{
    Console.WriteLine(file.Name);
}

// Page-level access when needed
await foreach (var page in client.ListFolderAsync("root").AsPages())
{
    Console.WriteLine($"Page: {page.Value.Count} items, NextLink={page.NextLink}");
}

Cross-Language Design

This pattern maps directly to all three SDK languages:

  • C#: await foreach / IAsyncEnumerable<T>
  • Python: async for / __aiter__
  • Node/TS: for await...of / Symbol.asyncIterator

Samples

Companion PR in Connectors-NET-Samples will demonstrate the pattern end-to-end.

…tions (#58)

- Add IPageable<T> interface for page types with Value + NextLink
- Add ConnectorPageable<TPage, TItem> implementing IAsyncEnumerable<TItem>
  with AsPages() for page-level access
- Update BlobMetadataPage and GetMessagesFromConversationResponse to
  implement IPageable<T>
- Change ListFolderAsync, GetMessagesFromChannelAsync, and
  GetMessagesFromChatAsync to return ConnectorPageable (breaking change)
- Fix CallConnectorAsync in all 6 connectors to support absolute
  NextLink URLs via Uri.IsWellFormedUriString
- Add 12 new tests: 8 for ConnectorPageable, 4 for OneDrive pagination
  integration (single-page, multi-page, AsPages, absolute URL)

Closes #58

Co-authored-by: Dobby <dobby@microsoft.com>
@daviburg
Copy link
Copy Markdown
Member Author

Companion samples PR demonstrating the pagination pattern: Azure/Connectors-NET-Samples#25

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds SDK-wide async pagination support so generated connector operations can be consumed via await foreach, aligning with Azure SDK pagination patterns and enabling automatic nextLink traversal.

Changes:

  • Introduces IPageable<T> and ConnectorPageable<TPage, TItem> to support item- and page-level async iteration over paginated connector responses.
  • Updates select generated connector operations to return ConnectorPageable instead of Task<Page> and wires up nextLink traversal.
  • Updates generated CallConnectorAsync URL construction to accept absolute nextLink URLs and adds unit tests covering pagination behavior.

Reviewed changes

Copilot reviewed 4 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tests/Microsoft.Azure.Connectors.Sdk.Tests/OnedriveforbusinessClientTests.cs Adds OneDrive folder listing pagination tests (single/multi-page, AsPages(), absolute nextLink behavior).
tests/Microsoft.Azure.Connectors.Sdk.Tests/ConnectorPageableTests.cs New unit tests for ConnectorPageable enumeration semantics, constructor validation, and cancellation token propagation.
src/Microsoft.Azure.Connectors.Sdk/IPageable.cs Adds a public interface describing a paginated page shape (Value + NextLink).
src/Microsoft.Azure.Connectors.Sdk/ConnectorPageable.cs Adds IAsyncEnumerable<T> implementation that auto-follows NextLink, plus AsPages() for page-level iteration.
src/Microsoft.Azure.Connectors.Sdk/Generated/TeamsExtensions.cs Makes messages list response pageable and updates message listing methods to return ConnectorPageable; adjusts URL building for absolute paths.
src/Microsoft.Azure.Connectors.Sdk/Generated/SharepointonlineExtensions.cs Updates URL building to support absolute paths/nextLinks.
src/Microsoft.Azure.Connectors.Sdk/Generated/OnedriveforbusinessExtensions.cs Makes folder listing response pageable, changes ListFolderAsync to return ConnectorPageable, and updates URL building for absolute paths/nextLinks.
src/Microsoft.Azure.Connectors.Sdk/Generated/Office365Extensions.cs Updates URL building to support absolute paths/nextLinks.
src/Microsoft.Azure.Connectors.Sdk/Generated/MsgraphgroupsanduserExtensions.cs Updates URL building to support absolute paths/nextLinks.
src/Microsoft.Azure.Connectors.Sdk/Generated/KustoExtensions.cs Updates URL building to support absolute paths/nextLinks.

Comment thread src/Microsoft.Azure.Connectors.Sdk/IPageable.cs
Comment thread src/Microsoft.Azure.Connectors.Sdk/ConnectorPageable.cs
Comment thread src/Microsoft.Azure.Connectors.Sdk/ConnectorPageable.cs
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/OnedriveforbusinessExtensions.cs Outdated
Comment thread tests/Microsoft.Azure.Connectors.Sdk.Tests/OnedriveforbusinessClientTests.cs Outdated
@daviburg daviburg self-assigned this Apr 29, 2026
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/KustoExtensions.cs Outdated
daviburg and others added 2 commits April 28, 2026 23:47
- Revert all Generated/ file changes (must go through BPM generator)
- Change IPageable<T>.Value from List<T> to IReadOnlyList<T>
- Fix nextPageFunc doc: clarify URL is as-returned by service
- Add cancellationToken.ThrowIfCancellationRequested() in item and
  page iteration loops for prompt cancellation within large pages
- Add .ConfigureAwait(continueOnCapturedContext: false) to all
  await foreach loops in tests
- Fix useless variable in CancellationToken test: use explicit
  GetAsyncEnumerator + MoveNextAsync instead of unused loop variable
- Remove 4 OneDrive integration tests that depended on generated
  code changes (will be re-added with companion BPM PR)

Co-authored-by: Dobby <dobby@microsoft.com>
Regenerated using CodefulSdkGenerator with pagination support:

- OneDrive: BlobMetadataPage implements IPageable<BlobMetadata>,
  ListFolderAsync returns ConnectorPageable
- Teams: GetMessagesFromConversationResponse implements
  IPageable<ChatMessage>, GetMessagesFromChannelAsync and
  GetMessagesFromChatAsync return ConnectorPageable
- All connectors: ResolveUrl() with SSRF host validation,
  ManagedIdentityCredential updated to new API

Co-authored-by: Dobby <dobby@microsoft.com>
Copilot AI review requested due to automatic review settings April 29, 2026 07:13
@daviburg
Copy link
Copy Markdown
Member Author

Companion BPM generator PR: AzureUX-BPM#15561655. Generated code now regenerated from the updated generator — pagination types, ConnectorPageable returns, ResolveUrl with SSRF host validation, and ManagedIdentityCredential fix all included.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 10 changed files in this pull request and generated 12 comments.

Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/OnedriveforbusinessExtensions.cs Outdated
Comment thread tests/Microsoft.Azure.Connectors.Sdk.Tests/ConnectorPageableTests.cs Outdated
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/TeamsExtensions.cs Outdated
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/TeamsExtensions.cs Outdated
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/TeamsExtensions.cs
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/Office365Extensions.cs
daviburg and others added 2 commits April 29, 2026 00:24
Generated code (from BPM generator update):
- Remove async/await wrapper from ConnectorPageable lambdas
  (direct Task<T> delegation avoids unnecessary state machine)
- Add scheme validation to ResolveUrl (validates both scheme and
  host, not just host, to prevent protocol downgrade attacks)
- Expand 'ct' to 'cancellationToken' in generated lambda parameters

Test fixes:
- Expand 'ct' to 'cancellationToken' in all test lambdas
- Add try/finally with DisposeAsync for IAsyncEnumerator in
  CancellationToken test

Co-authored-by: Dobby <dobby@microsoft.com>
- Remove async/await wrappers from ConnectorPageable lambdas
  (direct Task<T> delegation, no state machine overhead)
- Add scheme validation to ResolveUrl (validates both scheme and
  host to prevent protocol downgrade)
- Expand 'ct' to 'cancellationToken' in generated lambda parameters

Co-authored-by: Dobby <dobby@microsoft.com>
Copilot AI review requested due to automatic review settings April 29, 2026 07:26
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 10 changed files in this pull request and generated 7 comments.

Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/MsgraphgroupsanduserExtensions.cs Outdated
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/Office365Extensions.cs
Comment thread src/Microsoft.Azure.Connectors.Sdk/Generated/KustoExtensions.cs
Comment thread src/Microsoft.Azure.Connectors.Sdk/IPageable.cs
…om non-paginated connectors

Co-authored-by: Dobby <dobby@microsoft.com>
@daviburg daviburg requested a review from Copilot April 29, 2026 17:56
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 10 changed files in this pull request and generated 1 comment.

daviburg added a commit that referenced this pull request Apr 30, 2026
- Generated Office365usersExtensions.cs and AzureloganalyticsExtensions.cs via CodefulSdkGenerator
- Updated ConnectorNames.cs and ManagedConnectors.cs (alphabetical order)
- Added Office365usersClientTests.cs (12 tests: constructor, dispose, mocked API, error handling, serialization)
- Updated README.md validated connectors table
- Updated connection-setup skill connector list

Depends on #70 for IPageable<T> and ConnectorPageable<TPage,TItem> pagination abstractions.

Resolves #8
@daviburg daviburg merged commit a43938f into main Apr 30, 2026
10 checks passed
@daviburg daviburg deleted the feature/pagination-support branch April 30, 2026 22:57
daviburg added a commit that referenced this pull request Apr 30, 2026
- Generated Office365usersExtensions.cs and AzureloganalyticsExtensions.cs via CodefulSdkGenerator
- Updated ConnectorNames.cs and ManagedConnectors.cs (alphabetical order)
- Added Office365usersClientTests.cs (12 tests: constructor, dispose, mocked API, error handling, serialization)
- Updated README.md validated connectors table
- Updated connection-setup skill connector list

Depends on #70 for IPageable<T> and ConnectorPageable<TPage,TItem> pagination abstractions.

Resolves #8
daviburg added a commit that referenced this pull request Apr 30, 2026
- Generated Office365usersExtensions.cs and AzureloganalyticsExtensions.cs via CodefulSdkGenerator
- Updated ConnectorNames.cs and ManagedConnectors.cs (alphabetical order)
- Added Office365usersClientTests.cs (12 tests: constructor, dispose, mocked API, error handling, serialization)
- Updated README.md validated connectors table
- Updated connection-setup skill connector list

Depends on #70 for IPageable<T> and ConnectorPageable<TPage,TItem> pagination abstractions.

Resolves #8
daviburg added a commit that referenced this pull request Apr 30, 2026
## Summary

Adds the **Office 365 Users** (\office365users\) connector client to the
SDK, generated via CodefulSdkGenerator.

### Changes

**Generated code:**
- \Office365usersExtensions.cs\ — typed client with 12 operations:
MyProfile, UserProfile, Manager, DirectReports, SearchUser,
RelevantPeople, TrendingDocuments, UserPhoto, UpdateMyProfile,
UpdateMyPhoto, HttpRequest
- \AzureloganalyticsExtensions.cs\ — also generated (was missing from
source, already in NuGet package)

**Registration:**
- Updated \ConnectorNames.cs\ and \ManagedConnectors.cs\ (alphabetical
order)

**Tests:**
- 12 new tests in \Office365usersClientTests.cs\: constructor, dispose,
mocked API (MyProfile, UserProfile, Manager, DirectReports), error
handling, serialization round-trips (GraphUser, User, Person), IPageable
interface verification
- All **152 tests pass** (0 failures, includes 8 pagination tests from
#70)

**Docs:**
- Updated README validated connectors table
- Updated connection-setup skill connector list

### E2E Validation
- Connection created and consented on \sdk-test-gateway\ (brazilsouth)
- \GetMyProfile\ and \GetManager\ tested locally via \unc start\
- Rebased on main after #70 merged — pagination types (\IPageable<T>\,
\ConnectorPageable\) come from #70

Resolves #8
daviburg added a commit that referenced this pull request May 1, 2026
## Summary

Add comprehensive unit tests and documentation for the
**azureloganalytics** connector (Phase 3.2 - Azure Services).

> **Note:** The generated code (\AzureloganalyticsExtensions.cs\),
connector registration (\ConnectorNames.cs\, \ManagedConnectors.cs\),
and pagination infrastructure (\IPageable.cs\, \ConnectorPageable.cs\)
were merged to main as part of #70 and #75. This PR adds the remaining
pieces.

## What's included

### Tests
- 13 unit tests in \AzureloganalyticsClientTests.cs\: constructor,
dispose, mocked API calls, error handling, serialization round-trips,
multi-page pagination
- All 167 tests pass (expanded from 145 baseline after #70 and #75
merged)

### Docs
- Updated \README.md\ validated connectors table (added Azure Log
Analytics with generated operations)
- Updated connection-setup skill supported names list (added
\�zureloganalytics\)

## E2E Validation

Connection \�zureloganalytics-test\ created and consented on the
\sdk-test-gateway\ Connector Gateway in \�razilsouth\.

## Merge sequencing

This PR is now **ready to merge** — no remaining dependencies. Rebased
onto main after #70 (pagination) and #75 (Office 365 Users) merged.

Closes #8

Co-authored-by: Dobby <dobby@microsoft.com>
daviburg added a commit to Azure/Connectors-NET-Samples that referenced this pull request May 1, 2026
* Update ListOneDriveFolder to use IAsyncEnumerable pagination

- ListFolderAsync now returns IAsyncEnumerable<BlobMetadata> that
  automatically follows NextLink across all pages
- Replace manual page.Value/page.NextLink with await foreach pattern
- Bump SDK to 0.6.0-preview.1.dev (local build with pagination support)
- Add local NuGet source for pre-release validation

Depends on: Azure/Connectors-NET-SDK#70

Co-authored-by: Dobby <dobby@microsoft.com>

* Add GetChannelMessages demonstrating IAsyncEnumerable pagination E2E

- New GetChannelMessages function uses await foreach to iterate
  Teams channel messages across all pages automatically
- Validated E2E: 37 messages returned across 2 pages (20 + 17)
  from live Teams connector with automatic NextLink following

Co-authored-by: Dobby <dobby@microsoft.com>

* Fix CI: revert to published SDK version, remove local NuGet source

Reverts function code to pre-pagination pattern so CI builds with
the currently published SDK (0.6.0-preview.1). The pagination
samples (await foreach on ListFolderAsync, GetChannelMessages) will
be re-added after SDK v0.7.0-preview.1 is published.

Co-authored-by: Dobby <dobby@microsoft.com>

* Revert "Fix CI: revert to published SDK version, remove local NuGet source"

This reverts commit a0833e9.

* Update to published SDK 0.7.0-preview.1 from nuget.org, remove local source

Co-authored-by: Dobby <dobby@microsoft.com>

---------

Co-authored-by: Dobby <dobby@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generated clients lack pagination support (@odata.nextLink)

2 participants