-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Issue with setting up x-ms-pageable in Custom Connector #2959
Comments
I have handled paging before as part of a client project. I am unable to share that code, but the direction I would point you towards is having the custom code retrieve a set number of records (5000?) before returning one response. If you look here, you will see x-ms-pageable is not fully supported: https://learn.microsoft.com/en-us/connectors/custom-connectors/openapi-extensions |
I have looked into the suggestion above. I have code that does this, bit it struggles to keep within the 5 seconds even with all kind of optimizations where I get multiple pages in parallel. I also looked further and apparently I did not do a good enough search in this repo. There are multiple connectors that use the The AzureADApplications-connector is created by @paulculmsee and @Amjed-Ayoub. Can any of you comment on this if the |
Would love to get this going as well, so any hints are more than welcome. Also, here is a suggestion to officially support x-ms-pageable in custom connectors: https://ideas.powerautomate.com/d365community/idea/db595779-e186-ee11-a81c-6045bdbcd0a8 Please vote if you agree. |
I have voted :) Thanks for creating the Idea. |
I've also got an open ticket with Microsoft about pagination. We're trying to enable pagination for calls to the Microsoft Graph API endpoint, however, in our case, when pagination is enabled on the action after adding to our custom connectors swagger file for the get users action,
We see the "pagination" option in the actions settings. However, when pagination is enabled we get the following error,
Yet when disabled, authentication works fine. Trying to pin the problem down is like extracting teeth (painful). Can anyone shed any light on the issue we're seeing? If I can get it working, I'll document the steps I took and will post it back here for anyone else having issues with pagination! |
Thanks @SOE-YoungS , |
I've come to the conclusion that the only way to "properly" handle "x-ms-pageable" actions is to use the paconn tool to create the connector. So, I set about converting my connectors swagger to "swagger.json" format. It already had "x-ms-pageable" defined. Next up, I created the two files required to create a connector with paconn by renaming my "swagger.json" file to "apiDefinition.swagger.json" and a blank "apiProperties.json" file. I then set about adding the required JSON to "apiProperties.json", to enable me to upload the connector to PA. I then used paconn to download the connector & edited the apiProperties.json file. The crucial bit here is that the following needs to be added to the "apiProperties .json",
Note: Edit {OperationId*} to match your target "OperationId(s)". Once this was added, I reuploaded my connector via paconn, then tested it out. Success! Pagination works, without error! |
Wow, that looks very promising @SOE-YoungS ! However, I am getting 504 error: "The gateway did not receive a response within the specified time period" |
I couldn't honestly say. I didn't have that issue. When it displays the list of environments, are you selecting your default environment? I have an open ticket for ours currently as creating / modifying connectors within it hasn't worked since the "copilot" updates were released last week. Targeting other environments works fine, but our default env is throwing errors constantly which has somewhat hindered us. |
Thanks @SOE-YoungS , However, when executing with pagination set on, I am then getting Without pagination works fine, but with it, this error appears. |
Auth scheme of "Key"? Should this be "API Key"? in which case, you may be missing some single / double quotes in the schema somewhere. |
Thanks again for bearing with me :)
And in the Definition file the relevant sections look like so:
.. using delegated permissions. Trying to grab Power Automate data from a basically undocumented API: Using Logic Apps endpoints on this URL: https://emea.api.flow.microsoft.com/providers/Microsoft.ProcessSimple |
You don't appear to have any scopes defined. I believe you'll need at least "User.Read" to enable login. You can configure and test the auth via the UI to ensure you have that correct, then once tested as working, switch to paconn & download the connector again, insert the snippet to enable the pagination into "apiProperties.json", then update the connector. Note: If you make any changes in the future to the connector via the PA UI, you will need to run that second step again after clicking "update connector" in the PA UI. |
Hi @SOE-YoungS , But of course, will give the scopes a try in the definition of the Power Automate candidate asap. |
No luck with the scopes in the security definition. |
@ImkeF it dawned on me this evening that the api you are calling may not support '@odata.nextLink' usage. Having looked at the TribeCRM.nl postman collection, I've come to the conclusion that your C# should work. Looking deeper at it, I have a feeling that in your case pagination is not working, because "$orderby" may not be set (it's flagged as required for pagination in the API documentation).
Secondly, I think there is an issue with your $skip logic.
shouldn't this be,
|
Type of Connector
Custom Connector
Name of Connector
Tribe CRM
Describe the bug
I'm building a Custom Connector on the Odata service of Tribe CRM. The default page size is 100 rows. This is also the maximum number of rows that can be retrieved in one call. For listing rows I want to use Pagination to retrieve more rows.
I've added the following to the OpenAPI definition to my list rows.
By default the Tribe CRM API does not have a @odata.nextLink element. So with Custom Code I construct the correct URL and add the @odata.nextLink in the JSON response. This works OK.
I can now enable pagination in the action in Power Automate and set the threshold to like 5000. But the result is just 200 records, or better said the first two pages.
How do I know? I ended up with a Unauthorized error when pagination was enabled, it retrieves OK when a single page is retrieved. When I manually put the correct authorization in the query string when constructing the next link the error went away. So no Authorization headers were added by the Custom Connector. This second page also did not have the nextLink added to the output. Also, I added additional Custom Code for debugging, to make sure that my Custom Code is indeed not called.
Is this a security bug?
No, this is not a security bug
What is the severity of this bug?
Severity 1 - Connector is broken or there's a very serious issue
To Reproduce
Expected behavior
When pagination is enabled, retrieving each page should be handled by the Custom Connector. This includes any authentication, policies or Custom Code that needs to run in order to retrieve the next page correctly.
Environment summary
I have a dev environment in Power Platform where I develop my Custom Connector.
I moved the connector to a Sandbox environment. Both environments have the same result.
Additional context
What have I tried and/or found?
I've looked at the x-ms-pageable docs in Autorest This mentioned that you can also supply 'operationName'. But trying to do this I get a validation error in de Custom Connector editor.
I've looked at the underlying JSON Schema of the Custom connector here on GitHub. There it document x-ms-pageable as well (and indeed does not include 'operationName'). It mentions the following as description for the nextlink url:
I'm unable to find any reference or documentation to policy template 'updatenextlink'.
I've tried adding a 'Next' operation to my OpenAPI definition, as is indicated being the default operationName on x-ms-pageable. But you cannot have multiple operations on the same path with the same HTTP Verb. So I'm not sure how I would add this to my OpenAPI definition.
Who can help me get the paging working for my Custom Connector? What am I doing wrong? It needs to Custom Code in order to add the @odata.nextLink in de response. So it needs to call my Custom Connector to do so.
Swagger definition:
Custom Code:
The text was updated successfully, but these errors were encountered: