Connector builder documentation: Add sections about custom params (#2…

…5546)

* add sections about custom params

* Update docs/connector-development/connector-builder-ui/incremental-sync.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update docs/connector-development/connector-builder-ui/incremental-sync.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

---------

Co-authored-by: Lake Mossman <lake@airbyte.io>

docs/connector-development/connector-builder-ui/incremental-sync.md

@@ -130,4 +130,17 @@ Reiterating the example from above with a "Lookback window" of 2 days configured
 Then when a sync is triggered for the same connection the next day, the following request is made:
 <pre>
 curl 'https://content.guardianapis.com/search?order-by=oldest&from-date=<b>2023-04-13T07:30:58Z</b>&to-date={`<now>`}'
-</pre>
+</pre>
+
+## Custom parameter injection
+
+Using the "Inject start time / end time into outgoing HTTP request" option in the incremental sync form works for most cases, but sometimes the API has special requirements that can't be handled this way:
+* The API requires adding a prefix or a suffix to the actual value
+* Multiple values need to be put together in a single parameter
+* The value needs to be injected into the URL path
+* Some conditional logic needs to be applied
+
+To handle these cases, disable injection in the incremental sync form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers and properties of the JSON body, by using jinja expressions and [available variables](/connector-development/config-based/understanding-the-yaml-file/reference/#/variables). You can also use these variables as part of the URL path.
+
+For example the [Sendgrid API](https://docs.sendgrid.com/api-reference/e-mail-activity/filter-all-messages) requires setting both start and end time in a `query` parameter.
+For this case, you can use the `stream_interval` variable to configure a request parameter with "key" `query` and "value" `last_event_time BETWEEN TIMESTAMP "{{stream_interval.start_time}}" AND TIMESTAMP "{{stream_interval.end_time}}"` to filter down to the right window in time.

docs/connector-development/connector-builder-ui/pagination.md

@@ -261,3 +261,16 @@ The following APIs implement cursor pagination in various ways:
 - [Twitter API](https://developer.twitter.com/en/docs/twitter-api/pagination) - includes `next_token` IDs in its responses which are passed in as request parameters to subsequent requests
 - [GitHub API](https://docs.github.com/en/rest/guides/using-pagination-in-the-rest-api?apiVersion=2022-11-28) - includes full-URL `link`s to subsequent pages of results
 - [FourSquare API](https://location.foursquare.com/developer/reference/pagination) - includes full-URL `link`s to subsequent pages of results
+
+## Custom parameter injection
+
+Using the "Inject page size / limit / offset into outgoing HTTP request" option in the pagination form works for most cases, but sometimes the API has special requirements that can't be handled this way:
+* The API requires to add a prefix or a suffix to the actual value
+* Multiple values need to be put together in a single parameter
+* The value needs to be injected into the URL path
+* Some conditional logic needs to be applied
+
+To handle these cases, disable injection in the pagination form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers and properties of the JSON body, by using jinja expressions and [available variables](/connector-development/config-based/understanding-the-yaml-file/reference/#/variables). You can also use these variables as part of the URL path.
+
+For example the [Prestashop API](https://devdocs.prestashop-project.org/8/webservice/cheat-sheet/#list-options) requires to set offset and limit separated by a comma into a single request parameter (`?limit=<offset>,<limit>`)
+For this case, you can use the `next_page_token` variable to configure a request parameter with key `limit` and value `{{ next_page_token['next_page_token'] or '0' }},50` to inject the offset from the pagination strategy and a hardcoded limit of 50 into the same parameter.

docs/connector-development/connector-builder-ui/partitioning.md

@@ -115,4 +115,13 @@ However the order id can be added by taking the following steps:
 Using this configuration, the notes record looks like this:
 ```
 { "id": 999, "author": "Jon Doe", "note": "Great product!", "order_id": 123 }
-```
+```
+## Custom parameter injection
+
+Using the "Inject partition value into outgoing HTTP request" option in the partitioning form works for most cases, but sometimes the API has special requirements that can't be handled this way:
+* The API requires to add a prefix or a suffix to the actual value
+* Multiple values need to be put together in a single parameter
+* The value needs to be injected into the URL path
+* Some conditional logic needs to be applied
+
+To handle these cases, disable injection in the partitioning form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers and properties of the JSON body, by using jinja expressions and [available variables](/connector-development/config-based/understanding-the-yaml-file/reference/#/variables). You can also use these variables (like `stream_partition`) as part of the URL path as shown in the Woocommerce example above.