Skip to content

Update universal-connector-source.md #5550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 3, 2025
Merged

Update universal-connector-source.md #5550

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
17cd09e
Update universal-connector-source.md
yleiferman Jul 3, 2025
966f0ee
Update universal-connector-source.md
yleiferman Jul 3, 2025
4a773f4
Updates from review
jpipkin1 Jul 3, 2025
4b767a1
Update universal-connector-source.md
yleiferman Jul 3, 2025
3ea4380
Merge branch 'yleiferman-patch-1' of github.com:SumoLogic/sumologic-d…
jpipkin1 Jul 3, 2025
826ce33
Merge branch 'yleiferman-patch-1' of github.com:SumoLogic/sumologic-d…
jpipkin1 Jul 3, 2025
ebd5f82
Fix broken links
jpipkin1 Jul 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions ...d-collectors/cloud-to-cloud-integration-framework/universal-connector-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Invalid examples:

#### Request Headers

Include any HTTP request headers required by the vendor API. The key names are static text, but the values can access our [template feature](#template-dynamic-values) to make them dynamic.
Include any HTTP request headers required by the vendor API. The key names are static text, but the values can access our [variables feature](#dynamic-values-variables) to make them dynamic.

| Example Header Key | Example Header Value |
| :----------------- | :--------------------------- |
Expand All @@ -135,7 +135,7 @@ Include any HTTP request headers required by the vendor API. The key names are s

#### Request Parameters

Include any URL query parameters required by the vendor API. The key names are static text, but the values can access our [template feature](#template-dynamic-values) to make them dynamic.
Include any URL query parameters required by the vendor API. The key names are static text, but the values can access our [variables feature](#dynamic-values-variables) to make them dynamic.

| Example Header Key | Example Header Value |
| :----------------- | :--------------------------------------------- |
Expand All @@ -148,7 +148,7 @@ Examples URL encoded:

#### Request Body

This is optional and only used if the HTTP `POST` method is configured above. You can use this field to include any information in the HTTP request body. The data included in this field can access our [template feature](#template-dynamic-values).
This is optional and only used if the HTTP `POST` method is configured above. You can use this field to include any information in the HTTP request body. The data included in this field can access our [dynamic values](#dynamic-values-variables).

</div>
</details>
Expand All @@ -159,9 +159,9 @@ This is optional and only used if the HTTP `POST` method is configured above. Yo

#### Time Window

The source will provide both a start and end timestamp for you to [dynamically](#template-dynamic-values) use in your HTTP request. The window will only move forward if no errors are raised when collecting logs from the vendor API for the current window.
The source will provide both a start and end timestamp for you to [dynamically](#dynamic-values-variables) use in your HTTP request. The window will only move forward if no errors are raised when collecting logs from the vendor API for the current window.

Use the [template feature](#template-dynamic-values) to include the window start and end timestamps within your HTTP request.
Use the [dynamic values](#dynamic-values-variables) to include the window start and end timestamps within your HTTP request.

The start time is inclusive and the end time is exclusive as that is the behavior of most APIs.

Expand All @@ -176,7 +176,7 @@ The start time is inclusive and the end time is exclusive as that is the behavio
<details>
<summary>HTTP Response Log Ingest Configuration</summary>
<div>
Select the format of the data returned by the vendor and configure how the source should break down the response into into individual logs with the correct timestamp.
Select the format of the data returned by the vendor and configure how the source should break down the response into individual logs with the correct timestamp.

#### JSON with JPath

Expand Down Expand Up @@ -330,7 +330,7 @@ Sources can be configured using UTF-8 encoded JSON files with the Collector Ma
| sourceType | String | `"Universal Connector"` | Yes | Type of source. |
| config | JSON Object | [Configuration object](#configuration-object) | Yes | Source type specific values. |

### Configuration Object
### Configuration object

| Parameter | Type | Required | Default | Description | Example |
| :------------------------- | :---------- | :------- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand Down Expand Up @@ -377,17 +377,17 @@ Sources can be configured using UTF-8 encoded JSON files with the Collector Ma
| clientRateLimitBurst | Integer | Yes | `1000` | The number of requests the source is allowed to burst. | `1000` |
| pollingInterval | String | Yes | `"5m"` | Set how frequently to poll for new data. It must be between 5 minutes and 48 hours. | `"5m"` |

## Template Dynamic Values
## Dynamic values variables

The source has the ability to template in dynamic text into the values of certain fields providing flexibility in crafting the HTTP requests sent to the vendor API.
The source has the ability to use dynamic values, like the window start time, in the values of certain fields providing flexibility in crafting the HTTP requests sent to the vendor API.

The following fields values are allowed to access dynamic text from the template functions described in this section:

- HTTP Request Header Values
- HTTP Request Parameter Values
- HTTP Request Body

To start using the template with one of the supported config values listed above, you will need to enclose the template logic inside double curly braces `{{}}`. Any text outside these double curly braces will be treated as normal unmodified text.
To start using the variables with one of the supported config values listed above, you will need to enclose the template logic inside double curly braces `{{}}`. Any text outside these double curly braces will be treated as normal unmodified text.

Here are some syntax examples calling functions with and without arguments:

Expand All @@ -397,7 +397,7 @@ Here are some syntax examples calling functions with and without arguments:
{{ .FunctionName "string argument 1" "string argument 2" }}
```

**Available Template Functions**
**Available variables**

- [WindowStartUTC](#windowstartutc)
- [WindowStartLocation](#windowstartlocation)
Expand All @@ -406,9 +406,9 @@ Here are some syntax examples calling functions with and without arguments:

### WindowStartUTC

This function will template in the `start timestamp` of the source window when source is configured to use the `Time Window` progression. The timestamp will always use `UTC` time and never adjust for a specific timezone.
This variable will get the value in the `start timestamp` of the source window when the source is configured to use the `Time Window` progression. The timestamp will always use `UTC` time and never adjust for a specific timezone.

The syntax for this function requires a timestamp format as a single argument. See the [Timestamp Formatting](#timestamp-formatting) section for more information on how to format the timestamp.
The syntax for this function requires a timestamp format as a single argument. See the [Timestamp formatting](#timestamp-formatting) section for more information on how to format the timestamp.

```sh
{{ .WindowStartUTC "<timestamp format>" }}
Expand All @@ -425,13 +425,13 @@ The syntax for this function requires a timestamp format as a single argument. S

### WindowStartLocation

This function is the same as [WindowStartUTC](#windowstartutc) except it has an additional argument to specify the timezone location.
This variable is the same as [WindowStartUTC](#windowstartutc) except it has an additional argument to specify the timezone location.

:::sumo[Best Practice]
We strongly recommend you always use `WindowStartUTC` instead of `WindowStartLocation`. Most vendors support and expect UTC timestamps when using their APIs.
:::

Refer to the [TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for specifying the time zone in the first argument and refer to the [Timestamp Formatting](#timestamp-formatting) section for more information on how to format the timestamp.
Refer to the [TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for specifying the time zone in the first argument and refer to the [Timestamp formatting](#timestamp-formatting) section for more information on how to format the timestamp.

```sh
{{ .WindowStartLocation "<time zone location>" "<timestamp format>" }}
Expand All @@ -448,9 +448,9 @@ Refer to the [TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_t

### WindowEndUTC

This function will template in the `end timestamp` of the source window when source is configured to use the `Time Window` progression. The timestamp will always use `UTC` time and never adjust for a specific timezone.
This variable will get the value in the `end timestamp` of the source window when the source is configured to use the `Time Window` progression. The timestamp will always use `UTC` time and never adjust for a specific timezone.

The syntax for this function requires a timestamp format as a single argument. Refer to the [Timestamp Formatting](#timestamp-formatting) section for more information on how to format the timestamp.
The syntax for this variable requires a timestamp format as a single argument. Refer to the [Timestamp formatting](#timestamp-formatting) section for more information on how to format the timestamp.

```sh
{{ .WindowEndUTC "<timestamp format>" }}
Expand All @@ -465,17 +465,17 @@ The syntax for this function requires a timestamp format as a single argument. R
| `{{ .WindowEndUTC "epoch" }}` | `1709842556` |
| `{{ .WindowEndUTC "epochMilli" }}` | `1709842556000` |
| `lessThan:{{ .WindowEndUTC "2006-01-02T15:04:05.999Z07:00" }}` | `lessThan:2024-03-07T20:15:56.905Z` |
| `lessThan:{{ .WindowEndUTC "yyyy-MM-ddTHH:mm:ss.SSSZ" }}` | `lessThan:2024-03-07T20:15:56.905Z` |
| `{"startTime":"{{ .WindowEndUTC "yyyy-MM-ddTHH:mm:ss.SSSZ" }}"}` | `{"startTime":"{{ .WindowEndUTC "2024-03-07T20:15:56.905Z" }}"` |

### WindowEndLocation

This function is the same as [WindowEndUTC](#windowendutc) except it has an additional argument to specify the timezone location.
This variable is the same as [WindowEndUTC](#windowendutc) except it has an additional argument to specify the timezone location.

:::sumo[Best Practice]
We strongly recommend you always use `WindowEndUTC` instead of `WindowEndLocation`. Most vendors support and expect UTC timestamps when using their APIs.
:::

Refer to the [TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for specifying the time zone in the first argument and refer to the [Timestamp Formatting](#timestamp-formatting) section for more information on how to format the timestamp.
Refer to the [TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for specifying the time zone in the first argument and refer to the [Timestamp formatting](#timestamp-formatting) section for more information on how to format the timestamp.

```sh
{{ .WindowEndLocation "<time zone location>" "<timestamp format>" }}
Expand All @@ -490,15 +490,15 @@ Refer to the [TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_t
| `lessThan:{{ .WindowEndLocation "Europe/Berlin" "2006-01-02T15:04:05.999Z07:00" }}` | `lessThan:2024-03-07T21:15:56.905+01:00` |
| `lessThan:{{ .WindowEndLocation "Europe/Berlin" "yyyy-MM-ddTHH:mm:ss.SSSZ" }}` | `lessThan:2024-03-07T21:15:56.905+01:00` |

## Timestamp Formatting
## Timestamp formatting

The source uses the [Go programming language timestamp formatting](https://go.dev/src/time/format.go) and the Human-readable timestamp formatting. See the table below for references and examples.

:::sumo[Best Practice]
We recommend using [this code snippet](https://goplay.tools/snippet/WTFe5ZLU9PO) as a quick way to locally test timestamp parsing with a format before configuring the source.
:::

### Format Reference
### Format reference

| Date Format | Reference Value | Human Readable Referencce Value |
| :------------------------------------------ | :-------------------------------------------------------------------- | :--------------------------------------------------------------------- |
Expand Down Expand Up @@ -527,7 +527,7 @@ We recommend using [this code snippet](https://goplay.tools/snippet/WTFe5ZLU9PO)
| Timezone Offset with Colon | `-07:00` | `-HH:mm` |
| Timezone Abbreviated Name | `MST` | `zzz` |

### Format Examples
### Format examples

| Standard | Timestamp in Log | Timestamp Format |
| :-------------------- | :------------------------------- | :------------------------------------ |
Expand Down