Skip to content

[azure logs] Add processor version options to Raw Events integration (v1) #15900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zmoog merged 6 commits into from
Nov 19, 2025
Merged

[azure logs] Add processor version options to Raw Events integration (v1) #15900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a5c04e8
Add processor version options
zmoog Nov 7, 2025
12ae1ea
Update packages/azure/data_stream/eventhub/manifest.yml
zmoog Nov 13, 2025
efb01d3
Update changelog
zmoog Nov 13, 2025
b494949
Switch from `type: text` to `type: select`
zmoog Nov 17, 2025
ce8984e
Switch from type:text to type:select
zmoog Nov 18, 2025
a243fc2
Update changelog
zmoog Nov 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/azure/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "1.30.0"
changes:
- description: Add processor version options for the Raw Events (v1) integration.
type: enhancement
link: https://github.com/elastic/integrations/pull/15900
- version: "1.29.1"
changes:
- description: Update `time` formats parsing and fix `durationMs` parsing in signinlogs data stream.
Expand Down
21 changes: 21 additions & 0 deletions packages/azure/data_stream/eventhub/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,31 @@ storage_account: {{storage_account}}
{{/if}}
{{#if storage_account_key}}
storage_account_key: {{storage_account_key}}
storage_account_connection_string: DefaultEndpointsProtocol=https;AccountName={{storage_account}};AccountKey={{storage_account_key}};EndpointSuffix={{endpoint_suffix}}
{{/if}}
{{#if resource_manager_endpoint}}
resource_manager_endpoint: {{resource_manager_endpoint}}
{{/if}}

{{#if processor_version}}
processor_version: {{processor_version}}
{{/if}}
{{#if migrate_checkpoint}}
migrate_checkpoint: {{migrate_checkpoint}}
{{/if}}
{{#if processor_update_interval}}
processor_update_interval: {{processor_update_interval}}
{{/if}}
{{#if processor_start_position}}
Copy link
Contributor

@gizas gizas Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add sth like if processor_version}==v2 then processor_start_position?

Same also for below fields that are only realted to v2

Copy link
Contributor Author

@zmoog zmoog Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also added a couple of checklist items for this in the next issue #15703, so I can do the same across all the integrations using this input.

Copy link
Contributor Author

@zmoog zmoog Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the built-in helper #if only supports checking if a value is empty to skip rendering a block:

From https://handlebarsjs.com/guide/builtin-helpers.html#if:

You can use the if helper to conditionally render a block. If its argument returns false, undefined, null, "", 0, or [], Handlebars will not render the block.

This problem should auto-resolve soon as we are switching to processor v2 as default and dropping v1 after that.

processor_start_position: {{processor_start_position}}
{{/if}}
{{#if partition_receive_timeout}}
partition_receive_timeout: {{partition_receive_timeout}}
{{/if}}
{{#if partition_receive_count}}
partition_receive_count: {{partition_receive_count}}
{{/if}}

data_stream:
dataset: {{data_stream.dataset}}
tags:
Expand Down
98 changes: 98 additions & 0 deletions packages/azure/data_stream/eventhub/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,104 @@ streams:
type: bool
multi: false
default: false

#
# Processor v2 only settings
#
- name: processor_version
type: select
title: Processor version
multi: false
required: false
show_user: false
default: v1
options:
- text: v1
value: v1
- text: v2
value: v2
description: "The processor version that the integration should use. Possible values are `v1` and `v2` (preview). \nThe v2 event hub processor is in preview, so using the v1 processor is recommended for typical use cases.\nDefault is `v1`."
- name: processor_update_interval
type: text
title: Processor update interval
multi: false
required: false
show_user: false
default: 10s
description: >-
(Processor v2 only) How often the processor should attempt to claim partitions.

Default is `10` seconds.
- name: processor_start_position
type: select
title: Processor start position
multi: false
required: false
show_user: false
default: earliest
options:
- text: earliest
value: earliest
- text: latest
value: latest
description: >-
(Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.

Possible values are `earliest` and `latest`.

`earliest` starts processing messages from the last checkpoint, or the beginning of the event hub if no checkpoint is available.

`latest` starts processing messages from the the latest event in the event hub and continues to process new events as they arrive.

Default is `earliest`.
- name: partition_receive_timeout
type: text
title: Partition receive timeout
multi: false
required: false
show_user: false
default: 5s
description: >-
(Processor v2 only) Maximum time to wait before processing the messages received from the event hub.

The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first.

Default is `5` seconds.
- name: partition_receive_count
type: text
title: Partition receive count
multi: false
required: false
show_user: false
default: 100
description: >-
(Processor v2 only) Maximum number of messages from the event hub to wait for before processing them.

The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first.

Default is `100` messages.
- name: migrate_checkpoint
type: bool
title: Migrate checkpoint information
multi: false
required: false
show_user: false
default: true
description: >-
(Processor v2 only) Flag to control if the processor should perform the checkpoint information migration from processor v1 to v2 at startup.

The checkpoint migration converts the checkpoint information from the v1 format to the v2 format.

Default is `false`, which means the processor will not perform the checkpoint migration.
- name: endpoint_suffix
type: text
default: core.windows.net
required: true
title: Storage account endpoint suffix
show_user: false
description: >-
(Processor v2 only) Override the default storage account endpoint suffix.

# Ensures agents have permissions to write data to `logs-*-*`
elasticsearch:
dynamic_dataset: true
Expand Down
14 changes: 12 additions & 2 deletions packages/azure/data_stream/events/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,17 @@ streams:
multi: false
default: false
- name: processor_version
type: text
type: select
title: Processor version
multi: false
required: false
show_user: false
default: v1
options:
- text: v1
value: v1
- text: v2
value: v2
description: "The processor version that the integration should use. Possible values are `v1` and `v2` (preview). \nThe v2 event hub processor is in preview, so using the v1 processor is recommended for typical use cases.\nDefault is `v1`."
- name: processor_update_interval
type: text
Expand All @@ -100,12 +105,17 @@ streams:

Default is `10` seconds.
- name: processor_start_position
type: text
type: select
title: Processor start position
multi: false
required: false
show_user: false
default: earliest
options:
- text: earliest
value: earliest
- text: latest
value: latest
description: >-
(Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.

Expand Down
2 changes: 1 addition & 1 deletion packages/azure/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: azure
title: Azure Logs
version: "1.29.1"
version: "1.30.0"
description: This Elastic integration collects logs from Azure
type: integration
icons:
Expand Down
5 changes: 5 additions & 0 deletions packages/azure_logs/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.4.2"
changes:
- description: Switch `processor_version` and `processor_start_position` configuration options to `select` type (UI change).
type: enhancement
link: https://github.com/elastic/integrations/pull/15900
- version: "0.4.1"
changes:
- description: Add support to override `Endpoint Suffix` based on deployment environment.
Expand Down
16 changes: 13 additions & 3 deletions packages/azure_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.3.0
name: azure_logs
title: "Custom Azure Logs"
version: "0.4.1"
version: "0.4.2"
source:
license: Elastic-2.0
description: "Collect log events from Azure Event Hubs with Elastic Agent"
Expand Down Expand Up @@ -138,12 +138,17 @@ policy_templates:
multi: false
default: false
- name: processor_version
type: text
type: select
title: Processor version
multi: false
required: false
show_user: false
default: v1
options:
- text: v1
value: v1
- text: v2
value: v2
description: "The processor version that the integration should use. Possible values are `v1` and `v2` (preview). \nThe v2 event hub processor is in preview, so using the v1 processor is recommended for typical use cases.\nDefault is `v1`."
- name: processor_update_interval
type: text
Expand All @@ -157,12 +162,17 @@ policy_templates:

Default is `10` seconds.
- name: processor_start_position
type: text
type: select
title: Processor start position
multi: false
required: false
show_user: false
default: earliest
options:
- text: earliest
value: earliest
- text: latest
value: latest
description: >-
(Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.

Expand Down