From 8dc5cc0ea5a90f14f3fe00e27b024a62ac5c5077 Mon Sep 17 00:00:00 2001 From: kosabogi Date: Fri, 21 Mar 2025 09:26:44 +0100 Subject: [PATCH 1/3] Adds delay parameter explanation --- explore-analyze/transforms/transform-checkpoints.md | 2 ++ explore-analyze/transforms/transform-usage.md | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/explore-analyze/transforms/transform-checkpoints.md b/explore-analyze/transforms/transform-checkpoints.md index c6f9e196da..0b7bb1722d 100644 --- a/explore-analyze/transforms/transform-checkpoints.md +++ b/explore-analyze/transforms/transform-checkpoints.md @@ -19,6 +19,8 @@ To create a checkpoint, the {{ctransform}}: Using a simple periodic timer, the {{transform}} checks for changes to the source indices. This check is done based on the interval defined in the transform’s `frequency` property. + If new data is ingested with a slight delay, it might not be immediately available when the transform runs. To prevent missing documents, you can use the `delay` parameter in the `sync` configuration. This shifts the search window backward, ensuring that late-arriving data is included before a checkpoint processes it. Adjusting this value based on your data ingestion patterns can help ensure completeness. + If the source indices remain unchanged or if a checkpoint is already in progress then it waits for the next timer. If changes are found a checkpoint is created. diff --git a/explore-analyze/transforms/transform-usage.md b/explore-analyze/transforms/transform-usage.md index cb94d5deef..a236ead282 100644 --- a/explore-analyze/transforms/transform-usage.md +++ b/explore-analyze/transforms/transform-usage.md @@ -27,3 +27,9 @@ You might want to consider using {{transforms}} instead of aggregations when: * You want to create summary tables to optimize queries. For example, if you have a high level dashboard that is accessed by a large number of users and it uses a complex aggregation over a large dataset, it may be more efficient to create a {{transform}} to cache results. Thus, each user doesn’t need to run the aggregation query. + +* You need to account for late-arriving data. + + In some cases, data might not be immediately available when a transform runs, leading to missing records in the destination index. This can happen due to ingestion delays, where documents take a few seconds or minutes to become searchable after being indexed. To handle this, the `delay` parameter in the transform’s sync configuration allows you to postpone processing new data. Instead of always querying the most recent records, the transform will skip a short period of time (e.g., 60 seconds) to ensure all relevant data has arrived before processing. + + For example, if a transform runs every 5 minutes, it usually processes data from 5 minutes ago up to the current time. However, if you set `delay` to 60 seconds, the transform will instead process data from 6 minutes ago up to 1 minute ago, making sure that any documents that arrived late are included. By adjusting the `delay` parameter, you can improve the accuracy of transformed data while still maintaining near real-time results. From 5da83e15a91c1398a4f91ea892f03ccf7d393102 Mon Sep 17 00:00:00 2001 From: kosabogi Date: Mon, 24 Mar 2025 11:50:54 +0100 Subject: [PATCH 2/3] Fixes attributes --- explore-analyze/transforms/transform-checkpoints.md | 2 +- explore-analyze/transforms/transform-usage.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/explore-analyze/transforms/transform-checkpoints.md b/explore-analyze/transforms/transform-checkpoints.md index 0b7bb1722d..479f28b63d 100644 --- a/explore-analyze/transforms/transform-checkpoints.md +++ b/explore-analyze/transforms/transform-checkpoints.md @@ -19,7 +19,7 @@ To create a checkpoint, the {{ctransform}}: Using a simple periodic timer, the {{transform}} checks for changes to the source indices. This check is done based on the interval defined in the transform’s `frequency` property. - If new data is ingested with a slight delay, it might not be immediately available when the transform runs. To prevent missing documents, you can use the `delay` parameter in the `sync` configuration. This shifts the search window backward, ensuring that late-arriving data is included before a checkpoint processes it. Adjusting this value based on your data ingestion patterns can help ensure completeness. + If new data is ingested with a slight delay, it might not be immediately available when the {{transform}} runs. To prevent missing documents, you can use the `delay` parameter in the `sync` configuration. This shifts the search window backward, ensuring that late-arriving data is included before a checkpoint processes it. Adjusting this value based on your data ingestion patterns can help ensure completeness. If the source indices remain unchanged or if a checkpoint is already in progress then it waits for the next timer. diff --git a/explore-analyze/transforms/transform-usage.md b/explore-analyze/transforms/transform-usage.md index a236ead282..afb38ab0cf 100644 --- a/explore-analyze/transforms/transform-usage.md +++ b/explore-analyze/transforms/transform-usage.md @@ -30,6 +30,6 @@ You might want to consider using {{transforms}} instead of aggregations when: * You need to account for late-arriving data. - In some cases, data might not be immediately available when a transform runs, leading to missing records in the destination index. This can happen due to ingestion delays, where documents take a few seconds or minutes to become searchable after being indexed. To handle this, the `delay` parameter in the transform’s sync configuration allows you to postpone processing new data. Instead of always querying the most recent records, the transform will skip a short period of time (e.g., 60 seconds) to ensure all relevant data has arrived before processing. + In some cases, data might not be immediately available when a {{transform}} runs, leading to missing records in the destination index. This can happen due to ingestion delays, where documents take a few seconds or minutes to become searchable after being indexed. To handle this, the `delay` parameter in the {{transform}}’s sync configuration allows you to postpone processing new data. Instead of always querying the most recent records, the {{transform}} will skip a short period of time (e.g., 60 seconds) to ensure all relevant data has arrived before processing. - For example, if a transform runs every 5 minutes, it usually processes data from 5 minutes ago up to the current time. However, if you set `delay` to 60 seconds, the transform will instead process data from 6 minutes ago up to 1 minute ago, making sure that any documents that arrived late are included. By adjusting the `delay` parameter, you can improve the accuracy of transformed data while still maintaining near real-time results. + For example, if a {{transform}} runs every 5 minutes, it usually processes data from 5 minutes ago up to the current time. However, if you set `delay` to 60 seconds, the {{transform}} will instead process data from 6 minutes ago up to 1 minute ago, making sure that any documents that arrived late are included. By adjusting the `delay` parameter, you can improve the accuracy of transformed data while still maintaining near real-time results. From f63ec71bdbd3cd018d8083aa0b549decadc984f2 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Mon, 24 Mar 2025 13:49:28 +0100 Subject: [PATCH 3/3] Update explore-analyze/transforms/transform-usage.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: István Zoltán Szabó --- explore-analyze/transforms/transform-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explore-analyze/transforms/transform-usage.md b/explore-analyze/transforms/transform-usage.md index afb38ab0cf..f7d1730648 100644 --- a/explore-analyze/transforms/transform-usage.md +++ b/explore-analyze/transforms/transform-usage.md @@ -30,6 +30,6 @@ You might want to consider using {{transforms}} instead of aggregations when: * You need to account for late-arriving data. - In some cases, data might not be immediately available when a {{transform}} runs, leading to missing records in the destination index. This can happen due to ingestion delays, where documents take a few seconds or minutes to become searchable after being indexed. To handle this, the `delay` parameter in the {{transform}}’s sync configuration allows you to postpone processing new data. Instead of always querying the most recent records, the {{transform}} will skip a short period of time (e.g., 60 seconds) to ensure all relevant data has arrived before processing. + In some cases, data might not be immediately available when a {{transform}} runs, leading to missing records in the destination index. This can happen due to ingestion delays, where documents take a few seconds or minutes to become searchable after being indexed. To handle this, the `delay` parameter in the {{transform}}’s sync configuration allows you to postpone processing new data. Instead of always querying the most recent records, the {{transform}} will skip a short period of time (for example, 60 seconds) to ensure all relevant data has arrived before processing. For example, if a {{transform}} runs every 5 minutes, it usually processes data from 5 minutes ago up to the current time. However, if you set `delay` to 60 seconds, the {{transform}} will instead process data from 6 minutes ago up to 1 minute ago, making sure that any documents that arrived late are included. By adjusting the `delay` parameter, you can improve the accuracy of transformed data while still maintaining near real-time results.