From 01e22d17e5c083eee129d4fc8e3f3898fd239d96 Mon Sep 17 00:00:00 2001 From: Dan Kortschak Date: Fri, 23 Feb 2024 10:21:30 +1030 Subject: [PATCH] x-pack/filebeat/input/httpjson: drop response bodies at end of execution The response bodies of the first and last responses were being held in a closed-over variable resulting in high static memory loads in some situations. The bodies are not used between periodic executions with the documentation stating that only cursor values are persisted across restarts. The difference in behaviour between using the body field over a restart versus over a sequence of executions in the same run make them unsafe, so clarify the persistence behaviour in the documentation and free the bodies at the end of an execution. A survey of integrations that use the httpjson input did not identify any that are using behaviour that is being removed, but we will need to keep an eye on cases that may have been missed. In general, if persistence is being depended on, the cursor should be being used. --- CHANGELOG.next.asciidoc | 1 + x-pack/filebeat/docs/inputs/input-httpjson.asciidoc | 2 +- x-pack/filebeat/input/httpjson/input.go | 6 ++++++ 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index d7761c359dca..c0f7d3c28ffb 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -96,6 +96,7 @@ fields added to events containing the Beats version. {pull}37553[37553] - Update github.com/lestrrat-go/jwx dependency. {pull}37799[37799] - [threatintel] MISP pagination fixes {pull}37898[37898] - Fix file handle leak when handling errors in filestream {pull}37973[37973] +- Prevent HTTPJSON holding response bodies between executions. {issue}35219[35219] {pull}[] *Heartbeat* diff --git a/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc b/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc index 5fbd5dc15a5f..bf2b9f195cc2 100644 --- a/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc +++ b/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc @@ -119,7 +119,7 @@ The state has the following elements: - `body`: A map containing the body. References the next request body when used in <> or <> configuration sections, and to the last response body when used in <> or <> configuration sections. - `cursor`: A map containing any data the user configured to be stored between restarts (See <>). -All of the mentioned objects are only stored at runtime, except `cursor`, which has values that are persisted between restarts. +All of the mentioned objects are only stored at runtime during the execution of the periodic request, except `cursor`, which has values that are persisted between periodic request and restarts. [[transforms]] ==== Transforms diff --git a/x-pack/filebeat/input/httpjson/input.go b/x-pack/filebeat/input/httpjson/input.go index 50a4f7f20a61..6757883a8a12 100644 --- a/x-pack/filebeat/input/httpjson/input.go +++ b/x-pack/filebeat/input/httpjson/input.go @@ -163,6 +163,12 @@ func run(ctx v2.Context, cfg config, pub inputcursor.Publisher, crsr *inputcurso trCtx.cursor.load(crsr) doFunc := func() error { + defer func() { + // Clear response bodies between evaluations. + trCtx.firstResponse.body = nil + trCtx.lastResponse.body = nil + }() + log.Info("Process another repeated request.") startTime := time.Now()