x-pack/filebeat/input/httpjson: allow elision of set and append failure logging

[efd6]

Proposed commit message

See title.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

In the httpjson package run the following commands. The first should show a set of messages at DEBUG level and none at ERROR level. The second should show no output.

go test -run 'TestInput/.*_not_log_fail' -v | grep ignored

go test -run 'TestInput/.*[^_][^n][^o][^t][^_][^l][^o][^g][^_][^f][^a][^i][^l]$' -v | grep ignored # ¯\_(ツ)_/¯

Related issues

• Closes [httpjson] Improve handling of end-of-pagination #34544
• Closes Elastic Agent shouldn't log httpjson pagination completion as an Agent Error #38291

Use cases

Screenshots

Logs

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[chrisberkhout]

This seems like a fine improvement for the noisy logs problem.

At some point it would be good to stop requiring failure to terminate pagination. Perhaps there could be a pagination_until option:

response.pagination_until: '[[not (index .last_response.body.meta "cursor")]]'
response.pagination:
- set:
    target: url.params.cursor
    value: '[[.last_response.body.meta.cursor]]'

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 34544-httpjson upstream/34544-httpjson
git merge upstream/main
git push upstream 34544-httpjson

[chrisberkhout]

👍