FEAT: Support for forwarding response headers from http task

[prasadlohakpure]

Description

Support for forwarding response headers from http task, and modification to jq task for non standard json record.
Why:

1. Response header: In certain scenarios, we want the capability to propagate headers down along with record data.
2. jq modification: In certain scenarios where data is non json, marshalling would return error. So if such failures pass stringified data
   (Note: Although we wont be anyway able to apply jq selector on it anyway, but this change gives capability to support selection of the complete response, and make the above JQ work.)

Use case:

1. Response header: response cookie from one response can be used in another http request to maintain session across multiple request.
2. jq modification: For jq tasks where we needed complete record selection for non json standard

Testing:

Done on test/pipelines/context_test.yaml file

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.

[Copilot]

Pull request overview

This PR adds support for forwarding HTTP response headers from the HTTP task to downstream tasks via the context mechanism. Headers are automatically stored with an http-header- prefix (e.g., http-header-Content-Type) and can be accessed using context variables like {{ context "http-header-Content-Type" }}.

Changes:

• Updated placeholder regex to support hyphens in context variable names
• HTTP task now stores all response headers in record context with http-header- prefix
• Added documentation and examples for accessing response headers
• Added integration test demonstrating header access in context_test.yaml

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
internal/pkg/config/config.go	Updated placeholderRegex from ([a-zA-Z0-9_]+?) to ([a-zA-Z0-9_-]+?) to support hyphens in context keys
internal/pkg/pipeline/task/http/http.go	Added logic to store all HTTP response headers in record context with http-header- prefix
internal/pkg/pipeline/task/http/README.md	Added documentation section explaining header forwarding feature with examples
test/pipelines/context_test.yaml	Added test cases accessing http-header-Content-Type and http-header-Age headers

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[prasadlohakpure]

For certain cases where record is non json (e.g. text data, html).
JQ like this was failing:

  - name: transform_with_context_jq
    type: jq
    path: |
      {
        "headers_content_type": "{{ context "http-header-Content-Type" }}",
        "headers_cookie": "{{ context "http-header-Set-Cookie" }}",
        "original_data": .
      }

Reason : Since data is non json, Marshalling would return error.
Although we wont be anyway able to apply jq selector on it anyway, but this change gives capability to supports selection of the complete response, and make the above JQ work.

[divyanshu-tiwari]

@prasadlohakpure Semi-colon is only suitable for joining cookie values; most other headers should use a comma. If we default to semi-colon, users will have to replace it for other headers. Maybe we should join with a semi-colon for cookies, and a comma for everything else. The goal is to keep values raw. Thoughts?

[prasadlohakpure]

I wasn't able to find any example where multi key-values are been sent by server for a same header(besides Cookie headers). So I am not sure if we will have a use case where we might get multiple mappings for non-cookie header
Also semi colon allows us to have logical separation in values for cases such as below:

e.g.:
WWW-Authenticate: Basic realm="simple", error="invalid_realm"
WWW-Authenticate: Bearer realm="users", error="invalid_token"

If we combine them using comma, so logical segregation between 2 different authentication_method: error_response would be lost and it would be difficult to recollect mapping later.
Semi-colon will help us maintain that.
Let me know what you think.