New http-request agent

[nicoloboschi]

• Added new http-request agent
• It accepts
  • url: http(s) url to call without query-string
  • query-string (map): accepts string as template. encoding is done automatically
  • method: "get" by default
  • headers (map): accepts string as template e.g. X-Custom-Header: {{{ value.id }}}
  • body: body to send, must be a string: accepts string as template e.g. X-Custom-Header: {{{ value.id }}}
  • output-field: field where to output in the record. in case the response return content-type: application/json it will be deserialized as map. otherwise will be threated as string
  • allow-redirects: set to false to never redirect
  • handle-cookies: set to false to not manage cookies during redirects

Example with authentication

                                  - name: "http-request"
                                    type: "http-request"
                                    input: input-topic
                                    output: output-topic
                                    id: step1
                                    configuration:
                                        output-field: value.api
                                        url: "https://somewhere/api/test/{{{ value.id }}}"
                                        method: POST
                                        body: '{"id": "{{{ value.id }}}"}'
                                        headers:
                                          Content-Type: application/json
                                          Authorization: Bearer {{{ secrets.s1.token }}}

[eolivelli]

I would make the query string parameters configurable, otherwise you have to provide a way to perform escaping.

Also I would make the url a constant in the first iteration, this way we prevent malicious usage, like calling URLs forged by a chatbot user

[eolivelli]

LGTM

[cbornet]

It can be interesting to look at what features LangChain loaders provide when we create these Sources.