Conduit Connector for HTTP

The HTTP connector is a Conduit plugin. It provides both, a source and a destination HTTP connectors.

How to build?

Run make build to build the connector's binary.

Testing

Run make test to run all the unit tests.

Source

The HTTP source connector pulls data from the HTTP URL every pollingPeriod, the source adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration. The returned data is used to create an openCDC record and return it.

Note: when using the OPTIONS method, the resulted options will be added to the record's metadata.

Configuration

name	description	required	default value	example
url	HTTP URL to send requests to.	true		https://example.com/api/v1
method	HTTP method to use in the request, supported methods are (GET,HEAD,OPTIONS).	false	GET	POST
headers	HTTP headers to use in the request, comma separated list of : separated pairs.	false		Authorization:Bearer TOKEN_VALUE,Content-Type:application/xml
params.*	parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1".	false		params.query="foobar"
pollingperiod	how often the connector will get data from the url, formatted as a time.Duration.	false	"5m"	"5m"
script.parseResponse	The path to a .js file containing the code to parse the response. The signature of the function needs to be: function parseResponse(bytes) where bytes is the original response's raw bytes (i.e. unparsed). The function needs to return a Response object.	false		/path/to/get_request_data.js An example script can be found in test/get_request_data.js
script.getRequestData	The path to a .js file containing the code to prepare the request data. The signature of the function needs to be: function getRequestData(cfg, previousResponse, position) where: cfg (a map) is the connector configuration previousResponse (a map) contains data from the previous response (if any), returned by parseResponse position (a byte array) contains the starting position of the connector. The function needs to return a Request object.	false		/path/to/parse_response.js An example script can be found in test/parse_response.js

Destination

The HTTP destination connector pushes data from upstream resources to an HTTP URL via Conduit. the destination adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration.

Note: The request Body that will be sent is the value under record.Payload.After, if you want to change the format of that or manipulate the field in any way, please check our Builtin Processors Docs , or check Standalone Processors Docs if you'd like to build your own processor .

Configuration

name	description	required	default value
url	Is a Go template expression for the URL used in the HTTP request, using Go templates. The value provided to the template is sdk.Record, so the template has access to all its fields (e.g. .Position, .Key, .Metadata, and so on). We also inject all template functions provided by sprig to make it easier to write templates.	true
method	Http method to use in the request, supported methods are (POST,PUT,DELETE,PATCH).	false	POST
headers	Http headers to use in the request, comma separated list of : separated pairs.	false
params.*	parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1".	false