New exec sink

[mamcx]

I like to use vector to process logs to http://timescale.com. But because it not have that as sink, I was looking into implementing a http collector for that.

However, considering that vector can be used to aggregate multiple vector agents, how about a "universal" sink? So I could do:

[sinks.out_vector]
	type = "cmd"
	inputs = ["in"]
	cmd = echo {msg} // or psql, or whatever...

This way, it could cover much more cases where the sink is not available and build a small cli utility is the more easy way to get running...

[binarylogic]

@lukesteensen do you mind reviewing this and determining if this is valid or something we should do? I don't have enough context to say. Thanks.

[lukesteensen]

Thanks for this idea @mamcx! It's definitely an interesting one.

My main hesitance is around the management of the long-running subprocess. There are a lot of cases to handle there and we'd probably be better off leaving that to a dedicated process manager like systemd or supervisord.

As an alternative, you could use the file source and point it at a named pipe instead of a normal file. You could then run your small cli utility under a normal process manager and simply connect that named pipe to its standard input.

We haven't specifically tested the file source against named pipes, but that's something we could definitely tackle. Does that seem like it would cover the use case you have in mind? I understand it's probably not quite as convenient, but hopefully it would be a bit more robust than building a process manager into vector itself.

Let me know what you think. Thanks again!

[mamcx]

I was not thinking about long running commands, just one-shoot (alike cgi). Probably more robust? I think that a long running command is more probably a web server or similar or an interactive one.

[lukesteensen]

Gotcha, yes I think that would be simpler and something we could consider. It's unlikely to be a high priority for our own development efforts, but I'd be happy to review a PR that added such a feature.

At a high level, the design would roughly be to use our prebuilt batching components combined with a Service implementation that spawned tokio-managed Commands with the incoming request wired up as the spawned process's stdin. We'd want to make sure to time out the command, retry on non-successful exit codes, etc.

[yurtaev]

Temporary workaround for me:

[transforms.sentry]
  type = "lua"
  inputs = ["some-source"]
  source = """
  command_template = 'SENTRY_DSN=%s sentry-cli send-event --no-environ -m "%s"'

  SENTRY_DSN = "https://*****@sentry.io/123"
  message = event["message']

  command = string.format(command_template, SENTRY_DSN, message)

  os.execute(command)
  """

[jszwedko]

Just noting this would be similar to fluentd's exec output: https://docs.fluentd.org/output/exec