Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New `http` source #328

Open
binarylogic opened this issue May 7, 2019 · 3 comments
Open

New `http` source #328

binarylogic opened this issue May 7, 2019 · 3 comments

Comments

@binarylogic
Copy link
Member

@binarylogic binarylogic commented May 7, 2019

An HTTP server as a source

  • Configurable port
  • HTTPs support?
  • JSON, NDJSON, msgpack, and new line delimited decoding support

Anything else I'm missing? This is just a preliminary issue, we'll get it spec'd out further before we begin work.

@binarylogic binarylogic changed the title HTTP source New `http` source Jun 20, 2019
@dneray

This comment has been minimized.

Copy link

@dneray dneray commented Dec 15, 2019

I am also very interested in this feature, for an initial implementation I would be happy with:

  • plain http support (I can run a reverse proxy alongside to handle tls)
  • configurable IP + port to bind to
  • json
  • access to any http headers used to make the request

I think this would be enough to provide an avenue for integration with fluentbit/fluentd and logstash. I do agree that ultimately there should be support for all the listed formats, https and possibly basic auth?

@leshow

This comment has been minimized.

Copy link
Contributor

@leshow leshow commented Jan 18, 2020

I wrote some of the very beginnings of this, spawning a server with hyper & a configurable ip/port, and forwarding received messages to the Sender half of the channel provided to sources. I could submit a WIP PR and get some feedback on it? I realize you haven't fully gathered requirements but the implementation is so minimal right now I thought I could get the ball rolling.

@gedkins

This comment has been minimized.

Copy link

@gedkins gedkins commented Jan 31, 2020

My recommendation would be to start from the logplex source and refactor it to make more and more things configurable (e.g. body format parsing, path, etc). Then it would be really nice if the specific sinks like logplex could be small wrappers around the new generic http source.

I've submitted a pull request that addresses the first part of this. I've got separate changes on the way for https support and for refactoring the logplex sink to use the shared code.

Note there may be an issue with this implementation, which we haven't looked into yet, regarding telling vector to reload the configuration - I tried it once and saw it complain about the port conflicting with itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.