Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial metrics support #64

Closed
3 tasks
lukesteensen opened this issue Jan 4, 2019 · 1 comment · Fixed by #374
Closed
3 tasks

Initial metrics support #64

lukesteensen opened this issue Jan 4, 2019 · 1 comment · Fixed by #374
Assignees
Labels
domain: buffers Anything related to Vector's memory/disk buffers domain: data model Anything related to Vector's internal data model domain: metrics Anything related to Vector's metrics events domain: topology Anything related to Vector's topology code

Comments

@lukesteensen
Copy link
Member

We'd like the router to support both transforming logs into metrics and collecting metrics directly. For the first pass at this, we'll probably need a few basic components:

  • The ability to represent metrics in our internal data format. Both cernan and SSF could be good inspiration here.
  • A transform that can aggregate its input and emit output on a scheduled interval
  • A simple way to send that output somewhere we can see it. This could be as simple as treating it as log lines we can use with existing sinks, or just logging it out.

Some things to think about but not necessarily solve right away:

  • How "typed" do we want our internal data format? There's a whole spectrum between a simple map of attributes and full fledged "log line" / "metric" types with that apply to transforms/sources/sinks
  • How do will we support "bucketing" of aggregated metrics? Purely based off wall clock time of their arrival, or will there be a concept of event time vs processing time? How would late data be handled in that case?
@michaelfairley michaelfairley self-assigned this Jan 30, 2019
@michaelfairley michaelfairley removed their assignment Mar 26, 2019
@binarylogic binarylogic added this to the 0.2 milestone Mar 28, 2019
@binarylogic binarylogic added Core: Data Plane domain: topology Anything related to Vector's topology code labels Mar 28, 2019
@binarylogic
Copy link
Contributor

Noting, the goal for this first version is something simple that works. More of an experimental exercise than a full complete version. I'd recommend we start accepting metrics from a specific type (ex: statsd) and then simply forwarding that data to the same system. That will at least ensure we support the metrics data type in a fundamental way.

@lukesteensen lukesteensen self-assigned this Apr 16, 2019
@binarylogic binarylogic changed the title Metrics support Initial metrics support Apr 17, 2019
@lukesteensen lukesteensen mentioned this issue May 16, 2019
@binarylogic binarylogic added Core: Metric domain: buffers Anything related to Vector's memory/disk buffers labels Jun 19, 2019
@binarylogic binarylogic added domain: data model Anything related to Vector's internal data model domain: metrics Anything related to Vector's metrics events and removed event type: metric labels Aug 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: buffers Anything related to Vector's memory/disk buffers domain: data model Anything related to Vector's internal data model domain: metrics Anything related to Vector's metrics events domain: topology Anything related to Vector's topology code
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants