Skip to content

Implement the measure operator #3093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 3, 2023
Merged

Implement the measure operator #3093

merged 4 commits into from
May 3, 2023

Conversation

dominiklohmann
Copy link
Member

@dominiklohmann dominiklohmann commented Apr 21, 2023

The measure [--real-time] operator emits metrics about the input into the pipeline. The operator handles both events and bytes as input, and emits the metrics as events as described by these schemas, respectively:

type vast.metrics.events = record  {
  timestamp: time,
  schema: string,
  schema_id: string,
  events: uint64,
}

type vast.metrics.bytes = record  {
  timestamp: time,
  bytes: uint64,
}

The --real-time option causes the operator to instantly emit the metrics at the cost of throughput.

### Tasks
- [x] Implement the operator
- [x] Agree on schema names for the metrics events
- [x] Write tests
- [x] Document the operator
- [x] Add a changelog entry

@dominiklohmann dominiklohmann added the feature New functionality label Apr 21, 2023
@dominiklohmann dominiklohmann force-pushed the topic/inspect-operator branch 5 times, most recently from 2a87092 to c6d6481 Compare April 28, 2023 14:46
@dominiklohmann dominiklohmann marked this pull request as ready for review April 28, 2023 14:46
Copy link
Contributor

@jachris jachris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One general comment: You could consider renaming this operator, because inspect is not the most intuitive name for its behavior. Perhaps even something like metrics (although we'd normally avoid nouns). But you can also keep it as-is.

@dominiklohmann
Copy link
Member Author

@jachris: You could consider renaming this operator, because inspect is not the most intuitive name for its behavior. Perhaps even something like metrics (although we'd normally avoid nouns). But you can also keep it as-is.

@mavam do you have an idea for a better name? I agree generally that inspect is not the ideal name, but I don't know what we should use. Potential candidates that are verbs:

  1. inspect
  2. measure
  3. debug
  4. monitor
  5. view (similar to the UNIX tool pv)
  6. metric (which according to Wiktionary can also be used as a verb)

@jachris
Copy link
Contributor

jachris commented Apr 28, 2023

TBH, although I absolutely prefer verbs as well, we should not put this requirement above everything else. If there is no adequate verb, we should just use a noun. And we already do this with from and to (although you might argue that sources and sinks are special). Also, we might want to use inspect for something else that is more inspect-y.

@mavam
Copy link
Member

mavam commented Apr 28, 2023

I agree that inspect is probably more well-suited to yield the AST of the pipeline or some other structural information. Ideas for verbs, in order of preference:

  1. profile
  2. measure
  3. observe
  4. monitor

@dominiklohmann
Copy link
Member Author

Of the suggested alternatives I chose to go with measure for now; there was no objection internally.

@dominiklohmann dominiklohmann changed the title Implement the inspect operator Implement the measure operator May 3, 2023
The `inspect [--real-time]` operator emits metrics about the input into
the pipeline. The operator handles both events and bytes as input, and
emits the metrics as events as described by these schemas, respectively:

```
type vast.inspect.events = record  {
  timestamp: time,
  schema: string,
  events: uint64,
}

type vast.inspect.bytes = record  {
  timestamp: time,
  bytes: uint64,
}
```

The `--real-time` option causes the operator to instantly emit the
metrics at the cost of throughput.
@dominiklohmann dominiklohmann force-pushed the topic/inspect-operator branch from 5d8b98a to 9cdf261 Compare May 3, 2023 07:56
@dominiklohmann dominiklohmann enabled auto-merge May 3, 2023 07:56
@dominiklohmann dominiklohmann merged commit 529d9e7 into main May 3, 2023
@dominiklohmann dominiklohmann deleted the topic/inspect-operator branch May 3, 2023 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants