Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: RFC for Data Volume Insights #17322

Merged
merged 13 commits into from
May 26, 2023
Merged

Conversation

StephenWakely
Copy link
Contributor

@StephenWakely StephenWakely commented May 5, 2023

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
@netlify
Copy link

netlify bot commented May 5, 2023

Deploy Preview for vrl-playground canceled.

Name Link
🔨 Latest commit 71058bb
🔍 Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/6470d635b770e9000863c8e9

@netlify
Copy link

netlify bot commented May 5, 2023

Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit 71058bb
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/6470d635f3079d0008b9fa1f

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
@jszwedko jszwedko requested a review from bruceg May 8, 2023 17:52
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start! I left some comments below.

rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
Comment on lines +103 to +104
The additional tags being added to the metrics will increase the cardinality of
those metrics if they are enabled.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also curious if this breaks some of the event registration work we did with the need to add this new dynamic tag. cc/ @bruceg (who I also tagged as reviewer)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it will.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It absolutely will, ouch. Given that registered events have become a correctness issue due to expiring metrics, I think we will have to resolve that breakage. I see only one way out, and that is to introduce a registered event caching layer that is keyed on the source/service identifiers. Obviously, that adds more complexity and interacts negatively with expiring metrics.

Comment on lines 134 to 139
- [ ] Submit a PR with spike-level code _roughly_ demonstrating the change.
- [ ] Incremental change #1
- [ ] Incremental change #2
- [ ] ...

Note: This can be filled out during the review process.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note to fill this out.

rfcs/2023-05-03-data-volume-metrics.md Show resolved Hide resolved
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed
rfcs/2023-05-03-data-volume-metrics.md Fixed Show fixed Hide fixed

#### Metric tags

**service** - to attach the service, we need to add a new meaning to Vector - *service*. Any sources that
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about sources that don't have a specific field containing a service name, but it still associated with a source (such as a tcp source where the entire event is just a string). Using a template here might be reasonable (or even allowing VRL)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also wondering how this intersects with semantic meanings. It seems like "service" is a new semantic meaning so that users could set this in VRL if they wanted? Is that part of the plan?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a bit here to indicate VRL can be used for this in time.

rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Show resolved Hide resolved
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few last comments, otherwise looks good!

rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved

#### Metric tags

**service** - to attach the service, we need to add a new meaning to Vector -
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we include an example showing that users can modify this meaning (e.g. set it to point to a different field than service)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is currently possible. @fuchsnj can confirm, but I believe the plan will be to add some VRL functions to do this in time?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe they already exist? The set_semantic_meaning function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes. Just undocumented.

Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few last comments, otherwise looks good!

Copy link
Member

@sghall sghall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @StephenWakely 🚀

rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
Comment on lines +75 to +78
We will need to do an audit of all components to ensure the
bytes emitted for the `component_received_event_bytes_total` and
`component_sent_event_bytes_total` metrics are the estimated JSON size of the
event.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this, is the other internal "byte size" useful for anything? If not, this "audit" could be reduced to just eliminating trait ByteSizeOf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is still (incorrectly) used for building batches (#10020). It could be useful for #8679 but we could also wait until we implement that to re-add it if needed.

rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
rfcs/2023-05-03-data-volume-metrics.md Outdated Show resolved Hide resolved
Comment on lines +103 to +104
The additional tags being added to the metrics will increase the cardinality of
those metrics if they are enabled.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It absolutely will, ouch. Given that registered events have become a correctness issue due to expiring metrics, I think we will have to resolve that breakage. I see only one way out, and that is to introduce a registered event caching layer that is keyed on the source/service identifiers. Obviously, that adds more complexity and interacts negatively with expiring metrics.

rfcs/2023-05-03-data-volume-metrics.md Show resolved Hide resolved
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Comment on lines 141 to 144
It is currently not possible to have dynamic tags with preregistered metrics.

We will need to introduce a registered event caching layer that will register
and cache new events keyed on the tags that are sent to it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we definitely want to do this? It does seem like it'll add some complexity and somewhat duplicate the existing notion of "expiring metrics". Could we also just not register these metrics anymore? cc/ @bruceg

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we do need to do this, otherwise we risk the event counts expiring and resetting to zero.


## Future Improvements

- Logs emitted by Vector should also be tagged with `source_id` and `service`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know how much effort this would be? If it is on the order of a few days it might be nice to do as part of this work since I can see users expecting it. If it'll take weeks then we can defer it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more than we can fit in to the deadline, we agreed to defer for now.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
@StephenWakely StephenWakely requested a review from tobz May 24, 2023 10:14
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last comment, otherwise 👍

emitted use this. The `EstimatedJsonEncodedSizeOf` trait will be updated return a `JsonSize`.
- [ ] Add the Service meaning. Update any sources that potentially create a service to point the meaning
to the relevant field.
- [ ] Introduce an event caching layer that caches registered events based on the tags sent to it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add benchmarking as an explicit part of the plan of action here. I think it'll be important to understand the performance impact.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have added.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
@StephenWakely StephenWakely added this pull request to the merge queue May 26, 2023
Merged via the queue into master with commit a551f33 May 26, 2023
@StephenWakely StephenWakely deleted the stephen/data_volume_rfc branch May 26, 2023 21:18
aholmberg pushed a commit to aholmberg/vector that referenced this pull request Feb 14, 2024
# [1.30.0](answerbook/vector@v1.29.2...v1.30.0) (2023-12-19)

### Bug Fixes

* `aws_ec2_metadata` transform when using log namespacing (vectordotdev#17819) [4786743](answerbook/vector@4786743) - GitHub
* **auth**: Vector does not put the Proxy-Authorization header on the wire (vectordotdev#17353) (vectordotdev#17363) [6705bdd](answerbook/vector@6705bdd) - GitHub
* **buffers**: deadlock when seeking after entire write fails to be flushed (vectordotdev#17657) [37a662a](answerbook/vector@37a662a) - GitHub
* **ci**: add missing env var (vectordotdev#17872) [4f67695](answerbook/vector@4f67695) - Jesse Szwedko
* **ci**: add missing logic to mark required checks failed (vectordotdev#17543) [3b87e00](answerbook/vector@3b87e00) - GitHub
* **ci**: change command to find baseline sha from issue comment trigger (vectordotdev#17622) [5791083](answerbook/vector@5791083) - GitHub
* **ci**: checkout a greater depth in regression workflow (vectordotdev#17604) [baa04e5](answerbook/vector@baa04e5) - GitHub
* **ci**: post failed status to PR and isolate branch checkout on comment trigger (vectordotdev#17544) [e2c0255](answerbook/vector@e2c0255) - GitHub
* **ci**: reg workflow alt approach to getting baseline sha (vectordotdev#17645) [f1e1ae3](answerbook/vector@f1e1ae3) - GitHub
* **ci**: use correct ID for Triage in Gardener Board (vectordotdev#17647) [2638cca](answerbook/vector@2638cca) - GitHub
* **ci**: use correct secret for gardener board comment (vectordotdev#17605) [9395eba](answerbook/vector@9395eba) - GitHub
* **config**: Fix preloading log_schema (vectordotdev#17759) [659e1e6](answerbook/vector@659e1e6) - GitHub
* **databend sink**: use get for page request (vectordotdev#17373) [c7d7cf8](answerbook/vector@c7d7cf8) - GitHub
* **datadog_agent source**: remove duplicate internal metrics emission (vectordotdev#17720) [48ec2e8](answerbook/vector@48ec2e8) - GitHub
* **distribution**: Fix architecture detection for ARMv7 (vectordotdev#17484) [78fb469](answerbook/vector@78fb469) - GitHub
* **docs**: fix copy-paste issue in component spec (vectordotdev#17616) [b400acc](answerbook/vector@b400acc) - GitHub
* **file source**: Fix tailing problem when source number greater than 512 (vectordotdev#17717) [23a3e0e](answerbook/vector@23a3e0e) - GitHub
* **fluent source**: fix ack message format (vectordotdev#17407) [d194992](answerbook/vector@d194992) - GitHub
* **http_client source**: adapt int test to use breaking change of dep (vectordotdev#17583) [d7df520](answerbook/vector@d7df520) - GitHub
* **http_client source**: remove utf8 lossy conversion (vectordotdev#17655) [59e2cbf](answerbook/vector@59e2cbf) - GitHub
* **install.sh**: Correctly `shift` all parsed arguments (vectordotdev#17684) [f883575](answerbook/vector@f883575) - GitHub
* **loki sink, observability**: Drop non-fatal template render errors to warnings (vectordotdev#17746) [4ebc3e1](answerbook/vector@4ebc3e1) - GitHub
* **loki sink**: use json size of unencoded event (vectordotdev#17572) [25e7699](answerbook/vector@25e7699) - GitHub
* **observability**: correct emitted metrics (vectordotdev#17562) [7a4f1f7](answerbook/vector@7a4f1f7) - GitHub
* **observability**: issues with event_cache PR (vectordotdev#17768) [fdf02d9](answerbook/vector@fdf02d9) - GitHub
* remap behavior for root types when using the `Vector` namespace (vectordotdev#17807) [c19938c](answerbook/vector@c19938c) - GitHub
* **sinks**: Add missing component span for sink building (vectordotdev#17765) [219883e](answerbook/vector@219883e) - GitHub

### Chores

* Add docker config to dependabot (vectordotdev#17696) [079d895](answerbook/vector@079d895) - GitHub
* add sink prelude (vectordotdev#17595) [da939ca](answerbook/vector@da939ca) - GitHub
* Add submodules to all checkouts (vectordotdev#17770) [7196622](answerbook/vector@7196622) - GitHub
* **administration**: add domain label for vdev (vectordotdev#17748) [c35ebd1](answerbook/vector@c35ebd1) - GitHub
* **aws_s3 sink**: Update metadata to match the editorial review for the schema. (vectordotdev#17475) [c1262cd](answerbook/vector@c1262cd) - GitHub
* Bump version to 0.31.0 (vectordotdev#17466) [78bbfbc](answerbook/vector@78bbfbc) - GitHub
* **ci**: Add apt retries to cross builds (vectordotdev#17683) [ab1169b](answerbook/vector@ab1169b) - GitHub
* **ci**: Add schedule to component features workflow conditional check (vectordotdev#17816) [708b7f6](answerbook/vector@708b7f6) - GitHub
* **ci**: Bump aws-actions/configure-aws-credentials from 2.0.0 to 2.1.0 (vectordotdev#17565) [8a741d5](answerbook/vector@8a741d5) - GitHub
* **ci**: Bump aws-actions/configure-aws-credentials from 2.1.0 to 2.2.0 (vectordotdev#17697) [12bc4a7](answerbook/vector@12bc4a7) - GitHub
* **ci**: Bump docker/build-push-action from 4.0.0 to 4.1.0 (vectordotdev#17656) [cb9a3a5](answerbook/vector@cb9a3a5) - GitHub
* **ci**: Bump docker/build-push-action from 4.1.0 to 4.1.1 (vectordotdev#17687) [bce5e65](answerbook/vector@bce5e65) - GitHub
* **ci**: Bump docker/metadata-action from 4.4.0 to 4.5.0 (vectordotdev#17624) [a54a12f](answerbook/vector@a54a12f) - GitHub
* **ci**: Bump docker/metadata-action from 4.5.0 to 4.6.0 (vectordotdev#17686) [71273df](answerbook/vector@71273df) - GitHub
* **ci**: Bump docker/setup-buildx-action from 2.5.0 to 2.6.0 (vectordotdev#17625) [15bc42a](answerbook/vector@15bc42a) - GitHub
* **ci**: Bump docker/setup-buildx-action from 2.6.0 to 2.7.0 (vectordotdev#17685) [8006987](answerbook/vector@8006987) - GitHub
* **ci**: Bump docker/setup-buildx-action from 2.7.0 to 2.8.0 (vectordotdev#17786) [dbdff9e](answerbook/vector@dbdff9e) - GitHub
* **ci**: Bump docker/setup-qemu-action from 2.1.0 to 2.2.0 (vectordotdev#17623) [3005141](answerbook/vector@3005141) - GitHub
* **ci**: bump myrotvorets/set-commit-status-action from 1.1.6 to 1.1.7 (vectordotdev#17460) [bca45eb](answerbook/vector@bca45eb) - GitHub
* **ci**: Bump up OSX runners for release builds (vectordotdev#17823) [fe730ad](answerbook/vector@fe730ad) - GitHub
* **ci**: bump xt0rted/pull-request-comment-branch from 1 to 2 (vectordotdev#17461) [c425006](answerbook/vector@c425006) - GitHub
* **ci**: correctly validate comment author in k8s e2e job (vectordotdev#17818) [b8e3dbe](answerbook/vector@b8e3dbe) - GitHub
* **ci**: Drop VRL license exceptions (vectordotdev#17529) [aa01452](answerbook/vector@aa01452) - GitHub
* **ci**: fix a few logic bugs and more strict comment parsing (vectordotdev#17502) [bf372fd](answerbook/vector@bf372fd) - GitHub
* **ci**: fix comment author validation (vectordotdev#17794) [75ae967](answerbook/vector@75ae967) - GitHub
* **ci**: fix failure notify job conditional in publish workflow (vectordotdev#17468) [3699842](answerbook/vector@3699842) - GitHub
* **ci**: fix gardener issues comment workflow (vectordotdev#17825) [47c3da1](answerbook/vector@47c3da1) - GitHub
* **ci**: fix team membership action (vectordotdev#17791) [13c3c78](answerbook/vector@13c3c78) - GitHub
* **ci**: int test yaml file detection (vectordotdev#17590) [fa8a553](answerbook/vector@fa8a553) - GitHub
* **ci**: minor fixes to workflows post merge queue enabling  (vectordotdev#17462) [9f6f6ec](answerbook/vector@9f6f6ec) - GitHub
* **ci**: move component features check out of merge queue (vectordotdev#17773) [e6e776d](answerbook/vector@e6e776d) - GitHub
* **ci**: Move most CI checks to merge queue (vectordotdev#17340) [060399a](answerbook/vector@060399a) - GitHub
* **ci**: reduce runner sizing to 4 core and free tier (vectordotdev#17785) [53a575f](answerbook/vector@53a575f) - GitHub
* **ci**: remove /ci-run-install comment trigger (vectordotdev#17803) [a3770d8](answerbook/vector@a3770d8) - GitHub
* **ci**: Remove remaining Discord notification (vectordotdev#17805) [ed59f37](answerbook/vector@ed59f37) - GitHub
* **ci**: Remove upload of config schema (vectordotdev#17740) [ff6a1b4](answerbook/vector@ff6a1b4) - GitHub
* **ci**: Retry `make check-component-docs` check (vectordotdev#17718) [e8e7e04](answerbook/vector@e8e7e04) - GitHub
* **ci**: revert fix gardener issues comment workflow (vectordotdev#17829) [ee10b8c](answerbook/vector@ee10b8c) - GitHub
* **ci**: Set HOMEBREW_NO_INSTALL_FROM_API in CI (vectordotdev#17867) [00cc584](answerbook/vector@00cc584) - Jesse Szwedko
* **ci**: temporarily disable comment_trigger workflow (vectordotdev#17480) [58d7f3d](answerbook/vector@58d7f3d) - GitHub
* **ci**: temporarily disable flakey `aws_s3` integration test case `handles_errored_status`  (vectordotdev#17455) [8e40b68](answerbook/vector@8e40b68) - GitHub
* **ci**: update comment_trigger note about concurrency groups (vectordotdev#17491) [7699f4d](answerbook/vector@7699f4d) - GitHub
* **ci**: Update publish workflow test Ubuntu versions (vectordotdev#17781) [20d62f1](answerbook/vector@20d62f1) - GitHub
* **clickhouse sink**: refactor to new style (vectordotdev#17723) [77ac63c](answerbook/vector@77ac63c) - GitHub
* **codecs**: consolidate enum types (vectordotdev#17688) [9c45394](answerbook/vector@9c45394) - GitHub
* Codify flag naming including sentinel values (vectordotdev#17569) [134578d](answerbook/vector@134578d) - GitHub
* Codify the use of abbreviate time units in config option names (vectordotdev#17582) [8823561](answerbook/vector@8823561) - GitHub
* **config**: Convert top-level sinks enum to typetag (vectordotdev#17710) [9cd5404](answerbook/vector@9cd5404) - GitHub
* **config**: Make config schema output ordered (vectordotdev#17694) [9606353](answerbook/vector@9606353) - GitHub
* **config**: Update field labels for commonly used sources and transforms  (vectordotdev#17517) [f523f70](answerbook/vector@f523f70) - GitHub
* **config**: Update field labels for sinks (vectordotdev#17560) [e1ddd0e](answerbook/vector@e1ddd0e) - GitHub
* **config**: Update field labels for the rest of the sources and transforms fields (vectordotdev#17564) [6e45477](answerbook/vector@6e45477) - GitHub
* **datadog_archives sink**: Remove this component (vectordotdev#17749) [53f8bff](answerbook/vector@53f8bff) - GitHub
* **datadog_metrics sink**: incrementally encode sketches (vectordotdev#17764) [3f6df61](answerbook/vector@3f6df61) - GitHub
* **datadog_traces sink**: Add additional warning around APM stats for `peer.service` (vectordotdev#17733) [9a899c5](answerbook/vector@9a899c5) - GitHub
* **deps, releasing**: Update to Alpine 3.18 (vectordotdev#17695) [2263756](answerbook/vector@2263756) - GitHub
* **deps**: Bump async-graphql from 5.0.8 to 5.0.9 (vectordotdev#17486) [077a294](answerbook/vector@077a294) - GitHub
* **deps**: Bump async-graphql from 5.0.9 to 5.0.10 (vectordotdev#17619) [2931542](answerbook/vector@2931542) - GitHub
* **deps**: Bump async-graphql-warp from 5.0.8 to 5.0.9 (vectordotdev#17489) [ac81fc1](answerbook/vector@ac81fc1) - GitHub
* **deps**: Bump async-graphql-warp from 5.0.9 to 5.0.10 (vectordotdev#17642) [b3885f6](answerbook/vector@b3885f6) - GitHub
* **deps**: bump aws-sigv4 from 0.55.1 to 0.55.3 (vectordotdev#17481) [2ad5b47](answerbook/vector@2ad5b47) - GitHub
* **deps**: bump base64 from 0.21.0 to 0.21.1 (vectordotdev#17451) [95cbba9](answerbook/vector@95cbba9) - GitHub
* **deps**: Bump base64 from 0.21.1 to 0.21.2 (vectordotdev#17488) [f261781](answerbook/vector@f261781) - GitHub
* **deps**: bump bstr from 1.4.0 to 1.5.0 (vectordotdev#17453) [7554d9c](answerbook/vector@7554d9c) - GitHub
* **deps**: Bump cached from 0.43.0 to 0.44.0 (vectordotdev#17599) [7a55210](answerbook/vector@7a55210) - GitHub
* **deps**: Bump chrono to 0.4.26 (vectordotdev#17537) [0dfa09c](answerbook/vector@0dfa09c) - GitHub
* **deps**: Bump chrono-tz from 0.8.2 to 0.8.3 (vectordotdev#17789) [f79947c](answerbook/vector@f79947c) - GitHub
* **deps**: bump clap_complete from 4.2.3 to 4.3.0 (vectordotdev#17447) [05bf262](answerbook/vector@05bf262) - GitHub
* **deps**: Bump clap_complete from 4.3.0 to 4.3.1 (vectordotdev#17586) [8549809](answerbook/vector@8549809) - GitHub
* **deps**: bump criterion from 0.4.0 to 0.5.0 (vectordotdev#17477) [84f0ada](answerbook/vector@84f0ada) - GitHub
* **deps**: Bump criterion from 0.5.0 to 0.5.1 (vectordotdev#17500) [da7bc95](answerbook/vector@da7bc95) - GitHub
* **deps**: Bump crossbeam-utils from 0.8.15 to 0.8.16 (vectordotdev#17674) [714ccf8](answerbook/vector@714ccf8) - GitHub
* **deps**: Bump csv from 1.2.1 to 1.2.2 (vectordotdev#17555) [bcc5b6c](answerbook/vector@bcc5b6c) - GitHub
* **deps**: bump data-encoding from 2.3.3 to 2.4.0 (vectordotdev#17452) [9aaf864](answerbook/vector@9aaf864) - GitHub
* **deps**: Bump getrandom from 0.2.9 to 0.2.10 (vectordotdev#17613) [bd880f5](answerbook/vector@bd880f5) - GitHub
* **deps**: Bump gloo-utils from 0.1.6 to 0.1.7 (vectordotdev#17707) [53e1785](answerbook/vector@53e1785) - GitHub
* **deps**: Bump graphql_client from 0.12.0 to 0.13.0 (vectordotdev#17541) [ecb707a](answerbook/vector@ecb707a) - GitHub
* **deps**: Bump h2 from 0.3.19 to 0.3.20 (vectordotdev#17767) [ac3bc72](answerbook/vector@ac3bc72) - GitHub
* **deps**: Bump hashbrown from 0.13.2 to 0.14.0 (vectordotdev#17609) [154e393](answerbook/vector@154e393) - GitHub
* **deps**: Bump hyper from 0.14.26 to 0.14.27 (vectordotdev#17766) [a2a3609](answerbook/vector@a2a3609) - GitHub
* **deps**: Bump indexmap from 1.9.3 to 2.0.0 (vectordotdev#17755) [248ccb8](answerbook/vector@248ccb8) - GitHub
* **deps**: Bump indicatif from 0.17.3 to 0.17.4 (vectordotdev#17532) [1565985](answerbook/vector@1565985) - GitHub
* **deps**: Bump indicatif from 0.17.4 to 0.17.5 (vectordotdev#17597) [a164952](answerbook/vector@a164952) - GitHub
* **deps**: Bump infer from 0.13.0 to 0.14.0 (vectordotdev#17737) [326ad08](answerbook/vector@326ad08) - GitHub
* **deps**: Bump itertools from 0.10.5 to 0.11.0 (vectordotdev#17736) [6e1878b](answerbook/vector@6e1878b) - GitHub
* **deps**: Bump lalrpop to 0.19.12 (vectordotdev#17457) [1f54415](answerbook/vector@1f54415) - GitHub
* **deps**: bump lapin from 2.1.1 to 2.1.2 (vectordotdev#17439) [a8b7899](answerbook/vector@a8b7899) - GitHub
* **deps**: bump lapin from 2.1.2 to 2.2.0 (vectordotdev#17443) [b639422](answerbook/vector@b639422) - GitHub
* **deps**: bump lapin from 2.2.0 to 2.2.1 (vectordotdev#17448) [618379a](answerbook/vector@618379a) - GitHub
* **deps**: Bump libc from 0.2.144 to 0.2.146 (vectordotdev#17615) [10cfd0a](answerbook/vector@10cfd0a) - GitHub
* **deps**: Bump libc from 0.2.146 to 0.2.147 (vectordotdev#17753) [1a75ec6](answerbook/vector@1a75ec6) - GitHub
* **deps**: Bump log from 0.4.17 to 0.4.18 (vectordotdev#17526) [5a2fea1](answerbook/vector@5a2fea1) - GitHub
* **deps**: Bump log from 0.4.18 to 0.4.19 (vectordotdev#17662) [e1b3357](answerbook/vector@e1b3357) - GitHub
* **deps**: Bump lru from 0.10.0 to 0.10.1 (vectordotdev#17810) [96e68f7](answerbook/vector@96e68f7) - GitHub
* **deps**: bump memmap2 from 0.6.1 to 0.6.2 (vectordotdev#17482) [79f7dfb](answerbook/vector@79f7dfb) - GitHub
* **deps**: Bump memmap2 from 0.6.2 to 0.7.0 (vectordotdev#17641) [593ea1b](answerbook/vector@593ea1b) - GitHub
* **deps**: Bump memmap2 from 0.7.0 to 0.7.1 (vectordotdev#17752) [4236e32](answerbook/vector@4236e32) - GitHub
* **deps**: Bump mock_instant from 0.3.0 to 0.3.1 (vectordotdev#17574) [1c1beb8](answerbook/vector@1c1beb8) - GitHub
* **deps**: Bump mongodb from 2.5.0 to 2.6.0 (vectordotdev#17726) [c96e3be](answerbook/vector@c96e3be) - GitHub
* **deps**: Bump notify from 6.0.0 to 6.0.1 (vectordotdev#17700) [cd6d154](answerbook/vector@cd6d154) - GitHub
* **deps**: Bump once_cell from 1.17.1 to 1.17.2 (vectordotdev#17531) [8e113ad](answerbook/vector@8e113ad) - GitHub
* **deps**: Bump once_cell from 1.17.2 to 1.18.0 (vectordotdev#17596) [dc6bef2](answerbook/vector@dc6bef2) - GitHub
* **deps**: bump opendal from 0.34.0 to 0.35.0 (vectordotdev#17471) [ebf958b](answerbook/vector@ebf958b) - GitHub
* **deps**: Bump opendal from 0.35.0 to 0.36.0 (vectordotdev#17540) [dbd7151](answerbook/vector@dbd7151) - GitHub
* **deps**: Bump opendal from 0.36.0 to 0.37.0 (vectordotdev#17614) [b5bd85f](answerbook/vector@b5bd85f) - GitHub
* **deps**: Bump opendal from 0.37.0 to 0.38.0 (vectordotdev#17777) [ec4785a](answerbook/vector@ec4785a) - GitHub
* **deps**: Bump openssl from 0.10.52 to 0.10.53 (vectordotdev#17534) [078de66](answerbook/vector@078de66) - GitHub
* **deps**: Bump openssl from 0.10.53 to 0.10.54 (vectordotdev#17573) [4af5e6d](answerbook/vector@4af5e6d) - GitHub
* **deps**: Bump openssl from 0.10.54 to 0.10.55 (vectordotdev#17716) [dd2527d](answerbook/vector@dd2527d) - GitHub
* **deps**: Bump percent-encoding from 2.2.0 to 2.3.0 (vectordotdev#17602) [8e04259](answerbook/vector@8e04259) - GitHub
* **deps**: Bump pin-project from 1.1.0 to 1.1.1 (vectordotdev#17806) [0b32626](answerbook/vector@0b32626) - GitHub
* **deps**: Bump PR limit for Dependabot to 100 (vectordotdev#17459) [85703e7](answerbook/vector@85703e7) - GitHub
* **deps**: bump proc-macro2 from 1.0.57 to 1.0.58 (vectordotdev#17426) [ae656c7](answerbook/vector@ae656c7) - GitHub
* **deps**: Bump proc-macro2 from 1.0.58 to 1.0.59 (vectordotdev#17495) [4ce3278](answerbook/vector@4ce3278) - GitHub
* **deps**: Bump proc-macro2 from 1.0.59 to 1.0.60 (vectordotdev#17643) [f20eb2f](answerbook/vector@f20eb2f) - GitHub
* **deps**: Bump proc-macro2 from 1.0.60 to 1.0.63 (vectordotdev#17757) [63ba2a9](answerbook/vector@63ba2a9) - GitHub
* **deps**: bump proptest from 1.1.0 to 1.2.0 (vectordotdev#17476) [9235fc2](answerbook/vector@9235fc2) - GitHub
* **deps**: bump pulsar from 5.1.1 to 6.0.0 (vectordotdev#17587) [3395cfd](answerbook/vector@3395cfd) - GitHub
* **deps**: Bump pulsar from 6.0.0 to 6.0.1 (vectordotdev#17673) [8d98bb8](answerbook/vector@8d98bb8) - GitHub
* **deps**: Bump quanta from 0.11.0 to 0.11.1 (vectordotdev#17524) [2388c2f](answerbook/vector@2388c2f) - GitHub
* **deps**: Bump quote from 1.0.27 to 1.0.28 (vectordotdev#17496) [cc30746](answerbook/vector@cc30746) - GitHub
* **deps**: Bump quote from 1.0.28 to 1.0.29 (vectordotdev#17798) [cba983e](answerbook/vector@cba983e) - GitHub
* **deps**: Bump quote from 1.0.28 to 1.0.29 (vectordotdev#17815) [bf9828d](answerbook/vector@bf9828d) - GitHub
* **deps**: bump rdkafka from 0.30.0 to 0.31.0 (vectordotdev#17428) [e7fa8d3](answerbook/vector@e7fa8d3) - GitHub
* **deps**: Bump rdkafka from 0.31.0 to 0.32.2 (vectordotdev#17664) [ac68a7b](answerbook/vector@ac68a7b) - GitHub
* **deps**: bump regex from 1.8.1 to 1.8.2 (vectordotdev#17469) [897e45d](answerbook/vector@897e45d) - GitHub
* **deps**: bump regex from 1.8.2 to 1.8.3 (vectordotdev#17494) [5d90cff](answerbook/vector@5d90cff) - GitHub
* **deps**: Bump regex from 1.8.3 to 1.8.4 (vectordotdev#17601) [657758d](answerbook/vector@657758d) - GitHub
* **deps**: bump reqwest from 0.11.17 to 0.11.18 (vectordotdev#17420) [2ed8ec7](answerbook/vector@2ed8ec7) - GitHub
* **deps**: bump security-framework from 2.9.0 to 2.9.1 (vectordotdev#17441) [ac0c7e8](answerbook/vector@ac0c7e8) - GitHub
* **deps**: Bump serde from 1.0.163 to 1.0.164 (vectordotdev#17632) [e35150e](answerbook/vector@e35150e) - GitHub
* **deps**: Bump serde_json from 1.0.96 to 1.0.97 (vectordotdev#17701) [25131ef](answerbook/vector@25131ef) - GitHub
* **deps**: Bump serde_json from 1.0.97 to 1.0.99 (vectordotdev#17754) [e07158c](answerbook/vector@e07158c) - GitHub
* **deps**: Bump serde_yaml from 0.9.21 to 0.9.22 (vectordotdev#17756) [e164b36](answerbook/vector@e164b36) - GitHub
* **deps**: Bump sha2 from 0.10.6 to 0.10.7 (vectordotdev#17698) [d122d32](answerbook/vector@d122d32) - GitHub
* **deps**: Bump tempfile from 3.5.0 to 3.6.0 (vectordotdev#17617) [c55c9ec](answerbook/vector@c55c9ec) - GitHub
* **deps**: Bump tokio from 1.28.1 to 1.28.2 (vectordotdev#17525) [cc703da](answerbook/vector@cc703da) - GitHub
* **deps**: Bump tokio from 1.28.2 to 1.29.0 (vectordotdev#17776) [e26e8b8](answerbook/vector@e26e8b8) - GitHub
* **deps**: bump toml from 0.7.3 to 0.7.4 (vectordotdev#17440) [91ba052](answerbook/vector@91ba052) - GitHub
* **deps**: Bump toml from 0.7.4 to 0.7.5 (vectordotdev#17751) [35c4581](answerbook/vector@35c4581) - GitHub
* **deps**: Bump tower-http from 0.4.0 to 0.4.1 (vectordotdev#17711) [e5e6b96](answerbook/vector@e5e6b96) - GitHub
* **deps**: Bump url from 2.3.1 to 2.4.0 (vectordotdev#17608) [d956092](answerbook/vector@d956092) - GitHub
* **deps**: Bump uuid from 1.3.3 to 1.3.4 (vectordotdev#17682) [c97d619](answerbook/vector@c97d619) - GitHub
* **deps**: Bump uuid from 1.3.4 to 1.4.0 (vectordotdev#17775) [935babf](answerbook/vector@935babf) - GitHub
* **deps**: Bump wasm-bindgen from 0.2.86 to 0.2.87 (vectordotdev#17672) [19c4d4f](answerbook/vector@19c4d4f) - GitHub
* **deps**: Bump wiremock from 0.5.18 to 0.5.19 (vectordotdev#17618) [460bbc7](answerbook/vector@460bbc7) - GitHub
* **deps**: Bump xml-rs from 0.8.4 to 0.8.14 (vectordotdev#17607) [a932489](answerbook/vector@a932489) - GitHub
* **deps**: Drop use of `hashlink` crate (vectordotdev#17678) [41ee394](answerbook/vector@41ee394) - GitHub
* **deps**: Export more common bits for components (vectordotdev#17788) [062224b](answerbook/vector@062224b) - GitHub
* **deps**: Update fs_extra to 1.3.0 (vectordotdev#17458) [299fd6a](answerbook/vector@299fd6a) - GitHub
* **deps**: Upgrade Ruby version to 3.1.4 (vectordotdev#17722) [ddebde9](answerbook/vector@ddebde9) - GitHub
* **deps**: Upgrade rust to 1.70.0 (vectordotdev#17585) [6c48565](answerbook/vector@6c48565) - GitHub
* **dev**: Add @dsmith3197 to CODEOWNERS (vectordotdev#17729) [a08443c](answerbook/vector@a08443c) - GitHub
* **docs**: Add info about Vector Operator to Kubernetes instalation page (vectordotdev#17432) [54d9c99](answerbook/vector@54d9c99) - GitHub
* **docs**: add instructions for regenerating component docs and licenses (vectordotdev#17828) [93ef6c3](answerbook/vector@93ef6c3) - GitHub
* **docs**: Add Log Namespacing docs (vectordotdev#16571) [7d098e4](answerbook/vector@7d098e4) - GitHub
* **docs**: add note about const strings (vectordotdev#17774) [d7bc531](answerbook/vector@d7bc531) - GitHub
* **docs**: Clarify `bytes` framing for streams (vectordotdev#17745) [7d10fc9](answerbook/vector@7d10fc9) - GitHub
* **docs**: Clarify when component received and sent bytes events should be emitted (vectordotdev#17464) [547783d](answerbook/vector@547783d) - GitHub
* **docs**: Move CONTRIBUTING.md to top-level (vectordotdev#17744) [7a0dec1](answerbook/vector@7a0dec1) - GitHub
* Download submodules in the CI checkouts (vectordotdev#17760) [5417a06](answerbook/vector@5417a06) - GitHub
* Dropped error field from StreamClosed Error (vectordotdev#17693) [ee480cd](answerbook/vector@ee480cd) - GitHub
* **enrichment**: avoid importing vector-common in enrichment module (vectordotdev#17653) [45a28f8](answerbook/vector@45a28f8) - GitHub
* **enterprise**: Extend library functionality for secret scanning (vectordotdev#17483) [541bb00](answerbook/vector@541bb00) - GitHub
* **external docs**: fix reference to supported aarch64 architecture (vectordotdev#17553) [247bb80](answerbook/vector@247bb80) - GitHub
* **external docs**: update fluentd link (vectordotdev#17436) [187f142](answerbook/vector@187f142) - GitHub
* Fix publish workflow for older OS images (vectordotdev#17787) [ab39c6a](answerbook/vector@ab39c6a) - GitHub
* **flush on shutdown**: validate s3 sink flushes (vectordotdev#17667) [c21f892](answerbook/vector@c21f892) - GitHub
* **kubernetes_logs source**: Add warning about Windows support (vectordotdev#17762) [a53c7a2](answerbook/vector@a53c7a2) - GitHub
* **kubernetes**: Bump k8s manifests to 0.22.0 (vectordotdev#17467) [f547871](answerbook/vector@f547871) - GitHub
* **observability**: emit `component_sent` events by `source` and `service` (vectordotdev#17549) [dcf7f9a](answerbook/vector@dcf7f9a) - GitHub
* **observability**: ensure `sent_event` and `received_event` metrics are estimated json size (vectordotdev#17465) [3b2a2be](answerbook/vector@3b2a2be) - GitHub
* **observability**: Have `tower_limit` use configured log level (vectordotdev#17715) [08099a8](answerbook/vector@08099a8) - GitHub
* **observability**: remove deprecated internal metrics + massive cleanup to vector top and graphql API (vectordotdev#17516) [98c54ad](answerbook/vector@98c54ad) - GitHub
* **observability**: remove more deprecated internal metrics (vectordotdev#17542) [b0ed167](answerbook/vector@b0ed167) - GitHub
* **observability**: set source fields to mean service (vectordotdev#17470) [670bdea](answerbook/vector@670bdea) - GitHub
* **releasing**: Prepare v0.30.0 release [af2b2af](answerbook/vector@af2b2af) - Jesse Szwedko
* **releasing**: Prepare v0.31.0 release [0f13b22](answerbook/vector@0f13b22) - Jesse Szwedko
* remove custom async sleep impl (vectordotdev#17493) [b28d915](answerbook/vector@b28d915) - GitHub
* Remove links to roadmap (vectordotdev#17554) [349c718](answerbook/vector@349c718) - GitHub
* Revert all submodule introductions to fix CI (vectordotdev#17800) [d8d57e5](answerbook/vector@d8d57e5) - GitHub
* RFC for Data Volume Insights (vectordotdev#17322) [a551f33](answerbook/vector@a551f33) - GitHub
* **sinks**: Drop the custom `SinkContext::default` implementation (vectordotdev#17804) [e66e285](answerbook/vector@e66e285) - GitHub
* **sinks**: mark VectorSink::from_event_sink as deprecated (vectordotdev#17649) [0dc450f](answerbook/vector@0dc450f) - GitHub
* **statsd sink**: refactor `statsd` sink to stream-based style (vectordotdev#16199) [2a76cac](answerbook/vector@2a76cac) - GitHub
* update `vrl` to `0.4.0` (vectordotdev#17378) [426d660](answerbook/vector@426d660) - GitHub
* Update the NOTICE file (vectordotdev#17430) [9a44e6e](answerbook/vector@9a44e6e) - GitHub
* Upgrade aws-smithy and aws-sdk crates (vectordotdev#17731) [6a6b42b](answerbook/vector@6a6b42b) - GitHub
* **website**: Fix upgrade guide dates [80de738](answerbook/vector@80de738) - Jesse Szwedko

### Features

* add metadata support to `set_semantic_meaning` (vectordotdev#17730) [44be378](answerbook/vector@44be378) - GitHub
* **codecs**: add lossy option to `gelf`, `native_json`, and `syslog` deserializers (vectordotdev#17680) [2dfa850](answerbook/vector@2dfa850) - GitHub
* **codecs**: Add lossy option to JSON deserializer (vectordotdev#17628) [bf7d796](answerbook/vector@bf7d796) - GitHub
* **configurable shutdown duration**: make shutdown duration configurable (vectordotdev#17479) [23ed0e3](answerbook/vector@23ed0e3) - GitHub
* **error code when shutdown fails**: set exit flag to non-zero when shutdown times out (vectordotdev#17676) [cc52c0e](answerbook/vector@cc52c0e) - GitHub
* **internal telemetry at shutdown**: close internal sources after external ones (vectordotdev#17741) [812929b](answerbook/vector@812929b) - GitHub
* **journald source**: add journal_namespace option (vectordotdev#17648) [a324a07](answerbook/vector@a324a07) - GitHub
* **kinesis sinks**: implement full retry of partial failures in firehose/streams (vectordotdev#17535) [bebac21](answerbook/vector@bebac21) - GitHub
* **prometheus**: add more compression algorithms to Prometheus Remote Write (vectordotdev#17334) [380d7ad](answerbook/vector@380d7ad) - GitHub
* track runtime schema definitions for log events (vectordotdev#17692) [6eecda5](answerbook/vector@6eecda5) - GitHub

### Miscellaneous

* Merge pull request vectordotdev#371 from answerbook/feature/LOG-18200 [491ea31](answerbook/vector@491ea31) - GitHub [LOG-18200](https://logdna.atlassian.net/browse/LOG-18200)
* Merge pull request vectordotdev#374 from answerbook/dominic/LOG-18535-temp-revert [df87df7](answerbook/vector@df87df7) - GitHub [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535) [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535)
* Revert "Merge pull request vectordotdev#370 from answerbook/dominic/LOG-18535" [73a8b28](answerbook/vector@73a8b28) - dominic-mcallister-logdna [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535)
* Merge branch 'master' into feature/LOG-18200 [fcd21f3](answerbook/vector@fcd21f3) - Darin Spivey [LOG-18200](https://logdna.atlassian.net/browse/LOG-18200)
* Merge tag 'v0.31.0' into upstream-0.31.0 [9443fb1](answerbook/vector@9443fb1) - Darin Spivey
* Update VRL to `0.5.0` (vectordotdev#17793) [671aa79](answerbook/vector@671aa79) - GitHub
* fix `demo_logs` metadata source name (vectordotdev#17689) [83af7ea](answerbook/vector@83af7ea) - GitHub
* enhancement(s3 source) Add minimal support to unwrap an S3-SQS event from an SNS event (vectordotdev#17352) [7a7bc9a](answerbook/vector@7a7bc9a) - GitHub
* Additional notes on proposing new integrations (vectordotdev#17658) [2ad964d](answerbook/vector@2ad964d) - GitHub
* **ci**: reduce billable time of Test Suite (vectordotdev#17714) [bc69255](answerbook/vector@bc69255) - GitHub
* **ci**: refactor logic for int test file path changes detection (vectordotdev#17725) [92a36e0](answerbook/vector@92a36e0) - GitHub
* **compression**: zstd compression support (vectordotdev#17371) [ced219e](answerbook/vector@ced219e) - GitHub
* **dev**:  move blocked/waiting gardener issues to triage on comment (vectordotdev#17588) [6b34868](answerbook/vector@6b34868) - GitHub
* explain how to run tests locally (vectordotdev#17783) [3b67a80](answerbook/vector@3b67a80) - GitHub
* remove aggregator beta warning (vectordotdev#17750) [94e3f15](answerbook/vector@94e3f15) - GitHub
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants