feat: track runtime schema definitions for log events #17692
Conversation
✅ Deploy Preview for vector-project canceled.
|
✅ Deploy Preview for vrl-playground canceled.
|
github-actions
bot
added
domain: core
Datadog ReportBranch report: ✅ |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
I'm not always getting the schema definition through to the sources.
To test I've added dbg!(event.metadata().schema_definition()); to line 35 of src/sink/console/sink.rs.
With this config:
sources:
stdin:
type: stdin
transforms:
mapple1:
type: remap
inputs: [stdin]
source: |
.nog = "nork"
set_semantic_meaning(.nog, "service")
mapple2:
type: remap
inputs: [mapple1]
source: .nug = "nunk"
sinks:
hole:
type: console
inputs: [mapple1]
target: stdout
encoding:
codec: jsonWhen the input is mapple1 I am getting the schema definition with the correct meaning through. But if I change the input to mapple2 there are no meanings in the schema definition. I presume this should come through regardless of the number of transforms it passes through?
The parent_id seems always correct.
This is because you haven't enabled log namespacing. The function that gets the schema definition only adds meanings when log namespacing is enabled. (you wrote this function too 😆 ) Add the following to the top of the config and it seems to work fine: schema:
log_namespace: trueEdit: after some discussion, I ended up adding semantic meanings to all events, regardless of the log namespacing setting. |
|
@jszwedko I'm thinking we will likely want this in the v0.31.0 milestone. Thoughts? |
Yes, I talked to him previously about this, and we are wanting to release this soon so we can start beta testing log namespacing. I added it to the milestone. |
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Regression Detector ResultsRun ID: 8f0e6c60-c9a1-4bcc-befd-5a77165d9c97 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:
Fine details of change detection per experiment.
|
Regression Detector ResultsRun ID: b213a498-7851-4ea5-9eb0-9840eacc5176 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%. Fine details of change detection per experiment.
|
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Regression Detector ResultsRun ID: d02cb026-134d-427b-8b22-99297cbbb777 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%. Fine details of change detection per experiment.
|
Regression Detector ResultsRun ID: cecb651f-0989-472b-b35c-0e7837142c37 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:
Fine details of change detection per experiment.
|
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Regression Detector ResultsRun ID: a4174c42-74cd-42c8-80bd-05ddc8b200c5 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%. Fine details of change detection per experiment.
|
Regression Detector ResultsRun ID: 85aeccc2-bf85-4408-98b8-de3fba2aad34 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%. Fine details of change detection per experiment.
|
Regression Detector ResultsRun ID: 15392eb8-f617-426d-8b79-d6247aeabe6c ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%. Fine details of change detection per experiment.
|
# [1.30.0](answerbook/vector@v1.29.2...v1.30.0) (2023-12-19) ### Bug Fixes * `aws_ec2_metadata` transform when using log namespacing (vectordotdev#17819) [4786743](answerbook/vector@4786743) - GitHub * **auth**: Vector does not put the Proxy-Authorization header on the wire (vectordotdev#17353) (vectordotdev#17363) [6705bdd](answerbook/vector@6705bdd) - GitHub * **buffers**: deadlock when seeking after entire write fails to be flushed (vectordotdev#17657) [37a662a](answerbook/vector@37a662a) - GitHub * **ci**: add missing env var (vectordotdev#17872) [4f67695](answerbook/vector@4f67695) - Jesse Szwedko * **ci**: add missing logic to mark required checks failed (vectordotdev#17543) [3b87e00](answerbook/vector@3b87e00) - GitHub * **ci**: change command to find baseline sha from issue comment trigger (vectordotdev#17622) [5791083](answerbook/vector@5791083) - GitHub * **ci**: checkout a greater depth in regression workflow (vectordotdev#17604) [baa04e5](answerbook/vector@baa04e5) - GitHub * **ci**: post failed status to PR and isolate branch checkout on comment trigger (vectordotdev#17544) [e2c0255](answerbook/vector@e2c0255) - GitHub * **ci**: reg workflow alt approach to getting baseline sha (vectordotdev#17645) [f1e1ae3](answerbook/vector@f1e1ae3) - GitHub * **ci**: use correct ID for Triage in Gardener Board (vectordotdev#17647) [2638cca](answerbook/vector@2638cca) - GitHub * **ci**: use correct secret for gardener board comment (vectordotdev#17605) [9395eba](answerbook/vector@9395eba) - GitHub * **config**: Fix preloading log_schema (vectordotdev#17759) [659e1e6](answerbook/vector@659e1e6) - GitHub * **databend sink**: use get for page request (vectordotdev#17373) [c7d7cf8](answerbook/vector@c7d7cf8) - GitHub * **datadog_agent source**: remove duplicate internal metrics emission (vectordotdev#17720) [48ec2e8](answerbook/vector@48ec2e8) - GitHub * **distribution**: Fix architecture detection for ARMv7 (vectordotdev#17484) [78fb469](answerbook/vector@78fb469) - GitHub * **docs**: fix copy-paste issue in component spec (vectordotdev#17616) [b400acc](answerbook/vector@b400acc) - GitHub * **file source**: Fix tailing problem when source number greater than 512 (vectordotdev#17717) [23a3e0e](answerbook/vector@23a3e0e) - GitHub * **fluent source**: fix ack message format (vectordotdev#17407) [d194992](answerbook/vector@d194992) - GitHub * **http_client source**: adapt int test to use breaking change of dep (vectordotdev#17583) [d7df520](answerbook/vector@d7df520) - GitHub * **http_client source**: remove utf8 lossy conversion (vectordotdev#17655) [59e2cbf](answerbook/vector@59e2cbf) - GitHub * **install.sh**: Correctly `shift` all parsed arguments (vectordotdev#17684) [f883575](answerbook/vector@f883575) - GitHub * **loki sink, observability**: Drop non-fatal template render errors to warnings (vectordotdev#17746) [4ebc3e1](answerbook/vector@4ebc3e1) - GitHub * **loki sink**: use json size of unencoded event (vectordotdev#17572) [25e7699](answerbook/vector@25e7699) - GitHub * **observability**: correct emitted metrics (vectordotdev#17562) [7a4f1f7](answerbook/vector@7a4f1f7) - GitHub * **observability**: issues with event_cache PR (vectordotdev#17768) [fdf02d9](answerbook/vector@fdf02d9) - GitHub * remap behavior for root types when using the `Vector` namespace (vectordotdev#17807) [c19938c](answerbook/vector@c19938c) - GitHub * **sinks**: Add missing component span for sink building (vectordotdev#17765) [219883e](answerbook/vector@219883e) - GitHub ### Chores * Add docker config to dependabot (vectordotdev#17696) [079d895](answerbook/vector@079d895) - GitHub * add sink prelude (vectordotdev#17595) [da939ca](answerbook/vector@da939ca) - GitHub * Add submodules to all checkouts (vectordotdev#17770) [7196622](answerbook/vector@7196622) - GitHub * **administration**: add domain label for vdev (vectordotdev#17748) [c35ebd1](answerbook/vector@c35ebd1) - GitHub * **aws_s3 sink**: Update metadata to match the editorial review for the schema. (vectordotdev#17475) [c1262cd](answerbook/vector@c1262cd) - GitHub * Bump version to 0.31.0 (vectordotdev#17466) [78bbfbc](answerbook/vector@78bbfbc) - GitHub * **ci**: Add apt retries to cross builds (vectordotdev#17683) [ab1169b](answerbook/vector@ab1169b) - GitHub * **ci**: Add schedule to component features workflow conditional check (vectordotdev#17816) [708b7f6](answerbook/vector@708b7f6) - GitHub * **ci**: Bump aws-actions/configure-aws-credentials from 2.0.0 to 2.1.0 (vectordotdev#17565) [8a741d5](answerbook/vector@8a741d5) - GitHub * **ci**: Bump aws-actions/configure-aws-credentials from 2.1.0 to 2.2.0 (vectordotdev#17697) [12bc4a7](answerbook/vector@12bc4a7) - GitHub * **ci**: Bump docker/build-push-action from 4.0.0 to 4.1.0 (vectordotdev#17656) [cb9a3a5](answerbook/vector@cb9a3a5) - GitHub * **ci**: Bump docker/build-push-action from 4.1.0 to 4.1.1 (vectordotdev#17687) [bce5e65](answerbook/vector@bce5e65) - GitHub * **ci**: Bump docker/metadata-action from 4.4.0 to 4.5.0 (vectordotdev#17624) [a54a12f](answerbook/vector@a54a12f) - GitHub * **ci**: Bump docker/metadata-action from 4.5.0 to 4.6.0 (vectordotdev#17686) [71273df](answerbook/vector@71273df) - GitHub * **ci**: Bump docker/setup-buildx-action from 2.5.0 to 2.6.0 (vectordotdev#17625) [15bc42a](answerbook/vector@15bc42a) - GitHub * **ci**: Bump docker/setup-buildx-action from 2.6.0 to 2.7.0 (vectordotdev#17685) [8006987](answerbook/vector@8006987) - GitHub * **ci**: Bump docker/setup-buildx-action from 2.7.0 to 2.8.0 (vectordotdev#17786) [dbdff9e](answerbook/vector@dbdff9e) - GitHub * **ci**: Bump docker/setup-qemu-action from 2.1.0 to 2.2.0 (vectordotdev#17623) [3005141](answerbook/vector@3005141) - GitHub * **ci**: bump myrotvorets/set-commit-status-action from 1.1.6 to 1.1.7 (vectordotdev#17460) [bca45eb](answerbook/vector@bca45eb) - GitHub * **ci**: Bump up OSX runners for release builds (vectordotdev#17823) [fe730ad](answerbook/vector@fe730ad) - GitHub * **ci**: bump xt0rted/pull-request-comment-branch from 1 to 2 (vectordotdev#17461) [c425006](answerbook/vector@c425006) - GitHub * **ci**: correctly validate comment author in k8s e2e job (vectordotdev#17818) [b8e3dbe](answerbook/vector@b8e3dbe) - GitHub * **ci**: Drop VRL license exceptions (vectordotdev#17529) [aa01452](answerbook/vector@aa01452) - GitHub * **ci**: fix a few logic bugs and more strict comment parsing (vectordotdev#17502) [bf372fd](answerbook/vector@bf372fd) - GitHub * **ci**: fix comment author validation (vectordotdev#17794) [75ae967](answerbook/vector@75ae967) - GitHub * **ci**: fix failure notify job conditional in publish workflow (vectordotdev#17468) [3699842](answerbook/vector@3699842) - GitHub * **ci**: fix gardener issues comment workflow (vectordotdev#17825) [47c3da1](answerbook/vector@47c3da1) - GitHub * **ci**: fix team membership action (vectordotdev#17791) [13c3c78](answerbook/vector@13c3c78) - GitHub * **ci**: int test yaml file detection (vectordotdev#17590) [fa8a553](answerbook/vector@fa8a553) - GitHub * **ci**: minor fixes to workflows post merge queue enabling (vectordotdev#17462) [9f6f6ec](answerbook/vector@9f6f6ec) - GitHub * **ci**: move component features check out of merge queue (vectordotdev#17773) [e6e776d](answerbook/vector@e6e776d) - GitHub * **ci**: Move most CI checks to merge queue (vectordotdev#17340) [060399a](answerbook/vector@060399a) - GitHub * **ci**: reduce runner sizing to 4 core and free tier (vectordotdev#17785) [53a575f](answerbook/vector@53a575f) - GitHub * **ci**: remove /ci-run-install comment trigger (vectordotdev#17803) [a3770d8](answerbook/vector@a3770d8) - GitHub * **ci**: Remove remaining Discord notification (vectordotdev#17805) [ed59f37](answerbook/vector@ed59f37) - GitHub * **ci**: Remove upload of config schema (vectordotdev#17740) [ff6a1b4](answerbook/vector@ff6a1b4) - GitHub * **ci**: Retry `make check-component-docs` check (vectordotdev#17718) [e8e7e04](answerbook/vector@e8e7e04) - GitHub * **ci**: revert fix gardener issues comment workflow (vectordotdev#17829) [ee10b8c](answerbook/vector@ee10b8c) - GitHub * **ci**: Set HOMEBREW_NO_INSTALL_FROM_API in CI (vectordotdev#17867) [00cc584](answerbook/vector@00cc584) - Jesse Szwedko * **ci**: temporarily disable comment_trigger workflow (vectordotdev#17480) [58d7f3d](answerbook/vector@58d7f3d) - GitHub * **ci**: temporarily disable flakey `aws_s3` integration test case `handles_errored_status` (vectordotdev#17455) [8e40b68](answerbook/vector@8e40b68) - GitHub * **ci**: update comment_trigger note about concurrency groups (vectordotdev#17491) [7699f4d](answerbook/vector@7699f4d) - GitHub * **ci**: Update publish workflow test Ubuntu versions (vectordotdev#17781) [20d62f1](answerbook/vector@20d62f1) - GitHub * **clickhouse sink**: refactor to new style (vectordotdev#17723) [77ac63c](answerbook/vector@77ac63c) - GitHub * **codecs**: consolidate enum types (vectordotdev#17688) [9c45394](answerbook/vector@9c45394) - GitHub * Codify flag naming including sentinel values (vectordotdev#17569) [134578d](answerbook/vector@134578d) - GitHub * Codify the use of abbreviate time units in config option names (vectordotdev#17582) [8823561](answerbook/vector@8823561) - GitHub * **config**: Convert top-level sinks enum to typetag (vectordotdev#17710) [9cd5404](answerbook/vector@9cd5404) - GitHub * **config**: Make config schema output ordered (vectordotdev#17694) [9606353](answerbook/vector@9606353) - GitHub * **config**: Update field labels for commonly used sources and transforms (vectordotdev#17517) [f523f70](answerbook/vector@f523f70) - GitHub * **config**: Update field labels for sinks (vectordotdev#17560) [e1ddd0e](answerbook/vector@e1ddd0e) - GitHub * **config**: Update field labels for the rest of the sources and transforms fields (vectordotdev#17564) [6e45477](answerbook/vector@6e45477) - GitHub * **datadog_archives sink**: Remove this component (vectordotdev#17749) [53f8bff](answerbook/vector@53f8bff) - GitHub * **datadog_metrics sink**: incrementally encode sketches (vectordotdev#17764) [3f6df61](answerbook/vector@3f6df61) - GitHub * **datadog_traces sink**: Add additional warning around APM stats for `peer.service` (vectordotdev#17733) [9a899c5](answerbook/vector@9a899c5) - GitHub * **deps, releasing**: Update to Alpine 3.18 (vectordotdev#17695) [2263756](answerbook/vector@2263756) - GitHub * **deps**: Bump async-graphql from 5.0.8 to 5.0.9 (vectordotdev#17486) [077a294](answerbook/vector@077a294) - GitHub * **deps**: Bump async-graphql from 5.0.9 to 5.0.10 (vectordotdev#17619) [2931542](answerbook/vector@2931542) - GitHub * **deps**: Bump async-graphql-warp from 5.0.8 to 5.0.9 (vectordotdev#17489) [ac81fc1](answerbook/vector@ac81fc1) - GitHub * **deps**: Bump async-graphql-warp from 5.0.9 to 5.0.10 (vectordotdev#17642) [b3885f6](answerbook/vector@b3885f6) - GitHub * **deps**: bump aws-sigv4 from 0.55.1 to 0.55.3 (vectordotdev#17481) [2ad5b47](answerbook/vector@2ad5b47) - GitHub * **deps**: bump base64 from 0.21.0 to 0.21.1 (vectordotdev#17451) [95cbba9](answerbook/vector@95cbba9) - GitHub * **deps**: Bump base64 from 0.21.1 to 0.21.2 (vectordotdev#17488) [f261781](answerbook/vector@f261781) - GitHub * **deps**: bump bstr from 1.4.0 to 1.5.0 (vectordotdev#17453) [7554d9c](answerbook/vector@7554d9c) - GitHub * **deps**: Bump cached from 0.43.0 to 0.44.0 (vectordotdev#17599) [7a55210](answerbook/vector@7a55210) - GitHub * **deps**: Bump chrono to 0.4.26 (vectordotdev#17537) [0dfa09c](answerbook/vector@0dfa09c) - GitHub * **deps**: Bump chrono-tz from 0.8.2 to 0.8.3 (vectordotdev#17789) [f79947c](answerbook/vector@f79947c) - GitHub * **deps**: bump clap_complete from 4.2.3 to 4.3.0 (vectordotdev#17447) [05bf262](answerbook/vector@05bf262) - GitHub * **deps**: Bump clap_complete from 4.3.0 to 4.3.1 (vectordotdev#17586) [8549809](answerbook/vector@8549809) - GitHub * **deps**: bump criterion from 0.4.0 to 0.5.0 (vectordotdev#17477) [84f0ada](answerbook/vector@84f0ada) - GitHub * **deps**: Bump criterion from 0.5.0 to 0.5.1 (vectordotdev#17500) [da7bc95](answerbook/vector@da7bc95) - GitHub * **deps**: Bump crossbeam-utils from 0.8.15 to 0.8.16 (vectordotdev#17674) [714ccf8](answerbook/vector@714ccf8) - GitHub * **deps**: Bump csv from 1.2.1 to 1.2.2 (vectordotdev#17555) [bcc5b6c](answerbook/vector@bcc5b6c) - GitHub * **deps**: bump data-encoding from 2.3.3 to 2.4.0 (vectordotdev#17452) [9aaf864](answerbook/vector@9aaf864) - GitHub * **deps**: Bump getrandom from 0.2.9 to 0.2.10 (vectordotdev#17613) [bd880f5](answerbook/vector@bd880f5) - GitHub * **deps**: Bump gloo-utils from 0.1.6 to 0.1.7 (vectordotdev#17707) [53e1785](answerbook/vector@53e1785) - GitHub * **deps**: Bump graphql_client from 0.12.0 to 0.13.0 (vectordotdev#17541) [ecb707a](answerbook/vector@ecb707a) - GitHub * **deps**: Bump h2 from 0.3.19 to 0.3.20 (vectordotdev#17767) [ac3bc72](answerbook/vector@ac3bc72) - GitHub * **deps**: Bump hashbrown from 0.13.2 to 0.14.0 (vectordotdev#17609) [154e393](answerbook/vector@154e393) - GitHub * **deps**: Bump hyper from 0.14.26 to 0.14.27 (vectordotdev#17766) [a2a3609](answerbook/vector@a2a3609) - GitHub * **deps**: Bump indexmap from 1.9.3 to 2.0.0 (vectordotdev#17755) [248ccb8](answerbook/vector@248ccb8) - GitHub * **deps**: Bump indicatif from 0.17.3 to 0.17.4 (vectordotdev#17532) [1565985](answerbook/vector@1565985) - GitHub * **deps**: Bump indicatif from 0.17.4 to 0.17.5 (vectordotdev#17597) [a164952](answerbook/vector@a164952) - GitHub * **deps**: Bump infer from 0.13.0 to 0.14.0 (vectordotdev#17737) [326ad08](answerbook/vector@326ad08) - GitHub * **deps**: Bump itertools from 0.10.5 to 0.11.0 (vectordotdev#17736) [6e1878b](answerbook/vector@6e1878b) - GitHub * **deps**: Bump lalrpop to 0.19.12 (vectordotdev#17457) [1f54415](answerbook/vector@1f54415) - GitHub * **deps**: bump lapin from 2.1.1 to 2.1.2 (vectordotdev#17439) [a8b7899](answerbook/vector@a8b7899) - GitHub * **deps**: bump lapin from 2.1.2 to 2.2.0 (vectordotdev#17443) [b639422](answerbook/vector@b639422) - GitHub * **deps**: bump lapin from 2.2.0 to 2.2.1 (vectordotdev#17448) [618379a](answerbook/vector@618379a) - GitHub * **deps**: Bump libc from 0.2.144 to 0.2.146 (vectordotdev#17615) [10cfd0a](answerbook/vector@10cfd0a) - GitHub * **deps**: Bump libc from 0.2.146 to 0.2.147 (vectordotdev#17753) [1a75ec6](answerbook/vector@1a75ec6) - GitHub * **deps**: Bump log from 0.4.17 to 0.4.18 (vectordotdev#17526) [5a2fea1](answerbook/vector@5a2fea1) - GitHub * **deps**: Bump log from 0.4.18 to 0.4.19 (vectordotdev#17662) [e1b3357](answerbook/vector@e1b3357) - GitHub * **deps**: Bump lru from 0.10.0 to 0.10.1 (vectordotdev#17810) [96e68f7](answerbook/vector@96e68f7) - GitHub * **deps**: bump memmap2 from 0.6.1 to 0.6.2 (vectordotdev#17482) [79f7dfb](answerbook/vector@79f7dfb) - GitHub * **deps**: Bump memmap2 from 0.6.2 to 0.7.0 (vectordotdev#17641) [593ea1b](answerbook/vector@593ea1b) - GitHub * **deps**: Bump memmap2 from 0.7.0 to 0.7.1 (vectordotdev#17752) [4236e32](answerbook/vector@4236e32) - GitHub * **deps**: Bump mock_instant from 0.3.0 to 0.3.1 (vectordotdev#17574) [1c1beb8](answerbook/vector@1c1beb8) - GitHub * **deps**: Bump mongodb from 2.5.0 to 2.6.0 (vectordotdev#17726) [c96e3be](answerbook/vector@c96e3be) - GitHub * **deps**: Bump notify from 6.0.0 to 6.0.1 (vectordotdev#17700) [cd6d154](answerbook/vector@cd6d154) - GitHub * **deps**: Bump once_cell from 1.17.1 to 1.17.2 (vectordotdev#17531) [8e113ad](answerbook/vector@8e113ad) - GitHub * **deps**: Bump once_cell from 1.17.2 to 1.18.0 (vectordotdev#17596) [dc6bef2](answerbook/vector@dc6bef2) - GitHub * **deps**: bump opendal from 0.34.0 to 0.35.0 (vectordotdev#17471) [ebf958b](answerbook/vector@ebf958b) - GitHub * **deps**: Bump opendal from 0.35.0 to 0.36.0 (vectordotdev#17540) [dbd7151](answerbook/vector@dbd7151) - GitHub * **deps**: Bump opendal from 0.36.0 to 0.37.0 (vectordotdev#17614) [b5bd85f](answerbook/vector@b5bd85f) - GitHub * **deps**: Bump opendal from 0.37.0 to 0.38.0 (vectordotdev#17777) [ec4785a](answerbook/vector@ec4785a) - GitHub * **deps**: Bump openssl from 0.10.52 to 0.10.53 (vectordotdev#17534) [078de66](answerbook/vector@078de66) - GitHub * **deps**: Bump openssl from 0.10.53 to 0.10.54 (vectordotdev#17573) [4af5e6d](answerbook/vector@4af5e6d) - GitHub * **deps**: Bump openssl from 0.10.54 to 0.10.55 (vectordotdev#17716) [dd2527d](answerbook/vector@dd2527d) - GitHub * **deps**: Bump percent-encoding from 2.2.0 to 2.3.0 (vectordotdev#17602) [8e04259](answerbook/vector@8e04259) - GitHub * **deps**: Bump pin-project from 1.1.0 to 1.1.1 (vectordotdev#17806) [0b32626](answerbook/vector@0b32626) - GitHub * **deps**: Bump PR limit for Dependabot to 100 (vectordotdev#17459) [85703e7](answerbook/vector@85703e7) - GitHub * **deps**: bump proc-macro2 from 1.0.57 to 1.0.58 (vectordotdev#17426) [ae656c7](answerbook/vector@ae656c7) - GitHub * **deps**: Bump proc-macro2 from 1.0.58 to 1.0.59 (vectordotdev#17495) [4ce3278](answerbook/vector@4ce3278) - GitHub * **deps**: Bump proc-macro2 from 1.0.59 to 1.0.60 (vectordotdev#17643) [f20eb2f](answerbook/vector@f20eb2f) - GitHub * **deps**: Bump proc-macro2 from 1.0.60 to 1.0.63 (vectordotdev#17757) [63ba2a9](answerbook/vector@63ba2a9) - GitHub * **deps**: bump proptest from 1.1.0 to 1.2.0 (vectordotdev#17476) [9235fc2](answerbook/vector@9235fc2) - GitHub * **deps**: bump pulsar from 5.1.1 to 6.0.0 (vectordotdev#17587) [3395cfd](answerbook/vector@3395cfd) - GitHub * **deps**: Bump pulsar from 6.0.0 to 6.0.1 (vectordotdev#17673) [8d98bb8](answerbook/vector@8d98bb8) - GitHub * **deps**: Bump quanta from 0.11.0 to 0.11.1 (vectordotdev#17524) [2388c2f](answerbook/vector@2388c2f) - GitHub * **deps**: Bump quote from 1.0.27 to 1.0.28 (vectordotdev#17496) [cc30746](answerbook/vector@cc30746) - GitHub * **deps**: Bump quote from 1.0.28 to 1.0.29 (vectordotdev#17798) [cba983e](answerbook/vector@cba983e) - GitHub * **deps**: Bump quote from 1.0.28 to 1.0.29 (vectordotdev#17815) [bf9828d](answerbook/vector@bf9828d) - GitHub * **deps**: bump rdkafka from 0.30.0 to 0.31.0 (vectordotdev#17428) [e7fa8d3](answerbook/vector@e7fa8d3) - GitHub * **deps**: Bump rdkafka from 0.31.0 to 0.32.2 (vectordotdev#17664) [ac68a7b](answerbook/vector@ac68a7b) - GitHub * **deps**: bump regex from 1.8.1 to 1.8.2 (vectordotdev#17469) [897e45d](answerbook/vector@897e45d) - GitHub * **deps**: bump regex from 1.8.2 to 1.8.3 (vectordotdev#17494) [5d90cff](answerbook/vector@5d90cff) - GitHub * **deps**: Bump regex from 1.8.3 to 1.8.4 (vectordotdev#17601) [657758d](answerbook/vector@657758d) - GitHub * **deps**: bump reqwest from 0.11.17 to 0.11.18 (vectordotdev#17420) [2ed8ec7](answerbook/vector@2ed8ec7) - GitHub * **deps**: bump security-framework from 2.9.0 to 2.9.1 (vectordotdev#17441) [ac0c7e8](answerbook/vector@ac0c7e8) - GitHub * **deps**: Bump serde from 1.0.163 to 1.0.164 (vectordotdev#17632) [e35150e](answerbook/vector@e35150e) - GitHub * **deps**: Bump serde_json from 1.0.96 to 1.0.97 (vectordotdev#17701) [25131ef](answerbook/vector@25131ef) - GitHub * **deps**: Bump serde_json from 1.0.97 to 1.0.99 (vectordotdev#17754) [e07158c](answerbook/vector@e07158c) - GitHub * **deps**: Bump serde_yaml from 0.9.21 to 0.9.22 (vectordotdev#17756) [e164b36](answerbook/vector@e164b36) - GitHub * **deps**: Bump sha2 from 0.10.6 to 0.10.7 (vectordotdev#17698) [d122d32](answerbook/vector@d122d32) - GitHub * **deps**: Bump tempfile from 3.5.0 to 3.6.0 (vectordotdev#17617) [c55c9ec](answerbook/vector@c55c9ec) - GitHub * **deps**: Bump tokio from 1.28.1 to 1.28.2 (vectordotdev#17525) [cc703da](answerbook/vector@cc703da) - GitHub * **deps**: Bump tokio from 1.28.2 to 1.29.0 (vectordotdev#17776) [e26e8b8](answerbook/vector@e26e8b8) - GitHub * **deps**: bump toml from 0.7.3 to 0.7.4 (vectordotdev#17440) [91ba052](answerbook/vector@91ba052) - GitHub * **deps**: Bump toml from 0.7.4 to 0.7.5 (vectordotdev#17751) [35c4581](answerbook/vector@35c4581) - GitHub * **deps**: Bump tower-http from 0.4.0 to 0.4.1 (vectordotdev#17711) [e5e6b96](answerbook/vector@e5e6b96) - GitHub * **deps**: Bump url from 2.3.1 to 2.4.0 (vectordotdev#17608) [d956092](answerbook/vector@d956092) - GitHub * **deps**: Bump uuid from 1.3.3 to 1.3.4 (vectordotdev#17682) [c97d619](answerbook/vector@c97d619) - GitHub * **deps**: Bump uuid from 1.3.4 to 1.4.0 (vectordotdev#17775) [935babf](answerbook/vector@935babf) - GitHub * **deps**: Bump wasm-bindgen from 0.2.86 to 0.2.87 (vectordotdev#17672) [19c4d4f](answerbook/vector@19c4d4f) - GitHub * **deps**: Bump wiremock from 0.5.18 to 0.5.19 (vectordotdev#17618) [460bbc7](answerbook/vector@460bbc7) - GitHub * **deps**: Bump xml-rs from 0.8.4 to 0.8.14 (vectordotdev#17607) [a932489](answerbook/vector@a932489) - GitHub * **deps**: Drop use of `hashlink` crate (vectordotdev#17678) [41ee394](answerbook/vector@41ee394) - GitHub * **deps**: Export more common bits for components (vectordotdev#17788) [062224b](answerbook/vector@062224b) - GitHub * **deps**: Update fs_extra to 1.3.0 (vectordotdev#17458) [299fd6a](answerbook/vector@299fd6a) - GitHub * **deps**: Upgrade Ruby version to 3.1.4 (vectordotdev#17722) [ddebde9](answerbook/vector@ddebde9) - GitHub * **deps**: Upgrade rust to 1.70.0 (vectordotdev#17585) [6c48565](answerbook/vector@6c48565) - GitHub * **dev**: Add @dsmith3197 to CODEOWNERS (vectordotdev#17729) [a08443c](answerbook/vector@a08443c) - GitHub * **docs**: Add info about Vector Operator to Kubernetes instalation page (vectordotdev#17432) [54d9c99](answerbook/vector@54d9c99) - GitHub * **docs**: add instructions for regenerating component docs and licenses (vectordotdev#17828) [93ef6c3](answerbook/vector@93ef6c3) - GitHub * **docs**: Add Log Namespacing docs (vectordotdev#16571) [7d098e4](answerbook/vector@7d098e4) - GitHub * **docs**: add note about const strings (vectordotdev#17774) [d7bc531](answerbook/vector@d7bc531) - GitHub * **docs**: Clarify `bytes` framing for streams (vectordotdev#17745) [7d10fc9](answerbook/vector@7d10fc9) - GitHub * **docs**: Clarify when component received and sent bytes events should be emitted (vectordotdev#17464) [547783d](answerbook/vector@547783d) - GitHub * **docs**: Move CONTRIBUTING.md to top-level (vectordotdev#17744) [7a0dec1](answerbook/vector@7a0dec1) - GitHub * Download submodules in the CI checkouts (vectordotdev#17760) [5417a06](answerbook/vector@5417a06) - GitHub * Dropped error field from StreamClosed Error (vectordotdev#17693) [ee480cd](answerbook/vector@ee480cd) - GitHub * **enrichment**: avoid importing vector-common in enrichment module (vectordotdev#17653) [45a28f8](answerbook/vector@45a28f8) - GitHub * **enterprise**: Extend library functionality for secret scanning (vectordotdev#17483) [541bb00](answerbook/vector@541bb00) - GitHub * **external docs**: fix reference to supported aarch64 architecture (vectordotdev#17553) [247bb80](answerbook/vector@247bb80) - GitHub * **external docs**: update fluentd link (vectordotdev#17436) [187f142](answerbook/vector@187f142) - GitHub * Fix publish workflow for older OS images (vectordotdev#17787) [ab39c6a](answerbook/vector@ab39c6a) - GitHub * **flush on shutdown**: validate s3 sink flushes (vectordotdev#17667) [c21f892](answerbook/vector@c21f892) - GitHub * **kubernetes_logs source**: Add warning about Windows support (vectordotdev#17762) [a53c7a2](answerbook/vector@a53c7a2) - GitHub * **kubernetes**: Bump k8s manifests to 0.22.0 (vectordotdev#17467) [f547871](answerbook/vector@f547871) - GitHub * **observability**: emit `component_sent` events by `source` and `service` (vectordotdev#17549) [dcf7f9a](answerbook/vector@dcf7f9a) - GitHub * **observability**: ensure `sent_event` and `received_event` metrics are estimated json size (vectordotdev#17465) [3b2a2be](answerbook/vector@3b2a2be) - GitHub * **observability**: Have `tower_limit` use configured log level (vectordotdev#17715) [08099a8](answerbook/vector@08099a8) - GitHub * **observability**: remove deprecated internal metrics + massive cleanup to vector top and graphql API (vectordotdev#17516) [98c54ad](answerbook/vector@98c54ad) - GitHub * **observability**: remove more deprecated internal metrics (vectordotdev#17542) [b0ed167](answerbook/vector@b0ed167) - GitHub * **observability**: set source fields to mean service (vectordotdev#17470) [670bdea](answerbook/vector@670bdea) - GitHub * **releasing**: Prepare v0.30.0 release [af2b2af](answerbook/vector@af2b2af) - Jesse Szwedko * **releasing**: Prepare v0.31.0 release [0f13b22](answerbook/vector@0f13b22) - Jesse Szwedko * remove custom async sleep impl (vectordotdev#17493) [b28d915](answerbook/vector@b28d915) - GitHub * Remove links to roadmap (vectordotdev#17554) [349c718](answerbook/vector@349c718) - GitHub * Revert all submodule introductions to fix CI (vectordotdev#17800) [d8d57e5](answerbook/vector@d8d57e5) - GitHub * RFC for Data Volume Insights (vectordotdev#17322) [a551f33](answerbook/vector@a551f33) - GitHub * **sinks**: Drop the custom `SinkContext::default` implementation (vectordotdev#17804) [e66e285](answerbook/vector@e66e285) - GitHub * **sinks**: mark VectorSink::from_event_sink as deprecated (vectordotdev#17649) [0dc450f](answerbook/vector@0dc450f) - GitHub * **statsd sink**: refactor `statsd` sink to stream-based style (vectordotdev#16199) [2a76cac](answerbook/vector@2a76cac) - GitHub * update `vrl` to `0.4.0` (vectordotdev#17378) [426d660](answerbook/vector@426d660) - GitHub * Update the NOTICE file (vectordotdev#17430) [9a44e6e](answerbook/vector@9a44e6e) - GitHub * Upgrade aws-smithy and aws-sdk crates (vectordotdev#17731) [6a6b42b](answerbook/vector@6a6b42b) - GitHub * **website**: Fix upgrade guide dates [80de738](answerbook/vector@80de738) - Jesse Szwedko ### Features * add metadata support to `set_semantic_meaning` (vectordotdev#17730) [44be378](answerbook/vector@44be378) - GitHub * **codecs**: add lossy option to `gelf`, `native_json`, and `syslog` deserializers (vectordotdev#17680) [2dfa850](answerbook/vector@2dfa850) - GitHub * **codecs**: Add lossy option to JSON deserializer (vectordotdev#17628) [bf7d796](answerbook/vector@bf7d796) - GitHub * **configurable shutdown duration**: make shutdown duration configurable (vectordotdev#17479) [23ed0e3](answerbook/vector@23ed0e3) - GitHub * **error code when shutdown fails**: set exit flag to non-zero when shutdown times out (vectordotdev#17676) [cc52c0e](answerbook/vector@cc52c0e) - GitHub * **internal telemetry at shutdown**: close internal sources after external ones (vectordotdev#17741) [812929b](answerbook/vector@812929b) - GitHub * **journald source**: add journal_namespace option (vectordotdev#17648) [a324a07](answerbook/vector@a324a07) - GitHub * **kinesis sinks**: implement full retry of partial failures in firehose/streams (vectordotdev#17535) [bebac21](answerbook/vector@bebac21) - GitHub * **prometheus**: add more compression algorithms to Prometheus Remote Write (vectordotdev#17334) [380d7ad](answerbook/vector@380d7ad) - GitHub * track runtime schema definitions for log events (vectordotdev#17692) [6eecda5](answerbook/vector@6eecda5) - GitHub ### Miscellaneous * Merge pull request vectordotdev#371 from answerbook/feature/LOG-18200 [491ea31](answerbook/vector@491ea31) - GitHub [LOG-18200](https://logdna.atlassian.net/browse/LOG-18200) * Merge pull request vectordotdev#374 from answerbook/dominic/LOG-18535-temp-revert [df87df7](answerbook/vector@df87df7) - GitHub [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535) [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535) * Revert "Merge pull request vectordotdev#370 from answerbook/dominic/LOG-18535" [73a8b28](answerbook/vector@73a8b28) - dominic-mcallister-logdna [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535) * Merge branch 'master' into feature/LOG-18200 [fcd21f3](answerbook/vector@fcd21f3) - Darin Spivey [LOG-18200](https://logdna.atlassian.net/browse/LOG-18200) * Merge tag 'v0.31.0' into upstream-0.31.0 [9443fb1](answerbook/vector@9443fb1) - Darin Spivey * Update VRL to `0.5.0` (vectordotdev#17793) [671aa79](answerbook/vector@671aa79) - GitHub * fix `demo_logs` metadata source name (vectordotdev#17689) [83af7ea](answerbook/vector@83af7ea) - GitHub * enhancement(s3 source) Add minimal support to unwrap an S3-SQS event from an SNS event (vectordotdev#17352) [7a7bc9a](answerbook/vector@7a7bc9a) - GitHub * Additional notes on proposing new integrations (vectordotdev#17658) [2ad964d](answerbook/vector@2ad964d) - GitHub * **ci**: reduce billable time of Test Suite (vectordotdev#17714) [bc69255](answerbook/vector@bc69255) - GitHub * **ci**: refactor logic for int test file path changes detection (vectordotdev#17725) [92a36e0](answerbook/vector@92a36e0) - GitHub * **compression**: zstd compression support (vectordotdev#17371) [ced219e](answerbook/vector@ced219e) - GitHub * **dev**: move blocked/waiting gardener issues to triage on comment (vectordotdev#17588) [6b34868](answerbook/vector@6b34868) - GitHub * explain how to run tests locally (vectordotdev#17783) [3b67a80](answerbook/vector@3b67a80) - GitHub * remove aggregator beta warning (vectordotdev#17750) [94e3f15](answerbook/vector@94e3f15) - GitHub
closes #16732
In order for sinks to use semantic meaning, they need a mapping of meanings to fields. This is included in the schema definition of events, but the exact definition that needs to be used depends on the path the event took to get to the sink. The schema definition of an event is tracked at runtime so this can be determined.
A
parent_idwas added to event metadata to track the previous component that an event came from, which lets the topology select the correct schema definition to attach to events.For sources, there is only one definition that can be attached (for each port). This is automatically attached in the topology layer (after an event is emitted by a source), so there is no additional work in each source to support this.
For transforms, it's slightly more complicated. The schema definition depends on both the output port and the component the event came from. A map is generated at Vector startup, and the correct definition is obtained from that at runtime. This also happens in the topology layer so transforms don't need to worry about this.
Previously the
remaptransform had custom code to support runtime schema definitions (for the VRL meaning functions). This was removed since it's now handled automatically.The
reduceandluatransforms are special cases since there is no clear "path" that an event takes through the topology, since multiple events can be merged (from different inputs) inreduce. Forlua, output events may not be related to input events at all. In these cases the schema definition map will have the same value for all inputs (they are all merged). The topology will then arbitrarily pick one (since they are all the same).