Conversation
* chore(dev): update environment image * ran cargo fmt
* chore(ci): add timeout to component features job * Use bootstrap * Update .github/workflows/component_features.yml --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
* docs(external): add guide and example for AWS Secrets Manager secret backend * convert toml to yaml * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Update website/content/en/docs/reference/configuration/examples/aws-secrets-manager-integration.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/docs/reference/configuration/examples/aws-secrets-manager-integration.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/docs/reference/configuration/examples/aws-secrets-manager-integration.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Update website/content/en/docs/reference/configuration/examples/aws-secrets-manager-integration.md Co-authored-by: May Lee <may.lee@datadoghq.com> * Update website/content/en/guides/level-up/aws-secrets-manager.md Co-authored-by: May Lee <may.lee@datadoghq.com> * move docs to new aws dedicated folder * fill in aws index file * add mention for minimum vector version rquired * remove duplicate content * rename file * remove redundant info * fix spellcheck * Apply suggestions from code review * Apply missed docs suggestions * Table spacing formatting --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: May Lee <may.lee@datadoghq.com> Co-authored-by: Thomas <thomasqueirozb@gmail.com>
…_unfinalized (#23812) * chore(tests): increase timeouts in file_start_position_server_restart_unfinalized * Remove await * Fix warnings
* chore(ci): run IT suite once * fix workflow_dispatch * add pull_request trigger
…LOG (#23804) * chore(dev): consolidate usage of VECTOR_LOG in tests and remove TEST_LOG * Use VECTOR_LOG in workflow files
* feat(dev): enable colors when running in nextest * Fix and simplify colors * Remove CI check * Make test fail * Set CARGO_TERM_COLOR always in test.yml * Revert "Make test fail" This reverts commit bf6acf2b1f1241e1ed76be03da3881329b20084b.
* chore(website): improve indexing for memory table docs * add table
…3671) * feat(file sink): add options to truncate files in some conditions Closes: #22308 * Add changelog entry * Update website/cue/reference/components/sinks/generated/file.cue Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Fix docs spelling in code * make fmt * Fix issues from review * Fix spelling issues * Update changelog.d/23671_file_sink_truncate.feature.md * Update benches/files.rs * Update changelog.d/23671_file_sink_truncate.feature.md * Cargo fmt --------- Co-authored-by: Esther Kim <esther.kim@datadoghq.com> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Thomas <thomasqueirozb@gmail.com>
chore(ci): extract homebrew publishing into each own workflow
* cargo vdev build manifests * bump release num * cargo update -p vrl * chore(releasing): prepare v0.50.0 release (#23808) * chore(releasing): Pinned VRL version to 0.27.0 * chore(releasing): Generated release CUE file * chore(releasing): Updated website/cue/reference/administration/interfaces/kubectl.cue vector version to 0.50.0 * chore(releasing): Updated distribution/install.sh vector version to 0.50.0 * chore(releasing): Add 0.50.0 to versions.cue * chore(releasing): Created release md file * bump alpine to 3.22 * replace revert with fix - need to support this properly in the future * allow new words/names * fmt fixes * Try to remove new authors from allow.txt * release description and highlights TODO * highlights, breaking changes, fixes * typo * minor improvements * Update website/cue/reference/releases/0.50.0.cue Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * Update website/cue/reference/releases/0.50.0.cue Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(website): improve indexing for memory table docs (#23827) * chore(website): improve indexing for memory table docs * add table * add highlight for OTLP * fix date * add known issue note --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
* chore(ci): homebrew workflow improvements * fix auth to other repo * please clippy
…s (#23500) * feat(config): log changed global fields when reload is rejected * change diff return to Result * feat: add changelog * rename changelog * fix comparison * fix cargo fmt * feat: add metric vector_config_reload_rejected * modify changelog * feat: add metric vector_config_reloaded * add changelog * Apply suggestions from code review * modify feature md * add metric descriptions to cue * Update website/cue/reference/components/sources/internal_metrics.cue * Fix spelling * fix clippy * fix clippy * propogate compute global diff error * clippy fixes --------- Co-authored-by: suika <suika@XG-Mac14.kirakira> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Thomas <thomasqueirozb@gmail.com>
* fix(dev): add .md authors pattern * remove unused words
* chore(config): Allow loading config w/ secrets from str * Remove unecessary call to block_on - All that is needed is a call to await - Deadlock may be experienced if up the call chain there was a previous call to block_on * Remove pub from newly introduced load_builder_from_input* methods * Remove redundant else branch
…t table (#23815) * feat(enrichment tables): add expired items output to memory enrichment table Adds a new output to memory enrichment table source that exports items as they are expired out of this cache. Related: #23784 * Add changelog entry * Move `MemorySourceConfig` to `config.rs` * Add `MemoryEntryPair` struct * Extract out select branches into separate functions * Add output section * Log errors for failed expired items export * Add `internal_log_rate_limit` to expired items export error message * Add missing comma --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
* chore(e2e): use shared volume and remove hacky workaround for otel logs e2e test * ran cargo fmt * use dedicated folder * remove unused var * clean output dir and also ensure volumes before start * nit * clippy * ensure volumes are there when required
* add vrl pop function documentation * update description to reflect that the value array is modified in place * fix docs --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…nfig spec (#23866) * docs(config): Document best-practice of not ending with _config in config spec Follow-up to vectordotdev/vector#23864 Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com> * Update docs/specs/configuration.md Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> --------- Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
* fix(ci): windows rustup stable not installed by default * Use prepare.sh for windows * Fix windows os version * Only install nextest when RELEASE_BUILDER is unset --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…864) chore: Remove _config suffix from truncate_config This suffix is inconsistent with other component configuration option naming which lacks it. Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
chore(docs): improvements
* fix(azure_blob sink): use correct reqwest feature * Add changelog * Fix changelog * Rephrase changelog
* chore(ci): cache vdev * fix cache key * vdev dep, implies rust dep
* chore(dev): check modified files only for style * fix git command
* chore(ci): run K8s e2e test suite only on MQ * Don't build on pull_request event * Also skip compute test plan
… (#24080) * fix: prevent utilization metric loss on configuration reload On configuration reload, Vector starts spamming debug logs: ``` vector::utilization: Couldn't send utilization start wait message. ``` This is caused by building the config diff, which just creates a new utilization emitter, registers all the changed components to that new emitter and then just drops it at the end, because one already exists in the running topology. This changes it so that the existing emitter is reused on reloads, ensuring that reloaded components still send their utilization messages. Related: vectordotdev/vector#24060 (comment) * Add changelog entry * Remove duplicate `Arc` in `utilization.rs` * Split out `UtilizationRegistry` from `UtilizationEmitter` for easier registry reuse on rebuild * Remove components from utilization registry when they are removed from topology --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
* naively copy e2e otel-logs to otel-metrics * try to fix * Remove vector_default.yaml custom mapping * Test all metric types * Update vrl to main after fix was merged * Add additional attributes and verify them in tests * Format + remove Summary support * simplify assert_metric_attributes logic * Revert Cargo.toml changes * Add ci e2e trigger * Make test more concise * reuse common Dockerfiles * Use common functions in opentelemetry-logs * Hard code vector_otlp.yaml * Fix CI trigger * Add service to integration.yml matrix * Add and use generic assert_service_name_with * ranem data_type -> test_type --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
chore(ci): parallelize e2e tests
* fix(instrument): Buffer counter underflowed (#23872) * Update code to be more clearly with to let Co-authored-by: Bruce Guenter <bruce@untroubled.org> * format fix to meet format&clippy --------- Co-authored-by: Bruce Guenter <bruce@untroubled.org> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
* don't pull cargo bin dir when cargo-cache is enabled * Add specific cache layer to prepare.sh * Temporarily add github.event_name == 'pull_request' to integration.yml * Revert "Temporarily add github.event_name == 'pull_request' to integration.yml" This reverts commit 55a675c504d38c9de7a5b700e73da44e80af191f. * Hash all files under scripts/environment
…4119) * bump `avro-rs` crate to improve encoding error Signed-off-by: titaneric <chenyihuang001@gmail.com> * add changelog Signed-off-by: titaneric <chenyihuang001@gmail.com> * add newline Signed-off-by: titaneric <chenyihuang001@gmail.com> * build licenses Signed-off-by: titaneric <chenyihuang001@gmail.com> --------- Signed-off-by: titaneric <chenyihuang001@gmail.com>
* chore(releasing): Pinned VRL version to 0.28.0 * chore(releasing): Generated release CUE file * chore(releasing): Updated website/cue/reference/administration/interfaces/kubectl.cue vector version to 0.51.0 * chore(releasing): Updated distribution/install.sh vector version to 0.51.0 * chore(releasing): Add 0.51.0 to versions.cue * chore(releasing): Created release md file * fix/improve wording * Apply suggestions from code review * Add release highlights * Add telemetry highlight * Add note to otlp support page * Fix whitespace * Make file source breaking version more cleaer * Merge memory enrichment table highlights * Add upgrade guide * Add config watcher fix to highlights * Rename file to match release date * Update website/content/en/highlights/2025-11-04-0-51-0-upgrade-guide.md * Fix author spellcheck * Specify debug and trace for component_id logs * cue fmt * Remove backtick lang * Update multi-line config block upgrade guide * Remove config watcher highlight * Add backticks to debug and trace * Merge highlights * Revert "Remove backtick lang" This reverts commit 11ff870d2ab66337a9c6a5b7b8a9d43a8456bfd1. * Move VRL breaking change to the bottom * Fix markdown check * Update breaking changes wording * Add VRL info block
* chore(ci): reorganize integration test files (#24108) * chore(ci): reorganize integration test files * revert nats data move * fix paths to data * fix more paths * amqp fix * Fix logstash int tests shared files paths * Rename scripts/integration -> tests/integration * scripts/integration -> tests/integration in source files * Fix scripts->tests * Fix gcp paths * Fix http-client pem path * move nats data --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(vdev): move all utils in a new utils folder (#24143) * chore(vdev): move all utils in a new utils folder * move remaining files to utils dir * move remaining files to utils dir * fmt * chore(vdev): make modules visible to rustfmt (#24162) * Remove mod inside cli_subcommands macro * cargo fmt * chore(vdev): apply vdev rust check fixes * Link to rustfmt issue * fix(vdev): release prepare vrl version pinning (#24158) * Add --dry-run to release prepare * Add error handling and checks to pin_vrl_version * Add wrapper to toml * Remove wrapper, parse as Table instead * Fix vrl pinning logic * Enable preserve_order feature in toml crate * Use dependency instead of whole toml * Fix dry run docs * Fix dry run wording * refactor to use toml_edit * Add update_vrl_to_version to add unit test * Use indoc in prepare.rs * Remove preserve_order feature * chore(deps): update VRL to add missing stdlib fns from 0.28 (#24178) * chore(ci): temporarily remove homebrew publish step from publish workflow (#24185) This temporarily removes the publish-homebrew job from the publish workflow to address issue #24139. This is step 1 of the plan to fix the homebrew publishing process. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * fix(blackhole sink): disable rate limiting for periodic stats messages (#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in #24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes #24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com> * chore(internal logs): Disable rate limiting for critical internal error logs (#24192) * Improve debug log in aws_ecs_metrics * Add internal_log_rate_limit = false to config reload errors * Disable rate limiting for lua build failure * Disable rate limiting for amqp build failure * Add internal_log_rate_limit = false to important failures during reload * Disable log rate limit for repeated events in process.rs * Update src/config/watcher.rs * Fix message format * fix(tracing): prevent panic for traces without standard fields (#24191) * fix(tracing): prevent panic for traces without std fields * forbid unwrap in this lib * clippy fix * improve events_with_custom_fields_no_message_dont_panic, advance to next window * chore(tracing): do not rate limit utlization report (#24202) * chore(internal metrics): move config_reload_* metrics to VectorReload* (#24203) * chore(internal metrics): move config_reload_* metrics to VectorReloaded and VectorReloadedError * update topology_doesnt_reload_new_data_dir test * chore(dev): cargo fmt * strengthen tests, since now we have a better interface * chore(releasing): Add 0.51.0 known issues (#24211) * Format fixes * Add 0.51.0 known issues * Add dots * Reword log panic known issue * Reword VRL missing fns * Backtick versions * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Fix cue docs fmt --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Bump vector version to 0.51.1 * Bump VRL to 0.28.1 * Generate 0.51.1 cue * Add vrl_changelog * Remove whitespace * chore(releasing): Updated distribution/install.sh vector version to 0.51.1 * chore(releasing): Add 0.51.1 to versions.cue * chore(releasing): Created release md file * Add description * Reorder items and fix format * Finally fix styling * Bump date --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> chore(releasing): pull in `internal_logs` fix into 0.51.1 (#24225) * fix(internal_logs source): remove rate limit (#24218) * fix(internal_logs source): remove rate limit * add changelog * Debug commit * Fix validated - Revert "Debug commit" This reverts commit c7b9ec9c74ef31d235df08edc4681d7336c9cc1e. * add unit test * fix check-events * Pull in internal_logs source fix * Add internal_logs known issue * Bump date * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
v0.51.1 ref: LOG-23166
Update mold, rockdb and Rust.
new changes upstream to log supression and rate-limiting requires incremental changes to ensure that our internal systems still see and can take action on messages we expect. logs were not only being swallowed, but also two separate logs in the same component were being bucketed in the same rate-limiting bucket. ref: LOG-23166
ref: LOG-23166
darinspivey
approved these changes
Mar 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge upstream v0.51.1 + vrl 0.28.1