[pull] master from timberio:master#88
Merged
pull[bot] merged 1267 commits intoastradot:masterfrom Oct 21, 2021
Merged
Conversation
This commit introduces a `ready_chunks` into the pipeline for function transforms. This combinator polls the underlying stream until either it signals Pending or the chunk limit is reached, sending a vector of items downstream. In this commit we pull at most 128 `Event` instances off the input stream for processing, improving locality. I have not done the same for task transforms, though I think it could be done. Without changing the trait type we'd need the inverse of `ready_chunks` to move from a `Stream<Item = Vec<Event>>` into a `Stream<Item = Event>`. This is worth +7MB/s in #8512 Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
) * s/timberio/vectordotdev Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Fix remaining instances Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Undo Docker image rename Co-authored-by: Spencer Gilbert <Spencer.Gilbert@gmail.com> Co-authored-by: Spencer Gilbert <Spencer.Gilbert@gmail.com>
* Add source/edit buttons to editable pages Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Add page action buttons to component pages Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
* Re-introduce tokio-console under a feature flag This commit attempts to re-introduce the tokio console to the project, introduced and removed by @tobz in #8884. If you compile the project with `RUSTFLAGS="--cfg tokio_unstable"` and the `tokio-console` feature enabled this does _run_ but nothing shows up in the console yet, which I believe to be a misconfiguration on our part so far. Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * Flip on runtime/tokio traces for console After getting help from @hawkw on discord it turns out the thing the last PR was missing were targets for `runtime` and `tokio` at trace level in our environment filter. For now I've set these only to flag on if the console is enabled, as they're relatively chatty. Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
…ons (#9208) These functions were wrongly byte-swapping integers for their conversions, thus giving wrong results. The `u32::from(Ipv4Addr)` and `Ipv4Addr::from(u32)` functions do not require bitness conversions. Signed-off-by: Hugo Hromic <hhromic@gmail.com>
* fix subscription drop Signed-off-by: Lee Benson <lee@leebenson.com> * clippy Signed-off-by: Lee Benson <lee@leebenson.com> * handle signals Signed-off-by: Lee Benson <lee@leebenson.com> * fix const fn Signed-off-by: Lee Benson <lee@leebenson.com>
Bumps [bstr](https://github.com/BurntSushi/bstr) from 0.2.16 to 0.2.17. - [Release notes](https://github.com/BurntSushi/bstr/releases) - [Commits](BurntSushi/bstr@0.2.16...0.2.17) --- updated-dependencies: - dependency-name: bstr dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Closes: #9265 We should probably update the install script to not require bash, to target more platforms, but this fixes the immediate issue. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
This step appears to have been missed on 145f793 Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* Add VRL function default args Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Rework VRL function templating Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Add error return value Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Remove indentation on array returns Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Add brackets for option args Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Remove debug statement accidentally left in Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Fix required/optional ordering in args table Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Add missing word in description Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Fix indentation issue in examples Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
…ines. (#9177) * Allow if predicates over multiple lines. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Allow empty lines in predicate Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Update docs Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
…urce RFC (#9057) Signed-off-by: prognant <pierre.rognant@datadoghq.com> Co-authored-by: Ben Johnson <bjohnson@binarylogic.com> Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* Move InternalEvent to vector-core Signed-off-by: 001wwang <will.wang@datadoghq.com> * Add crate intro and clippy preamble Signed-off-by: 001wwang <will.wang@datadoghq.com> * Fix clippy unnecessary pass by value Signed-off-by: 001wwang <will.wang@datadoghq.com> * Fix format issue Signed-off-by: 001wwang <will.wang@datadoghq.com>
This shows up with `cargo clippy --no-default-features` Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* Fetch pod annotations in kubernetes_logs source Close #8338 Signed-off-by: LOU Xun <xun.lou@controlant.com> * Add doc for pod_annotations also test for disabling annotations scraping Signed-off-by: LOU Xun <xun.lou@controlant.com>
Signed-off-by: Lee Benson <lee@leebenson.com>
Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.11.0 to 1.12.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.11.0...tokio-1.12.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [chrono-tz](https://github.com/chronotope/chrono-tz) from 0.5.3 to 0.6.0. - [Release notes](https://github.com/chronotope/chrono-tz/releases) - [Changelog](https://github.com/chronotope/chrono-tz/blob/main/CHANGELOG.md) - [Commits](https://github.com/chronotope/chrono-tz/commits/v0.6.0) --- updated-dependencies: - dependency-name: chrono-tz dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [http](https://github.com/hyperium/http) from 0.2.4 to 0.2.5. - [Release notes](https://github.com/hyperium/http/releases) - [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md) - [Commits](hyperium/http@v0.2.4...v0.2.5) --- updated-dependencies: - dependency-name: http dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add initial JSON server impl Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Add CORS headers in Netlify config Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Update Makefile command comments Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Provide namespacing for JSON payload Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
…ility (#9205) * enhancement(file source): Update events to comply with component spec * Update the component spec with `error_type` * Rewrite metric names with component_ prefix Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
… HttpSink sinks (#9275) * enhancement(observability, sinks): Emit EventsSent in all HttpSink sinks * enhancement(observability, sinks): Emit BytesSent in all HttpSink sinks * Document the new component metrics for sinks Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.5 to 0.11.6. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](seanmonstar/reqwest@v0.11.5...v0.11.6) --- updated-dependencies: - dependency-name: reqwest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [headers](https://github.com/hyperium/headers) from 0.3.4 to 0.3.5. - [Release notes](https://github.com/hyperium/headers/releases) - [Commits](hyperium/headers@headers-v0.3.4...headers-v0.3.5) --- updated-dependencies: - dependency-name: headers dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.23 to 0.3.25. - [Release notes](https://github.com/TeXitoi/structopt/releases) - [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md) - [Commits](TeXitoi/structopt@v0.3.23...v0.3.25) --- updated-dependencies: - dependency-name: structopt dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* create rfc file Signed-off-by: Jérémie Drouet <jeremie.drouet@gmail.com> * Update 2021-10-12-9568-automatic-namespacing.md * Apply suggestions from code review Co-authored-by: Ben Johnson <bjohnson@binarylogic.com> * Update 2021-10-12-9568-automatic-namespacing.md * Update 2021-10-12-9568-automatic-namespacing.md * run grammarly * Update 2021-10-12-9568-automatic-namespacing.md * Update rfcs/2021-10-12-9568-automatic-namespacing.md Co-authored-by: Ben Johnson <bjohnson@binarylogic.com> * Update 2021-10-12-9568-automatic-namespacing.md * update user experience section * Update 2021-10-12-9568-automatic-namespacing.md * Update 2021-10-12-9568-automatic-namespacing.md * remove outstanding question * Update 2021-10-12-9568-automatic-namespacing.md * Update 2021-10-12-9568-automatic-namespacing.md * Update rfcs/2021-10-12-9568-automatic-namespacing.md Co-authored-by: Lee Benson <lee@leebenson.com> Co-authored-by: Ben Johnson <bjohnson@binarylogic.com> Co-authored-by: Lee Benson <lee@leebenson.com>
Signed-off-by: Jérémie Drouet <jeremie.drouet@gmail.com>
…#9614) * enhancement(observability, aws_s3 sink, datadog_archives sink): Add event processing metrics * Share the S3Metadata in S3Request Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
This allows the special `process` field that Papertrail uses to be templated to an event field. Closes: #9349 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
This currently doesn't build due to, at least, `ring`'s lack of support: briansmith/ring#389 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* Add config parameter warnings Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com> * Fix whitespace Signed-off-by: Jesse Szwedko <jesse@szwedko.me> Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Luc Perkins <luc.perkins@datadoghq.com>
Bumps [tokio-openssl](https://github.com/sfackler/tokio-openssl) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/sfackler/tokio-openssl/releases) - [Commits](tokio-rs/tokio-openssl@0.6.2...0.6.3) --- updated-dependencies: - dependency-name: tokio-openssl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [tokio-postgres](https://github.com/sfackler/rust-postgres) from 0.7.3 to 0.7.4. - [Release notes](https://github.com/sfackler/rust-postgres/releases) - [Commits](rust-postgres/rust-postgres@tokio-postgres-v0.7.3...tokio-postgres-v0.7.4) --- updated-dependencies: - dependency-name: tokio-postgres dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The charts have been migrated to `vectordotdev/helm-charts`. Partially closes #9714 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
…9611) Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
Signed-off-by: Vladimir Zhuk <vladimir.zhuk@datadoghq.com>
Noticed these while looking at #9723 These sinks just use the default concurrency. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
This only applies to the `v2` protocol and not the `v1` protocol but I think it is more important that the docs be accurate for the `v2` protocol than the `v1` protocol. Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [tower](https://github.com/tower-rs/tower) from 0.4.9 to 0.4.10. - [Release notes](https://github.com/tower-rs/tower/releases) - [Commits](tower-rs/tower@tower-0.4.9...tower-0.4.10) --- updated-dependencies: - dependency-name: tower dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(sinks): Default sink concurrency to adaptive It was actually still defaulting to 1024. Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * merge conflict Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Another merge conflict Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * clippy Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Bumps [encoding_rs](https://github.com/hsivonen/encoding_rs) from 0.8.28 to 0.8.29. - [Release notes](https://github.com/hsivonen/encoding_rs/releases) - [Commits](hsivonen/encoding_rs@v0.8.28...v0.8.29) --- updated-dependencies: - dependency-name: encoding_rs dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(sinks): Make more concurrency defaults adaptive Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
chore: Allow for localhost soak testing This pull-request introduces localhost soak testing to the vector project, part of the work to satisfy #9515. The soak introduced here is log aggregation, that is datadog_agent -> remap -> datadog_log. Running the soak is discussed in soak/README.md. Soaks are defined solely in terraform, which I expect will move around some as #9618 is worked. Vector containers are built locally for now, but once #9543 is available we can lean on the CI infrastructure to build containers some. Closes #9616 Closes #9617 Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Bumps [cached](https://github.com/jaemk/cached) from 0.25.1 to 0.26.2. - [Release notes](https://github.com/jaemk/cached/releases) - [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md) - [Commits](https://github.com/jaemk/cached/commits) --- updated-dependencies: - dependency-name: cached dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(buffers): buffer improvements RFC Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )