fix(remap): slice function collects subtypes #7092
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
StephenWakely
requested review from
a team,
pablosichert and
JeanMertz
and removed request for
a team
JeanMertz
approved these changes
Apr 14, 2021
langecode
added a commit
to neticdk/vector
that referenced
this pull request
May 31, 2021
* chore(deps): bump tokio-util from 0.6.5 to 0.6.6 (#7103)
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.5 to 0.6.6.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.5...tokio-util-0.6.6)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(ci): bump actions/cache from v2.1.4 to v2.1.5 (#7104)
Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.4...1a9e2138d905efd099035b49d8b7a3888c653ca8)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(datadog service): Document region parameter (#7093)
* fix(datadog service): Document region parameter
Closes #7079
This was added to the shared Datadog documentation in
https://github.com/timberio/vector/pull/4174 but never added
specifically to the sinks.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* Add missing cue config for region
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(socket source): Remove socket file on source shutdown (#7047)
* Delete socket
Signed-off-by: ktf <krunotf@gmail.com>
* Add tests
Signed-off-by: ktf <krunotf@gmail.com>
* Remove spaces
Signed-off-by: ktf <krunotf@gmail.com>
* chore(deps): bump reqwest from 0.11.2 to 0.11.3 (#7100)
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.2 to 0.11.3.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.2...v0.11.3)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(clickhouse sink): Fix clickhouse sink incorrectly encodes arrays (#7081)
* fix bug:clickhouse sink incorrectly encodes arrays
Signed-off-by: shumin <shumin1027@gmail.com>
* Add test for clickhouse arrays
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Co-authored-by: shumin <shumin@compubiq.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* Added collect_subtypes. (#7092)
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* fix(docker_logs source): Docker logs missing when container has a tty (#7111)
* fix(docker_logs source): Docker logs missing when container has a tty
Enum bollard::container::LogOutput is Console when docker started
with a tty. LogOutput::Console was missing in the new_event of a
stream value received
https://docs.rs/bollard/0.10.1/bollard/container/enum.LogOutput.html
Fixes #5903
Signed-off-by: Jean Prat <jean.prat@sinch.com>
* chore(ci): Remove unneeded advisory from deny.toml (#7114)
This advisory seems to no longer be valid.
Closes: #6223
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(external docs): Fix link to unit testing documentation (#7122)
Fixes #7121
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): Upgrade cue to 0.3.2 (#7128)
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(external docs): Document additional encoding fields (#7127)
Ensure `except_fields`, `only_fields`, and `timestamp_format` appear on
all sinks supporting `encoding`. Previously they only appeared on sinks
that had healthchecks.
Fixes: #6949
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(ci): Add advisory RUSTSEC-2021-0013 back to deny.toml (#7133)
* chore(ci): Add advisory RUSTSEC-2021-0013 back to deny.toml
I neglected to enable all features. The `wasm` dependencies still depend
on an affected version of rust-cpuid.
This reverts commit f30d4c6ed43838b36769753f2a591c2378162e04.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* Run cargo deny in CI if deny.toml changes
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump async-trait from 0.1.48 to 0.1.49 (#7131)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.48 to 0.1.49.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.48...0.1.49)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(ci): Ensure all benchmark artifacts are included (#7113)
* fix(ci): Ensure all benchmark artaficts are included
CRITERION_HOME having "s was causing it to put it in the wrong place
(locally it created a directory called ").
Also just upload `./target/criterion` since Github does the
zipping for us. We did need to zip when previously downloading the
artifact in the same workflow as this was not zipped yet and so we were
hitting limits trying to individually pull a very large number of files.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump tokio from 1.4.0 to 1.5.0 (#7102)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.4.0...tokio-1.5.0)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump notify from 4.0.15 to 4.0.16 (#7142)
Bumps [notify](https://github.com/notify-rs/notify) from 4.0.15 to 4.0.16.
- [Release notes](https://github.com/notify-rs/notify/releases)
- [Changelog](https://github.com/notify-rs/notify/blob/v4.0.16/CHANGELOG.md)
- [Commits](https://github.com/notify-rs/notify/compare/v4.0.15...v4.0.16)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(remap): Preserve type defs when assigning fields (#7118)
* fix(remap): Preserve type defs when assigning fields
Previously, remap would overwrite the type def of `.` whenever a new
field was assigned. That is:
```
.foo = 5
.bar = 6
```
Would result in the compiler only having type info for `.bar`.
This change causes it to merge the typedefs whenever a list of path
segments appears.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump pretty_assertions from 0.7.1 to 0.7.2 (#7143)
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v0.7.1...v0.7.2)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(kubernetes_logs source): Always use file checkpoints if they exist (#7140)
* fix(kubernetes_logs source): Always use file checkpoints if they exist
The `kubernetes_logs` source exposes a `PathProvider` that breaks one of
the `FileServer`s assumptions that all available files will be listed at
Vector startup time. Instead, the files are only returned once the k8s
metadata is available to the `kubernetes_logs` source. This caused the
`FileServer` to ignore any checkpoints that existed for these files.
As a short-term fix, we just always use the checkpoint, if available,
for any new files that are seen. This fixes the case for the
`kubernetes_logs` source where they are seen as "new" after start-up.
https://github.com/timberio/vector/pull/6564 exists to test this
behavior, but it seems to pass even without this change, so that test
will need to be updated.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(remap): Fix case_sensitive defaults for string matching functions (#7091)
Fixes #7044
The docs and examples indicated that it should default to case
sensitive; however, `contains`, `starts_with`, and `ends_with`,
defaulted to case insensitive matching.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(compression): Switch to MultiGzDecoder instead of GzDecoder (#7138)
* fix(compression): Switch to MultiGzDecoder instead ef GzDecoder
Fixes #7061
It appears that AWS's ALB logging gzip's multi-part files which we were
only reading the first part of. I tested that `MultiGzDecoder` works on
simple gzip files so I figured we should switch to it everywhere. It was
already being used by the `file` source.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* Add test for multi-part zst files
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(performance): Fix remap benches (#7155)
* chore(performance): Fix remap benches
I broke them in https://github.com/timberio/vector/pull/7118
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): `cargo upgrade` (#7154)
I backed out a few changes:
```
Upgrading grok v~1.0.1 -> v1.2.0
Upgrading async-graphql-warp v=2.6.4 -> v2.8.2
Upgrading once_cell v1.3 -> v1.7.2
Upgrading async-graphql v=2.6.4 -> v2.8.2
```
Due to incompatibilities in package dependencies.
I backed out
```
Upgrading db-key v0.0.5 -> v0.1.0
```
Because it requires code changes.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump rust_decimal from 1.10.3 to 1.11.0 (#7160)
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.10.3 to 1.11.0.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.10.3...1.11.0)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(docker_logs source): ensure that docker labels are flattened (#7152)
Fixes: #5716
Previously labels on docker containers were inserted in such a way that
dots in their names would end up creating a nested structure in the log
event due to the semantics of `LogEvent.insert`.
For example:
```json
{
"container_created_at": "2021-04-16T18:53:19.946155600Z",
"container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319",
"container_name": "docker_echo_1",
"host": "COMP-C02DV25MML87",
"image": "hashicorp/http-echo:latest",
"label": {
"com": {
"docker": {
"compose": {
"config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e",
"container-number": "1",
"oneoff": "False",
"project": {
"config_files": "docker-compose.yml",
"working_dir": "/private/tmp/docker"
},
"service": "echo",
"version": "1.27.4"
}
}
}
},
"message": "2021/04/16 19:14:10 localhost:5678 172.29.0.1:61824 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 35.6µs",
"source_type": "docker",
"stream": "stdout",
"timestamp": "2021-04-16T19:14:10.400790400Z"
}
```
This change ensures that labels are inserted as-is as keys:
```json
{
"container_created_at": "2021-04-16T18:53:19.946155600Z",
"container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319",
"container_name": "docker_echo_1",
"host": "COMP-C02DV25MML87",
"image": "hashicorp/http-echo:latest",
"label": {
"com.docker.compose.config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "docker",
"com.docker.compose.project.config_files": "docker-compose.yml",
"com.docker.compose.project.working_dir": "/private/tmp/docker",
"com.docker.compose.service": "echo",
"com.docker.compose.version": "1.27.4"
},
"message": "2021/04/16 19:12:01 localhost:5678 172.29.0.1:61820 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 18.1µs",
"source_type": "docker",
"stream": "stdout",
"timestamp": "2021-04-16T19:12:01.622769500Z"
}
```
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump gouth from 0.2.0 to 0.2.1 (#7162)
Bumps [gouth](https://github.com/mechiru/gouth) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/mechiru/gouth/releases)
- [Commits](https://github.com/mechiru/gouth/compare/v0.2.0...v0.2.1)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump async-trait from 0.1.49 to 0.1.50 (#7164)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.49 to 0.1.50.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.49...0.1.50)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): Upgrade curl-sys (#7174)
Resolves CVE-2021-22876 CVE-2021-22890
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore: Extract Core RFC (#7135)
This commit introduces a new RFC for extracting the core of vector. Closes #7027.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore: Create skeleton `lib/core` (#7178)
* Create skeleton `lib/core`
Per RFC 7027 we intend to create an extracted vector core. This is the first
step in that RFC's plan of action, to create a skeleton package "core" to
migrate code into. There's very little going on here, just a new library -- with
aggressive clippy settings that we might back out later -- and a correctness and
benchmark test.
Closes #7147
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Add newline to the end of Cargo.toml
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove unused benchmark
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore(deps): bump rust_decimal from 1.11.0 to 1.11.1 (#7182)
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.11.0 to 1.11.1.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.11.0...1.11.1)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump httpmock from 0.5.7 to 0.5.8 (#7181)
Bumps [httpmock](https://github.com/alexliesenfeld/httpmock) from 0.5.7 to 0.5.8.
- [Release notes](https://github.com/alexliesenfeld/httpmock/releases)
- [Changelog](https://github.com/alexliesenfeld/httpmock/blob/master/CHANGELOG.md)
- [Commits](https://github.com/alexliesenfeld/httpmock/compare/v0.5.7...v0.5.8)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(kubernetes_logs source): Refactor stream and invocation errors to support recoverable error types (#6816)
* Refactor stream and invocation errors to support recoverable error types
Signed-off-by: Ian Henry <ianjhenry00@gmail.com>
* feat(remap): add to_regex function (#7074)
* add to_regex function
Signed-off-by: Jake He <jhe@homex.com>
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* chore(host_metrics source): Document symlink warning workaround (#7179)
* chore(host_metrics source): Document symlink warning workaround
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* Fix spacing
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* Add adjustments to the kubernetes helm chart to match documentation
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* enhancement(remap): Support identifiers with leading numeric characters as fields in path (#7045)
* Support integers in path
Signed-off-by: ktf <krunotf@gmail.com>
* fmt
Signed-off-by: ktf <krunotf@gmail.com>
* Refactor
Signed-off-by: ktf <krunotf@gmail.com>
* New way
Signed-off-by: ktf <krunotf@gmail.com>
* Remove support for .0
Signed-off-by: ktf <krunotf@gmail.com>
* Add docs.
Signed-off-by: ktf <krunotf@gmail.com>
* Update name
Signed-off-by: ktf <krunotf@gmail.com>
* Rename `core` to `vector-core` (#7195)
Rust has its own crate `core` and I didn't anticipate that our own core would
collide with it. In retrospect it surely does.
REF #7148
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore(releasing): Merge down of 0.13.0 release (#7194)
* chore(releasing): Prep 0.13.0 release
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(dev): Bump to 0.14.0 (#7205)
Now that 0.13.0 is released.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(remap): Escape quotes and backslashes when printing VRL strings (#6739)
* Escape quotes when displaying VRL strings
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Escape backslashes when displaying VRL strings
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Move shared library to dev dependencies
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Escape newlines when displaying VRL strings
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump cidr-utils from 0.5.1 to 0.5.3 (#7211)
Bumps [cidr-utils](https://github.com/magiclen/cidr-utils) from 0.5.1 to 0.5.3.
- [Release notes](https://github.com/magiclen/cidr-utils/releases)
- [Commits](https://github.com/magiclen/cidr-utils/compare/v0.5.1...v0.5.3)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump async-stream from 0.3.0 to 0.3.1 (#7209)
Bumps [async-stream](https://github.com/tokio-rs/async-stream) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/tokio-rs/async-stream/releases)
- [Commits](https://github.com/tokio-rs/async-stream/compare/v0.3.0...v0.3.1)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump syn from 1.0.69 to 1.0.70 (#7210)
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.69...1.0.70)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump hyper from 0.14.5 to 0.14.6 (#7212)
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.5 to 0.14.6.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.5...v0.14.6)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Move `Result` and `Error` into vector-core (#7207)
These two types are required for the `Event` interface. I have re-exported them
back in the top-level package so no code changes are necessary at that level.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove RUSTSEC-2020-0053 from ignore list since `dirs` is maintained again (#7208)
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* chore(deps): Remove RUSTSEC-2020-0146 from deny.toml (#7216)
Appears to no longer be relevant.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(performance): Consider dropping jemalloc (#6781)
I was originally going to table this after some discussion with Luke about the poor performance of musl's liballoc in multi-threaded environment, however, I had some more discussion with Brian (thank you!) that convinced me that defaulting to the system allocator may still be a better default. An argument follows:
The primary motivator for using musl, and thus greatly benefiting from jemalloc in multi-threaded environments, is with alpine docker images. Users typically choose alpine images for size reasons; however, we also publish distroless images that include libc that are only 5MB larger (which, coincidentally is also bump in size that results from including jemalloc; https://github.com/timberio/vector/pull/310).
0.12.1 images:
```
timberio/vector 0.12.1-distroless-libc a01c8409bacb 4 days ago 93MB
timberio/vector 0.12.1-distroless-static b61e0880d16c 4 days ago 82.4MB
timberio/vector 0.12.1-alpine ef029e56b5ee 4 days ago 88MB
timberio/vector 0.12.1-debian 2ccea0b5ffdb 4 days ago 164MB
```
Given this, it seems like we can encourage users to install the `0.12.1-distroless-libc` image as the image of choice for size conscious installs. Distroless also has the security advantage by providing less in the container itself .
Additionally, jemalloc is historically preferred for certain workloads but it doesn't seem like we have an understanding of vector's allocation profile yet and which strategies will result in the best performance. Until we have a very strong reason to, just defaulting to the system allocator seems like a good default.
Thus the proposal is to:
* Default to the system allocator.
* Make it easy to swap the allocator to test out different allocators in the future.
* Strongly highlight this change with the 0.13 release and encourage anyone using a musl-based build to consider whether the glibc build would be usable for them.
* Document this performance difference between glibc and musl's liballoc.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(ci): bump actions/github-script from v3.1 to v4.0.1 (#7213)
Bumps [actions/github-script](https://github.com/actions/github-script) from v3.1 to v4.0.1.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v3.1...85e88a66eaa831097093a3d278536947f2984d20)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add a cargo-hack workflow (#7224)
In #7200 I ran into #7222 and figured we might do well to make cargo-hack use
official. These runs will take _a long_ time considering the feature surface of
this project so I've made it strictly a nightly run.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore: Adjust import of `crate::Event` to `crate::event::Event` (#7200)
* Adjust import of `crate::Event` to `crate::event::Event`
My end goal here is to move the `Event` and related types into
`vector-core`. This is possible but trickier if we maintain a top-level `Event`
import in vector and so here I've adjusted all imports of the relevant type to
`crate::event::Event`, removing the top-level `Event`.
This is all mechanical work and I have intentionally not changed any logic, in
deference to the size of this PR.
REF #7148
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Correct benchmark compilation failure
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* More build fixes
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* More build fixes, wasm
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* another bench import fix
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore(deps): bump hyper from 0.14.6 to 0.14.7 (#7232)
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.6 to 0.14.7.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.6...v0.14.7)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump regex from 1.4.5 to 1.4.6 (#7233)
Bumps [regex](https://github.com/rust-lang/regex) from 1.4.5 to 1.4.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.5...1.4.6)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(external docs): Correct highlight dates for 0.13.0
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump async-nats from 0.9.12 to 0.9.14 (#7234)
Bumps [async-nats](https://github.com/nats-io/nats.rs) from 0.9.12 to 0.9.14.
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Changelog](https://github.com/nats-io/nats.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nats-io/nats.rs/compare/0.9.12...0.9.14)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: Run cargo-hack on self-hosted machines (#7227)
* Run cargo-hack on self-hosted machines
We found in the previous run of cargo-hack that it was OOM'ed. While we were
expecting a build failure -- see #7222 -- we did not expect _that_ failure.
Run on self-hosted hardware to get more memory for the build.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Update .github/workflows/cargo_hack.yml
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* don't consume all memory
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* don't clean-per-run
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* remove PR run directive
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* chore(buffers): Upgrade DiskBuffer to futures 0.3 (#7165)
* Update writer
Signed-off-by: ktf <krunotf@gmail.com>
* Update reader
Signed-off-by: ktf <krunotf@gmail.com>
* Update tests
Signed-off-by: ktf <krunotf@gmail.com>
* Cleanup
Signed-off-by: ktf <krunotf@gmail.com>
* Update benches
Signed-off-by: ktf <krunotf@gmail.com>
* Fix benches
Signed-off-by: ktf <krunotf@gmail.com>
* Use start_send
Signed-off-by: ktf <krunotf@gmail.com>
* Bump
Signed-off-by: ktf <krunotf@gmail.com>
* Fmt
Signed-off-by: ktf <krunotf@gmail.com>
* Add comment
Signed-off-by: ktf <krunotf@gmail.com>
* chore(splunk_hec source): Update to `futures` 0.3 (#7225)
* Update source
Signed-off-by: ktf <krunotf@gmail.com>
* Use forward
Signed-off-by: ktf <krunotf@gmail.com>
* enhancement(graphql api): Expose `events_in` & `events_out` metrics (#6888)
* Add in/out to API
Signed-off-by: ktf <krunotf@gmail.com>
* Update client
Signed-off-by: ktf <krunotf@gmail.com>
* Update tests
Signed-off-by: ktf <krunotf@gmail.com>
* Add new line
Signed-off-by: ktf <krunotf@gmail.com>
* Apply suggestions
Co-authored-by: Lee Benson <lee@leebenson.com>
Signed-off-by: ktf <krunotf@gmail.com>
* Update schema
Signed-off-by: ktf <krunotf@gmail.com>
Co-authored-by: Lee Benson <lee@leebenson.com>
* chore(deps): bump rust_decimal from 1.11.1 to 1.12.2 (#7243)
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.11.1 to 1.12.2.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.11.1...1.12.2)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump tokio-postgres from 0.7.1 to 0.7.2 (#7244)
Bumps [tokio-postgres](https://github.com/sfackler/rust-postgres) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/sfackler/rust-postgres/releases)
- [Commits](https://github.com/sfackler/rust-postgres/compare/tokio-postgres-v0.7.1...tokio-postgres-v0.7.2)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(ci): bump actions/github-script from v4.0.1 to v4.0.2 (#7245)
Bumps [actions/github-script](https://github.com/actions/github-script) from v4.0.1 to v4.0.2.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v4.0.1...a3e7071a34d7e1f219a8a4de9a5e0a34d1ee1293)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump libc from 0.2.93 to 0.2.94 (#7254)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.93 to 0.2.94.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.93...0.2.94)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(external docs): Document VRL |= operator (#7252)
Closes #6900
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* enhancement(kubernetes_logs source): Allow configuration of max_line_bytes (#7226)
* enhancement(kubernetes_logs source): Allow configuration of max_line_bytes
Fixes: #6966
This is specifically to allow a user to workaround the too-low hardcoded
value that was being used.
We have https://github.com/timberio/vector/issues/6967 to address that.
I lost the logs that the user provided that ran over that limit but
I believe it was due to having a very large number of labels in the
metadata for each message which is not included in the normal 16 KB
docker limit.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump tower from 0.4.6 to 0.4.7 (#7260)
Bumps [tower](https://github.com/tower-rs/tower) from 0.4.6 to 0.4.7.
- [Release notes](https://github.com/tower-rs/tower/releases)
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.6...tower-0.4.7)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump syn from 1.0.70 to 1.0.71 (#7261)
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.70...1.0.71)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(kafka source): Fix runaway memory usage of kafka source (#7266)
* fix(kafka source): Fix runaway memory usage of kafka source
Fixes: #7258
Previously, we were cloning the `Pipeline` for each message which, due
to the behavior of futures::sync::mpsc::channel resulted in the
effective channel buffer growing by one for each message rather than any
back pressure.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(external docs): typo (#7268)
* chore(deps): bump openssl from 0.10.33 to 0.10.34 (#7270)
Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.33 to 0.10.34.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.33...openssl-v0.10.34)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump pulsar from 2.0.0 to 2.0.1 (#7271)
Bumps [pulsar](https://github.com/wyyerd/pulsar-rs) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/wyyerd/pulsar-rs/releases)
- [Commits](https://github.com/wyyerd/pulsar-rs/compare/v2.0.0...v2.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(releasing): Update install script to 0.13.1 (#7275)
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* feat(new source): Initial `exec` source (#6876)
* feat(exec source): new `exec` source (#992) (#7)
Added exec source with streaming and scheduled capability.
Generates log events from stdout and stderr of a command execution.
Signed-off-by: Stuart Broad <stuart@moogsoft.com>
* enhancement: created new lookup library (#7087)
* Created new lookup library
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Rudimentary form of lookup working with Vrl.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Lookup can parse paths itself.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Restored VRL coalesce
Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>
* Updated property tests.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Fixed issue with inserting a path that changes the type.
Plus a number of other tidy ups.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Clippy and formatting.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Removed unneeded file
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Removed commented out Path.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Formatting
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Vrl needs to include the . when displaying paths.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Feedback from blt.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Fixes following feedback from Jean.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Fixes following feedback from Bruce.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Clippy
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* More clippy fixes
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Renamed clone_lookup to be to_lookup
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Enable negative indexing
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Added quickcheck tests for lookups
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Fixed setting zero index
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Add limit to alternative component path
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Removed irrelevant tests
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Clippy
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Move rerun-if-changed instruction
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Tidied some comments.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Don't serialize metadata
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Update types when array index is changed.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Removed insert_null.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Clippy
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Minor comment fixes
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Remove serialize test as it is incorrect
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Feedback from Bruce.
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* Don't wipe out metadata on vrl remove
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* chore(deps): bump rust_decimal from 1.12.2 to 1.12.3 (#7280)
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.2 to 1.12.3.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.12.2...1.12.3)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump libz-sys from 1.1.2 to 1.1.3 (#7278)
Bumps [libz-sys](https://github.com/rust-lang/libz-sys) from 1.1.2 to 1.1.3.
- [Release notes](https://github.com/rust-lang/libz-sys/releases)
- [Commits](https://github.com/rust-lang/libz-sys/compare/1.1.2...1.1.3)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump async-nats from 0.9.14 to 0.9.16 (#7279)
Bumps [async-nats](https://github.com/nats-io/nats.rs) from 0.9.14 to 0.9.16.
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Changelog](https://github.com/nats-io/nats.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nats-io/nats.rs/compare/0.9.14...0.9.16)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: Update `top` with `events_in_total` and `events_out_total` metrics (#7286)
* Use in/out
Signed-off-by: ktf <krunotf@gmail.com>
* Remove processed_events
Signed-off-by: ktf <krunotf@gmail.com>
* Remove empty line
Signed-off-by: ktf <krunotf@gmail.com>
* Events header
Signed-off-by: ktf <krunotf@gmail.com>
* Update column width
Signed-off-by: ktf <krunotf@gmail.com>
* Pluralize
Signed-off-by: ktf <krunotf@gmail.com>
* Fixed doc tests from lookup (#7288)
Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>
* chore(ci): Fix action-discord version for test workflow (#7285)
Accidentally replaced with 0.3.2 when upgrading cue due to an eager
find/replace.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(graphql api): `--no-default-features --features api` build failure (#7284)
* host metrics
Signed-off-by: Lee Benson <lee@leebenson.com>
* -pub
Signed-off-by: Lee Benson <lee@leebenson.com>
* fix(external docs): Document scrape_interval_secs for `internal_metrics` source (#7289)
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(tests): Simplify `cargo test` usage (#7287)
We currently use `--all-targets` to include benches (`--bench`), but `--all-targets` does not include `--doc` apparently (https://github.com/rust-lang/cargo/issues/6669). Instead of using `--all-targets` we just mark benches with `test = true` so that they will be tested along with the default targets (https://doc.rust-lang.org/cargo/commands/cargo-test.html#target-selection)
Also:
* Avoid setting `SCOPE` by default in the `Makefile` as this causes examples to be skipped for some reason.
* Use `make test` for Windows to take advantage of existing make targets. I tried `make test-components` there too, but it failed to build krb5.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump tracing-core from 0.1.17 to 0.1.18 (#7294)
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.17 to 0.1.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.17...tracing-core-0.1.18)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump tui from 0.14.0 to 0.15.0 (#7295)
Bumps [tui](https://github.com/fdehau/tui-rs) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/fdehau/tui-rs/releases)
- [Changelog](https://github.com/fdehau/tui-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/fdehau/tui-rs/compare/v0.14.0...v0.15.0)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump built from 0.4.4 to 0.5.0 (#7297)
Bumps [built](https://github.com/lukaslueg/built) from 0.4.4 to 0.5.0.
- [Release notes](https://github.com/lukaslueg/built/releases)
- [Commits](https://github.com/lukaslueg/built/compare/0.4.4...0.5.0)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* enhancement(kafka source, kafka sink): Emit internal metrics provided by rdkafka (#5874)
* Emit internal metrics for rdkafka
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Rename KafkaStatistics -> KafkaStatisticsReceived
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Emit KafkaStatisticsReceived as counter with JSON annotation
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Fix missing fields in KafkaSinkConfig
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Flatten into statistics_interval_ms
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Document statistics_interval_ms option for kafka source/sink
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Create producer in kafka sink with KafkaStatisticsContext
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Add component docs for kafka_statistics_received_total telemetry
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Emit selected metrics from kafka statistics
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Remove trivial constructor and take statistics by reference
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Remove statistics_interval_ms config options and set kafka statistics interval to 1s by default
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Change kafka metrics from gauge to counter where appropriate
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Fix unused import
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Rerun checks for benchmark results
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Fix punctuation
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Calculate counter delta locally instead of in global recorder/layer
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Emit delta before dropping mutex guard / document behavior
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Warn and emit 0 instead of panicking on non-monotonic values
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Fix non-capitalized log message
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Fix typo
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Use atomic instead of mutex in update_counter!
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Use const initializer instead of lazy_static!
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump rust_decimal from 1.12.3 to 1.12.4 (#7301)
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.3 to 1.12.4.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/commits)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump bstr from 0.2.15 to 0.2.16 (#7302)
Bumps [bstr](https://github.com/BurntSushi/bstr) from 0.2.15 to 0.2.16.
- [Release notes](https://github.com/BurntSushi/bstr/releases)
- [Commits](https://github.com/BurntSushi/bstr/compare/0.2.15...0.2.16)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(external docs): Typo (#7304)
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore: Make CI tests run a little more quiet (#7269)
* Separate bench and test, uh, tests
This commit re-introduces the `--quiet` flag -- this will still output for
failing tests -- and splits up the notion of the benchmark sanity tests versus
the correctness, correctness tests.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Split benches debug tests into their own workflow
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* correct 'bench unit' name
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* feedback updates
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* more elaborate caching
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Revert "more elaborate caching"
This reverts commit 9ed4fdd0b4e7ae32d59531262ba63a474668efdd.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* warning fix
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove unused target
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* enhancement(remap): Wdd `whitespace` option to `parse_key_value` function (#7292)
* Empty value
Signed-off-by: ktf <krunotf@gmail.com>
* Whitespace option
Signed-off-by: ktf <krunotf@gmail.com>
* Whitespace before
Signed-off-by: ktf <krunotf@gmail.com>
* Docs
Signed-off-by: ktf <krunotf@gmail.com>
* Replace bool with Whitespace
Signed-off-by: ktf <krunotf@gmail.com>
* chore(deps): bump tracing-subscriber from 0.2.17 to 0.2.18 (#7298)
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.17 to 0.2.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.17...tracing-subscriber-0.2.18)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump snap from 1.0.4 to 1.0.5 (#7300)
Bumps [snap](https://github.com/BurntSushi/rust-snappy) from 1.0.4 to 1.0.5.
- [Release notes](https://github.com/BurntSushi/rust-snappy/releases)
- [Commits](https://github.com/BurntSushi/rust-snappy/compare/1.0.4...1.0.5)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: Run cargo-flake on the project (#7307)
* Run cargo-flake on the project
I have struggled in #7240 to get reliable test results. This has inspired both
PR #7269 and #7306 to reduce both output and turn-around time for tests. In this
PR I introduce a tool that is meant to repeatedly run our tests and identify any
that are flakey.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* adjustments
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* `cargo flake` not `cargo-flake`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* remove temporary PR run
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore: Introduce more aggressive caching (#7306)
* Introduce more aggressive caching
This commit introduces caching of `target/debug` into our jobs as well as the
use of `restore-keys` to hit cache even when `Cargo.lock` has changed. This
commit is further work from #7269
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* `target/debug` not `~/target/debug`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Back off caching `target/debug`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* faster macOS dev builds (#7317)
Signed-off-by: Lee Benson <lee@leebenson.com>
* chore: Extract `Event` into vector-core (#7240)
* chore: Extract `Event` into vector-core
This PR extracts `Event`, `metrics` and `mapping` module into vector-core. I had
hoped to do these separately but they proved to be tied together in a way I
couldn't separate. I've intentionally not made many structural changes to this
code. As much as possible it's a simple move operation.
Closes #7151 #7148
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* address clippy warnings
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* address failing vector-core tests
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* clippy fixes
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* correct log_schema
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* correct panic! without formatting string
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* address build issue after rebase
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore(deps): bump rust_decimal from 1.12.4 to 1.13.0 (#7320)
Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.4 to 1.13.0.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/commits/1.13.0)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump ordered-float from 2.1.1 to 2.2.0 (#7321)
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 2.1.1 to 2.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v2.1.1...v2.2.0)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Move `benches/lookup` into vector-core (#7331)
While the fixture data for this benchmark was moved into `vector-core` as a part
of #7240 the benchmark itself was not. This broke the build.
Resolves #7329
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* fix(ci): Flag benchmark runs that fail to actually execute (#7330)
I had included `continue-on-error` originally since we didn't want to
flag regressions yet, but this also suppresses other errors (like the
benches couldn't run).
The `|| true` in `scripts/check-criterion-output.sh` should stop it from
flagging regressions as failed CI checks still.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(splunk_hec source): Default `host` to remote address (#7293)
* Use remote address
Signed-off-by: ktf <krunotf@gmail.com>
* Add test
Signed-off-by: ktf <krunotf@gmail.com>
* Add issue link
Signed-off-by: ktf <krunotf@gmail.com>
* fix(cli): Enable use of delimiters (#7316)
Signed-off-by: ktf <krunotf@gmail.com>
* enhancement(datadog sinks): Generic site support (#7281)
Add a `site` option for Datadog sinks, it aims to ultimately replace
the `region` option.
Signed-off-by: prognant <pierre.rognant@datadoghq.com>
* chore: Reference RFC template in RFC readme (#7344)
* chore: Reference RFC template in RFC readme
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
* Use relative link
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Pablo Sichert <mail@pablosichert.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* Tidy up iterator logic (#7338)
This commit tidies up the iterator logic suggested by clippy to be more
direct. Changed suggested by @bruceg in #7240.
Partially resolves #7327
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Make vrl optional in vector-core (#7335)
While VRL is essential to vector it is not a part of core. Allowing core to be
built without linking to VRL was originally a feature of #7240 but was removed
as it increased the testing surface overmuch.
By default, now, vector-core will not link with VRL.
Partially addresses #7327
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore: Use 'cargo hack' instead of check-component-features (#7311)
* Use 'cargo hack' instead of check-component-features
This commit replaces our check-component-features with 'cargo hack' to the same
purpose. We also remove the nightly check as it appears that it is recently
fixed.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Correct makefile typo
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Address build failure
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* kafka build issue resolved, all-metrics feature flag
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* re-introduce 'check-component-features' target
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Exclude 'utils' features from hack check
Turns out we have features that are only for utility purposes, perfectly valid
but if we check them alone we'll end up with export warnings.
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore(deps): bump syn from 1.0.71 to 1.0.72 (#7334)
Bumps [syn](https://github.com/dtolnay/syn) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.71...1.0.72)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump regex from 1.4.6 to 1.5.3 (#7319)
Bumps [regex](https://github.com/rust-lang/regex) from 1.4.6 to 1.5.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.6...1.5.3)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): bump tracing from 0.1.25 to 0.1.26 (#7296)
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.25 to 0.1.26.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.25...tracing-0.1.26)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* fix(ci): Run tests for all targets on all platforms (#7323)
Previous benches job was actually running unit tests too. This reverts to running all targets in one job.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* Derive `Default` for `MakeLogEvent` (#7339)
As noted by @bruceg in #7240 we could avoid the explicit call to
`LogEvent::default` by deriving `Default` for `MakeLogEvent`. This is now done
and I've tidied up the relevant call-site.
Partial resolves #7327
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* chore(sinks): Drop unnecessary `Request` parameter from types (#7251)
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* chore: Drop `futures01` (#7343)
Signed-off-by: ktf <krunotf@gmail.com>
* chore(deps): bump sha-1 from 0.9.4 to 0.9.5 (#7357)
Bumps [sha-1](https://github.com/RustCrypto/hashes) from 0.9.4 to 0.9.5.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha-1-v0.9.4...sha-1-v0.9.5)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix the postinst conditionnal for adding vector to the systemd-journal group (#7359)
Signed-off-by: Johann Queuniet <sub_code.git@queuniet.fr>
* chore(deps): bump metrics from 0.14.2 to 0.15.1 (#7313)
* chore(deps): bump metrics from 0.14.2 to 0.15.1
Bumps [metrics](https://github.com/metrics-rs/metrics) from 0.14.2 to 0.15.1.
- [Release notes](https://github.com/metrics-rs/metrics/releases)
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.14.2...metrics-v0.15.1)
Signed-off-by: dependabot[bot] <support@github.com>
* Upgrade other metrics dependencies and fix conflicts
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* Update metrics versions in vector core too
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
* chore(deps): Update grok and dependencies (#7176)
* chore(deps): Update grok and dependencies
This updates grok to 1.1 and its dependencies.
I couldn't upgrade to 1.2 due to a conflict with lucet on `userfaultfd`:
```
error: failed to select a version for `userfaultfd`.
... required by package `lucet-runtime-internals v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)`
... which is depended on by `lucet-runtime v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)`
... which is depended on by `lucet-wasi v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)`
... which is depended on by `vector v0.13.0 (/Users/jesse.szwedko/workspace/vector)`
versions that meet the requirements `^0.2.0` are: 0.2.1, 0.2.0
```
This required adding additional build dependencies to the cross and Windows builds: `llvm` and `clang`.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore: Drop redundant copy of built info (#7365)
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* chore: RFC for emitting multiple log events from remap (#7038)
* chore: RFC for emitting multiple log events from remap
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore: Enable stricter clippy lints in vector-core (#7341)
* Enable stricter clippy lints in vector-core
In the early drafts of #7240 we had strict clippy lints on. While these were
desirable they ended up bloating the PR. We now re-introduce those strict
lints.
Partially resolves #7327
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* A small host of clippy::pedantic suggestions
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* more clippy::pedentic suggestions
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* more clippy::pedantic fixes
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* fix reference issue
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Back off mandatory `must_use`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Allow `clippy::match_wildcard_for_single_variants`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove site grant for `redundant_closure_for_method_calls`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Slightly adjust Number parse
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Site grant for `needless_pass_by_value`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove unused import
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove allow for `clippy::doc_markdown`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove global grant for allow_too_many_lines
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove global grant for enum_glob_use
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Disallow global grant for cast_possible_truncation
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove global grant for `missing_error_docs`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Remove global grant for `missing_panic_docs`
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* clippy dings in lua-town
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* add documentation
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* clippy dings in test code
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* reverse non_ascii_literal
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* documentation, 'cargo hack' detected test failures
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Document clippy lint
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* more documentation
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Update lib/vector-core/src/config/log_schema.rs
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Co-authored-by: Bruce Guenter <bruce@untroubled.org>
* Don't use approximate constants
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* Value -> Self
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
* address clippy ding
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Co-authored-by: Bruce Guenter <bruce@untroubled.org>
* chore(metrics): Remove emits & alias `processed_events_total` metric (#7345)
* Add alias
Signed-off-by: ktf <krunotf@gmail.com>
* Remove emit
Signed-off-by: ktf <krunotf@gmail.com>
* Update docs
Signed-off-by: ktf <krunotf@gmail.com>
* Format cue
Signed-off-by: ktf <krunotf@gmail.com>
* chore(deps): bump regex from 1.5.3 to 1.5.4 (#7374)
Bumps [regex](https://github.com/rust-lang/regex) from 1.5.3 to 1.5.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.5.3...1.5.4)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(ci): Install llvm on Windows nightly and release bulids too (#7377)
Installed in test.yml as part of
https://github.com/timberio/vector/pull/7176
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* chore(metrics): Fix alias for processed_events_total (#7378)
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* fix(observability)!: Correct all `*_nanoseconds` metrics to `_seconds` (#7373)
* fix(observability): Correct all `*_nanoseconds` metrics to `_seconds`
All the referenced metrics labelled as "nanoseconds" are passed a
`Duration` as the value, which the `metrics` crate automatically
converts into (fractional) seconds internally. As such, all these names
are incorrect and need correction.
* Fix some more named `*_ns` but were really seconds too
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* chore(data model): Add metadata support for end-to-end acknowledgements (#7141)
* Drop unreferenced struct from RFC
* chore(data model): Add metadata for event finalization
* Add the event finalizers to the top-level metadata
* Move cloning into registration functions
This avoids extraneous use of `Arc::clone` when the item is already in
the table.
* Add a "counting" DashMap to provide serialization semantics
* Back out support for serialzing finalization status
In discussion with Brian and Luke, we decided to punt for now on
supporting end-to-end acknowledgements after an event has been buffered
to disk. This greatly simplifies the initial requirements, but we will
come back to this later.
* Drop remaining `impl From<LogEvent>` that silently drop metadata
* Quiet a clippy warning
In `enum BufferInputCloner`, the `Disk` variant is quite a bit larger
than `Memory` with the increased size of `Event` (233 vs 25 bytes).
Clippy recommends boxing this, but that is beyond the scope of this PR.
* Add `metadata_mut` accessors to event types
* Rename `EventStatus::NoOp` to `Recorded`
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* enhancement(http sink): Add support for end-to-end acknowledgements (#7265)
* Create `EventRef` wrapper for references to event inner types
* Rework MetricsBuffer to use the Metric type internally
* Merge some common bits in http sink tests
* Add tests for retries and failures to http sink
* Add Batch wrapper to store metadata
* Use metadata batch for batch sinks
* Update event metadata in `ServiceSink`
* Add metadata output to `trait HttpSink::encode_event`
* Handle metadata in `BatchSink` wrapper
* Modify http sink to pass along metadata
* Add batch status asserts to all HTTP tests
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* chore(ci): Fix deb verify steps (#7385)
They verify installation in a container without systemd running so the
`systemctl daemon-reload` step fails.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* feat(remap transform): Emitting multiple events (#7267)
Add support to the `remap` transform to emit multiple events by setting `.` to an array.
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
* enhancement(remap): Add parse_klog function (#7349)
Closes: #6793
This is very similar to parse_glog but I think a separate function is
still prudent in-case the continue to diverge.
The biggeste differences are just spacing and the fact that timestamps
don't have a year in the…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #6676
Given an array, the slice remap function won't necessarily know at compile time what indexes are being returned. So the returned type definition should be a collection of all the possible subtypes contained within that array. This means if the array contains a mixture of types, we can no longer determine the exact type definition of a given position in the array following that function.
Signed-off-by: Stephen Wakely fungus.humungus@gmail.com