Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): Improve vector generate syntax and formatting #1773

Merged
merged 3 commits into from
Feb 12, 2020

Conversation

Jeffail
Copy link
Contributor

@Jeffail Jeffail commented Feb 11, 2020

This PR changes the vector generate syntax to allow forward slashes: foo/bar,baz/boo, closing #1684. This is backwards compatible as the pipe syntax can still be used.

The output now also adds empty lines between component types to make the formatting nicer:

data_dir = "/var/lib/vector/"
dns_servers = []

[sources.source0]
max_length = 102400
type = "stdin"

[transforms.transform0]
inputs = ["source0"]
drop_field = true
drop_invalid = false
type = "json_parser"

[sinks.sink0]
healthcheck = true
inputs = ["transform0"]
type = "console"

[sinks.sink0.buffer]
type = "memory"
max_events = 500
when_full = "block"

This is the lowest hanging fruit of #1685, we're still missing:

  1. Proper ordering of fields (inputs first, followed by type, and so on)
  2. Nicer indentation
  3. All required and common fields for all components

But this stuff would require a fair amount of time investment, and I'd like to have a plan/spec for #1735 before we go ahead with it. I'll update #1685 accordingly.

Signed-off-by: Ashley Jeffs <ash@jeffail.uk>
Signed-off-by: Ashley Jeffs <ash@jeffail.uk>
src/generate.rs Outdated Show resolved Hide resolved
@Hoverbear
Copy link
Contributor

@Jeffail Are docs updates coming in another PR?

@Hoverbear
Copy link
Contributor

Looks so much better!

$ cargo run -- generate stdin,kafka/add_fields,remove_fields,json_parser,json_parser/console
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/vector generate stdin,kafka/add_fields,remove_fields,json_parser,json_parser/console`
data_dir = "/var/lib/vector/"
dns_servers = []

[sources.source0]
max_length = 102400
type = "stdin"

[sources.source1]
type = "kafka"

[transforms.transform0]
inputs = ["source0", "source1"]
type = "add_fields"

[transforms.transform1]
inputs = ["transform0"]
type = "remove_fields"

[transforms.transform2]
inputs = ["transform1"]
drop_field = true
drop_invalid = false
type = "json_parser"

[transforms.transform3]
inputs = ["transform2"]
drop_field = true
drop_invalid = false
type = "json_parser"

[sinks.sink0]
healthcheck = true
inputs = ["transform3"]
type = "console"

[sinks.sink0.buffer]
type = "memory"
max_events = 500
when_full = "block"

@Jeffail
Copy link
Contributor Author

Jeffail commented Feb 11, 2020

@Jeffail Are docs updates coming in another PR?

Not for this one, not sure there's any mention of generate on the site yet (won't be the case soon, after my configs guide)

@Hoverbear
Copy link
Contributor

Ok! :)

Signed-off-by: Ashley Jeffs <ash@jeffail.uk>
@Jeffail Jeffail merged commit 2535012 into master Feb 12, 2020
@Jeffail Jeffail deleted the generate-subcmd-improvements branch February 12, 2020 10:32
bednar pushed a commit to bonitoo-io/vector that referenced this pull request Feb 17, 2020
…dev#1773)

* Explicitly add spacing between component types

Signed-off-by: Ashley Jeffs <ash@jeffail.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants