Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Reduce transform has unexpected behavior on arrays #3074

Closed
Hoverbear opened this issue Jul 15, 2020 · 2 comments · Fixed by #3076
Closed

bug: Reduce transform has unexpected behavior on arrays #3074

Hoverbear opened this issue Jul 15, 2020 · 2 comments · Fixed by #3076
Assignees
Labels
domain: transforms Anything related to Vector's transform components transform: reduce Anything `reduce` transform related type: bug A code related bug.

Comments

@Hoverbear
Copy link
Contributor

Hoverbear commented Jul 15, 2020

Given the file test.json with content:

{ "operations": ["insert", "select"] }
{ "operations": ["select"] }
{ "operations": ["select"] }
{ "operations": ["select"] }
{ "operations": ["select"] }
{ "operations": ["select"] }
{ "operations": ["select"] }
{ "operations": ["drop"], "transaction_state": "commit" }

And the configuration of:

data_dir = "var"

[sources.database_file]
  type = "file"
  include = ["test.json"]


[transforms.database_log]
  type = "json_parser"
  inputs = ["database_file"]

[transforms.reducer]
  type = "reduce"
  inputs = ["database_log"]
  merge_strategies."operations" = "concat"
  ends_when.type = "check_fields"
  ends_when."transaction_state.eq" = "commit"

[sinks.sink0]
  inputs = ["reducer"]
  type = "console"
  encoding = "json"

On latest main branch checkout, seems to produce unexpected output.

hoverbear@OBSIDIAN:/git/timberio/vector$ ./target/debug/vector --config test.toml 
Jul 15 13:07:36.188  INFO vector: Log level "info" is enabled.
Jul 15 13:07:36.191  INFO vector: Loading configs. path=["test.toml"]
Jul 15 13:07:36.204  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-341-g6358cac" released="Mon, 13 Jul 2020 20:39:08 +0000" arch="x86_64"
Jul 15 13:07:36.205  INFO vector::topology: Running healthchecks.
Jul 15 13:07:36.205  INFO vector::topology: Starting source "database_file"
Jul 15 13:07:36.205  INFO vector::topology::builder: Healthcheck: Passed.
Jul 15 13:07:36.206  INFO vector::topology: Starting transform "database_log"
Jul 15 13:07:36.206  INFO vector::topology: Starting transform "reducer"
Jul 15 13:07:36.206  INFO source{name=database_file type=file}: vector::sources::file: Starting file server. include=["test.json"] exclude=[]
Jul 15 13:07:36.206  INFO vector::topology: Starting sink "sink0"
Jul 15 13:07:36.208  INFO source{name=database_file type=file}:file_server: file_source::file_server: Found file to watch. path="test.json" file_position=0
{"file":"test.json","host":"OBSIDIAN","operations":["insert","select"],"source_type":"file","timestamp":"2020-07-15T20:07:36.223375960Z","timestamp_end":"2020-07-15T20:07:36.224779276Z","transaction_state":"commit"}

Expected output should have the final line as :

{"file":"test.json","host":"OBSIDIAN","operations":["insert","select","select","select","select","select","select","select","drop"],"source_type":"file","timestamp":"2020-07-15T20:07:36.223375960Z","timestamp_end":"2020-07-15T20:07:36.224779276Z","transaction_state":"commit"}
@Hoverbear Hoverbear added type: bug A code related bug. transform: reduce Anything `reduce` transform related labels Jul 15, 2020
@lukesteensen lukesteensen self-assigned this Jul 15, 2020
@Hoverbear
Copy link
Contributor Author

@lukesteensen Do you want to block #2961 on this?

@Hoverbear
Copy link
Contributor Author

Hoverbear commented Jul 15, 2020

Blocks #3022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: transforms Anything related to Vector's transform components transform: reduce Anything `reduce` transform related type: bug A code related bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants