Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix dry run mode #2651

Merged
merged 7 commits into from Nov 5, 2019
Merged

Fix dry run mode #2651

merged 7 commits into from Nov 5, 2019
+69 −78

Conversation

ganmacs
Copy link
Member

@ganmacs ganmacs commented Oct 15, 2019 •
edited

Which issue(s) this PR fixes:
ref #2624

What this PR does / why we need it:

Show better log

Example config

<system>
  workers 2
</system>

<worker 0>
  <source>
    @type fake
    secret very-secret-string0
    test 1
  </source>

  <match>
    @type null
  </match>
</worker>

<worker 1>
  <source>
    @type fake
    secret very-secret-string1
  </source>
  <match>
    @type null
  </match>
</worker>

normal mode

$ fluentd -c example/debug/secret.conf --dry-run

Before (supervisor and worker show same log about @type fake)

2019-10-15 15:16:44 +0900 [info]: parsing config file is succeeded path="example/debug/secret.conf"
2019-10-15 15:16:44 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:16:44 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:16:44 +0900 [info]: adding source type="fake"
2019-10-15 15:16:44 +0900 [info]: adding source type="fake"
2019-10-15 15:16:44 +0900 [info]: using configuration file: <ROOT>
  <system>
    workers 2
  </system>
  <worker 0>
    <source>
      @type fake
      secret xxxxxx
      test 1
    </source>
    <match>
      @type null
    </match>
  </worker>
  <worker 1>
    <source>
      @type fake
      secret xxxxxx
    </source>
    <match>
      @type null
    </match>
  </worker>
</ROOT>
2019-10-15 15:16:44 +0900 [warn]: parameter 'test' in <source>
  @type fake
  secret xxxxxx
  test 1
</source> is not used.
2019-10-15 15:16:44 +0900 [info]: starting fluentd-1.7.3 pid=91300 ruby="2.6.3"
2019-10-15 15:16:44 +0900 [info]: spawn command to main:  cmdline=["/Users/yuta.iwama/.rbenv/versions/2.6.3/bin/ruby", "-Eascii-8bit:ascii-8bit", "-rbundler/setup", "bin/fluentd", "-c", "example/debug/secret.conf", "-p", "plugins", "--under-supervisor"]
2019-10-15 15:16:45 +0900 [info]: #1 adding match pattern="**" type="null"
2019-10-15 15:16:45 +0900 [info]: gem 'fluentd' version '1.7.3'
2019-10-15 15:16:45 +0900 [info]: #0 adding match pattern="**" type="null"
2019-10-15 15:16:45 +0900 [info]: #0 adding source type="fake"
2019-10-15 15:16:45 +0900 [warn]: #0 parameter 'test' in <source>
  @type fake
  secret xxxxxx
  test 1
</source> is not used.

After(Removed duplicated log)

2019-10-15 15:16:44 +0900 [info]: parsing config file is succeeded path="example/debug/secret.conf"
2019-10-15 15:16:44 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:16:44 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:16:44 +0900 [info]: adding source type="fake"
2019-10-15 15:16:44 +0900 [info]: adding source type="fake"
2019-10-15 15:16:44 +0900 [info]: using configuration file: <ROOT>
  <system>
    workers 2
  </system>
  <worker 0>
    <source>
      @type fake
      secret xxxxxx
      test 1
    </source>
    <match>
      @type null
    </match>
  </worker>
  <worker 1>
    <source>
      @type fake
      secret xxxxxx
    </source>
    <match>
      @type null
    </match>
  </worker>
</ROOT>
2019-10-15 15:16:44 +0900 [warn]: parameter 'test' in <source>
  @type fake
  secret xxxxxx
  test 1
</source> is not used.

dry-run mode

$ fluentd -c example/debug/secret.conf --dry-run

Before

2019-10-15 15:16:07 +0900 [info]: parsing config file is succeeded path="example/debug/secret.conf"
2019-10-15 15:16:07 +0900 [info]: starting fluentd-1.7.3 as dry run mode ruby="2.6.3"
2019-10-15 15:16:07 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:16:07 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:16:07 +0900 [info]: adding source type="fake"
2019-10-15 15:16:07 +0900 [info]: adding source type="fake"
2019-10-15 15:16:07 +0900 [warn]: parameter 'test' in <source>
  @type fake
  secret xxxxxx
  test 1
</source> is not used.

After(show config)

2019-10-15 15:15:23 +0900 [info]: parsing config file is succeeded path="example/debug/secret.conf"
2019-10-15 15:15:23 +0900 [info]: starting fluentd-1.7.3 as dry run mode ruby="2.6.3"
2019-10-15 15:15:23 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:15:23 +0900 [info]: adding source type="fake"
2019-10-15 15:15:23 +0900 [info]: using configuration file: <ROOT>
  <system>
    workers 2
  </system>
  <worker 0>
    <source>
      @type fake
      secret xxxxxx
      test 1
    </source>
    <match>
      @type null
    </match>
  </worker>
  <worker 1>
    <source>
      @type fake
      secret xxxxxx
    </source>
    <match>
      @type null
    </match>
  </worker>
</ROOT>
2019-10-15 15:15:23 +0900 [warn]: parameter 'test' in <source>
  @type fake
  secret xxxxxx
  test 1
</source> is not used.

when config has an error

Example config 
<system>
  workers 2
</system>

<worker 0-1>
  <source>
    @type fake
    secret very-secret-string0
    test 1
  </source>

  <match>
    @type null
  </match>
</worker>

<worker 0-1>
  <source>
    @type fake
    secret very-secret-string1
  </source>

  <match>
    @type null
  </match>
</worker>

Before

2019-10-15 15:25:41 +0900 [info]: parsing config file is succeeded path="example/debug/secret.conf"
2019-10-15 15:25:41 +0900 [info]: starting fluentd-1.7.3 as dry run mode ruby="2.6.3"
2019-10-15 15:25:41 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:25:41 +0900 [info]: adding match pattern="**" type="null"
2019-10-15 15:25:41 +0900 [info]: adding source type="fake"
2019-10-15 15:25:41 +0900 [info]: adding source type="fake"
2019-10-15 15:25:41 +0900 [warn]: parameter 'test' in <source>
  @type fake
  secret xxxxxx
  test 1
</source> is not used.

After(be able to detect an error when dry-run)

2019-10-15 15:25:37 +0900 [info]: parsing config file is succeeded path="example/debug/secret.conf"
2019-10-15 15:25:37 +0900 [info]: starting fluentd-1.7.3 as dry run mode ruby="2.6.3"
2019-10-15 15:25:37 +0900 [error]: config error file="example/debug/secret.conf" error_class=Fluent::ConfigError error="specified worker_id<0> collisions is detected on <worker> directive. Available worker id(s): []"

Docs Changes:

no need

Release Note:

Make starting log better and seal dumped config

@ganmacs ganmacs self-assigned this Oct 15, 2019
repeatedly
repeatedly reviewed Oct 16, 2019
View reviewed changes
lib/fluent/supervisor.rb Outdated
ensure
Fluent::Engine.dry_run_mode = false
end
def configure_supervise
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configure_supervisor?

@ganmacs
Dump config if dry-run mode
5702b33 
This change had been introduced since
https://github.com/fluent/fluentd/pull/1580/files .

but even if dry-run mode, config need to show itself

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs
Add supervisor_mode to Engine class
6f0ae3b 
and if supervisor_mode, worker_id returns -1 because
we need to identify worker0 and supervisor

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs
unify dry_run method and make @dry_run immutable
a1d6f36 
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs
unify the condition
1089d34 
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs
rearrange code
6b64cfc 
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs
no need to assign dry_run to ivar
98e3034 
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs
Copy link
Member Author

ganmacs commented Oct 31, 2019 •
edited

I changed to fix #1576 at the same time. Could you take a look at this again? @repeatedly

2019-10-31 11:31:44 +0900 [info]: parsing config file is succeeded path="example/debug/rpc.conf"
2019-10-31 11:31:44 +0900 [info]: gem 'fluentd' version '1.8.0.rc2'
2019-10-31 11:31:44 +0900 [info]: using configuration file: <ROOT>
  <system>
    workers 2
    rpc_endpoint "127.0.0.1:9000"
  </system>
  <worker 0>
    <source>
      @type fake
      secret xxxxxx
    </source>
  </worker>
  <worker 1>
    <source>
      @type fake
      secret xxxxxx
    </source>
  </worker>
  <match>
    @type null
  </match>
</ROOT>

curl http://127.0.0.1:9000/api/config.dump                                                                                                        
{"ok":true}

2019-10-31 11:31:48 +0900 [info]: dump in-memory config
2019-10-31 11:31:48 +0900 [info]: <ROOT>
  <system>
    workers 2
    rpc_endpoint "127.0.0.1:9000"
  </system>
  <worker 0>
    <source>
      @type fake
      secret xxxxxx
    </source>
  </worker>
  <worker 1>
    <source>
      @type fake
      secret xxxxxx
    </source>
  </worker>
  <match>
    @type null
  </match>
</ROOT>

@ganmacs
fixed failed spec
4db63ce 
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@repeatedly repeatedly self-requested a review November 1, 2019 11:56
@ganmacs ganmacs merged commit e4a1dd7 into fluent:master Nov 5, 2019
@ganmacs ganmacs deleted the fix-dry-run-mode branch November 5, 2019 01:55
@cosmo0920 cosmo0920 mentioned this pull request Dec 3, 2019
Merged
7 tasks
@abetomo abetomo mentioned this pull request Feb 21, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@repeatedly repeatedly repeatedly approved these changes

Assignees

@ganmacs ganmacs

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ganmacs @repeatedly