Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Date partitioned table error #173

Open
meoooh opened this issue Mar 20, 2019 · 1 comment
Open

Date partitioned table error #173

meoooh opened this issue Mar 20, 2019 · 1 comment

Comments

@meoooh
Copy link

meoooh commented Mar 20, 2019

Environments

  • fluentd version: 1.4.0
  • plugin version: 2.1.0

Configuration

<match applog.**>
  @type copy

  <store>
    @type stdout
  </store>

  <store>
    @type bigquery_load
    project tibo
    auth_method application_default
    dataset "mylog#{ENV['RAILS_ENV'] != 'production' ? '_alpha' : ''}"
    request_timeout_sec 300
    request_open_timeout_sec 300
    flush_interval 1800
    ignore_unknown_values true
    auto_create_table true
    time_partitioning_type day

    schema_path /fluentd/etc/bq_request_scheme.json

    <buffer>
      flush_at_shutdown true
      flush_interval 1
      timekey 60
    </buffer>

    <inject>
      localtime true
    </inject>

    table request$%Y%m%d
  </store>
</match>

Expected Behavior

I don't get error

Actual Behavior

I get some error

Log (if you have)

2019-03-20 12:38:21 +0000 [error]: fluent/log.rb:362:error: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Parameter 'bigquery_insert: project=tibo/dataset=mylog_alpha/table=request$%Y%m%d/fetch_schema_table=/template_suffix=' has timestamp placeholders, but chunk key 'time' is not configured"
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:639:in `validate_time!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:624:in `validate!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:574:in `block in placeholder_validate!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:573:in `each'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:573:in `placeholder_validate!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-bigquery-2.1.0/lib/fluent/plugin/out_bigquery_insert.rb:61:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin.rb:164:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/multi_output.rb:74:in `block in configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/multi_output.rb:63:in `each'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/multi_output.rb:63:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/out_copy.rb:36:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin.rb:164:in `configure'

I think some bug...?

Because str is project=tibo/dataset=mylog_alpha/table=request$%Y%m%d/fetch_schema_table=/template_suffix= in that context.

And str must be passed this logic, but str never passd that logic.

So can I use %d placeholder in project=tibo/dataset=mylog_alpha/table=request$%Y%m%d/fetch_schema_table=/template_suffix= ?

@joker1007
Copy link
Collaborator

Your configuration seems not correct.
Please add time to <buffer> chunk key.

like this.

<buffer time>
# 
</buffer>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants