Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

escape hell in_forward or record transformer #1736

Closed
epcim opened this issue Nov 3, 2017 · 9 comments
Closed

escape hell in_forward or record transformer #1736

epcim opened this issue Nov 3, 2017 · 9 comments

Comments

@epcim
Copy link

epcim commented Nov 3, 2017

Fluentd or td-agent version.
td-agent, 0.14.20

Environment information, e.g. OS.
container, k8s, https://github.com/Mirantis/k8s-apps/tree/master/charts/fluentd

Configuration
First, I don't know relation to in_forward, as timestamp attribute, I set later, by the record_transformer plugin. Assuming "timestamp" is native time attribute of fluent interface, this issue can be also related to fluentbit that may do an escaping (I use - 0.12.4) as the record can be broken at source.

Config on fluentd

         <source>
   29
   30                 @type forward
   31
   32                 @id input_forward_24224
   33
   34                 @label @MAINSTREAM
   35
   36                 port 24224
   37
   38             </source>


...

 <filter ** >
   77
   78                     # strip record, keep only req related metrics to store in influxdb
   79
   80                         @type record_transformer
   81
   82                         renew_record false
   83
   84                         enable_ruby true
   85
   86                         <record>
   87
   88                             timestamp ${dollar}{time.strftime("%s")}
+  89                             tag metrics.req.${dollar}{tag}
   90
   91                         </record>
   92
   93                 </filter>

The "incoming" record got messy in escape hell. We fail somewhere and remove a "t" from timestamp attribute.

2017-11-03 09:45:47 +0000 [error]: #0 [input_forward_24224] unexpected error on reading data host="192.168.237.64" port=47306 error_cla
ss=Yajl::ParseError error="lexical error: inside a string, '\\' occurs before a character which it may not.\n           {\"imestamp\\\"
:\":\"\\\"\\\\\\\"\\\\\\\\\\\\\\{\"imestamp\\\":\":\"\\\"\\\\\\\"\\\\\\\\\\\\\\\"\n                     (right here) ------^\n"
  2017-11-03 09:45:47 +0000 [error]: #0 suppressed same stacktrace

On another occurence I see jsut after the above outpout:

  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:242:in `<<'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:242:in `block (2
levels) in read_messages'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in
read_messages'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_rea
d_without_connection'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.5.1/lib/cool.io/io.rb:123:in `on_readable'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.5.1/lib/cool.io/io.rb:186:in `on_readable'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.5.1/lib/cool.io/loop.rb:88:in `run_once'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.5.1/lib/cool.io/loop.rb:88:in `run'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `blo
ck in start'
  2017-11-03 09:39:51 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block i
n thread_create'
2017-11-03 09:40:01 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_
stream` method.
2017-11-03 09:40:01 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin
::RecordTransformerFilter] uses `#filter_stream` method.

full log

te_exception", "reason"=>"Can't get text on a START_OBJECT at 1:70"}}}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"[3/6024]
, "_id"=>"AV-BRSbJNXGAynmgWe40", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>2, "failed"=>0}, "cr[2/6024]
true, "status"=>201}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe41", "status"=[1/6024]
rror"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"type"=>"illegal_state_exception"[0/6024]
n"=>"Can't get text on a START_OBJECT at 1:70"}}}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXG
AynmgWe42", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"type"=>"ill
egal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:70"}}}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"
fluentd", "_id"=>"AV-BRSbJNXGAynmgWe43", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]
", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:70"}}}}, {"index"=>{"_index"=>"log
stash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe44", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "succ
essful"=>2, "failed"=>0}, "created"=>true, "status"=>201}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-
BRSbJNXGAynmgWe45", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"typ
e"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:70"}}}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_
type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe46", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>2, "failed"=
>0}, "created"=>true, "status"=>201}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe47", "
_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>2, "failed"=>0}, "created"=>true, "status"=>201}}, {"index"=>{
"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe48", "_version"=>1, "result"=>"created", "_shards"=>{"to
tal"=>2, "successful"=>2, "failed"=>0}, "created"=>true, "status"=>201}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd
", "_id"=>"AV-BRSbJNXGAynmgWe49", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "cau
sed_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:100"}}}}, {"index"=>{"_index"=>"logstash-
2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe4-", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>
"failed to parse [time]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:100"}}}}, {
"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe4_", "status"=>400, "error"=>{"type"=>"mapper_
parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a
 START_OBJECT at 1:100"}}}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BRSbJNXGAynmgWe5A", "status"=>4
00, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"type"=>"illegal_state_exception",
"reason"=>"Can't get text on a START_OBJECT at 1:70"}}}}, {"index"=>{"_index"=>"logstash-2017.11.03", "_type"=>"fluentd", "_id"=>"AV-BR
SbJNXGAynmgWe5B", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"type"
=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:70"}}}}]}
2017-11-03 09:45:47 +0000 [error]: #0 [input_forward_24224] unexpected error on reading data host="192.168.237.64" port=47306 error_cla
ss=Yajl::ParseError error="lexical error: inside a string, '\\' occurs before a character which it may not.\n           {\"imestamp\\\"
:\":\"\\\"\\\\\\\"\\\\\\\\\\\\\\{\"imestamp\\\":\":\"\\\"\\\\\\\"\\\\\\\\\\\\\\\"\n                     (right here) ------^\n"
  2017-11-03 09:45:47 +0000 [error]: #0 suppressed same stacktrace
@epcim epcim mentioned this issue Nov 3, 2017
Closed
@okkez
Copy link
Contributor

okkez commented Nov 6, 2017

@epcim Could you paste your full configuration?

@epcim
Copy link
Author

epcim commented Nov 6, 2017

First of all I have similar:

2017-11-06 14:43:08 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2017-11-06 14:43:09 +0000 ch[346/9051]
753cc079ba455ed2f2bcd63ad4b" error_class=InfluxDB::Error error="{\"error\":\"partial write: unable to parse 'os_req_metrics,os_service=
neutron,tag=metrics.req.openstack.neutron.server path=\\\"/var/log/neutron/neutron-server.log\\\",statement=\\\"INSERT INTO ml2_vxlan_e
ndpoints (ip_address, udp_port, host) VALUES ('10.60.3.150', 4789, NULL)\\\",[1509979325\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\
\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:05.367\\\\\\\"\\\
\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26307\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"WARNING\\\\\\\"\\\\,\\
\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.plugins.ml2.drivers.type_tunnel\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u
003e\\\\\\\"req-f0d32b0d-3fdc-4eaf-a043-4cefd07e0d30\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\
\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\
\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\
\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"Endpoint\\\\ with\\\\ ip\\\\ 10.60.3.150\\\\ already\\\\ exists:\\\\ DBDuplicateEntry:\\\\
 (psycopg2.IntegrityError)\\\\ duplicate\\\\ key\\\\ value\\\\ violates\\\\ unique\\\\ constraint\\\\ \\\\\\\\\\\"ml2_vxlan_endpoints_p
key\\\\\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message_extra\\\\\\\"\\\\=\\u003e\\\\\\\"[SQL:\\\\ 'INSERT\\\\ INTO\\\\ ml2_vxlan_endpoints\\\
\ (ip_address\\\\,\\\\ udp_port\\\\,\\\\ host)\\\\ VALUES\\\\ (%(ip_address)s\\\\,\\\\ %(udp_port)s\\\\,\\\\ %(host)s)']\\\\ [parameter
s:\\\\ {'host':\\\\ None\\\\,\\\\ 'ip_address':\\\\ u'10.60.3.150'\\\\,\\\\ 'udp_port':\\\\ 4789}]\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\
\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979325, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-ser
ver.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:05.842\\\", \\\"pid\\\"=\\u003e\\\"26304\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\
", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"
=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\
\", \\\"message\\\"=\\u003e\\\"(26304) accepted ('10.60.2.83', 38924)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.cor
p\\\"}],[1509979325\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time
\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:05.990\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26304\\\\\\\"\\\\,\\\\
\\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\
,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-dc540930-32e6-478f-b8ff-741ea47f506d\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=
\\u003e\\\\\\\"4387e8a36f4346a493f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"8668275e0cfa443abd2ff53
21d5ab5a7\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e
\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e
\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:05]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/ports.json?device_id\\\\=88ec6012-d6a1-42a7
-ae32-4746752a6090\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1197\\\\ 0.146809\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"
control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"
=\\u003e\\\"2017-11-06 14:42:06.353\\\", \\\"pid\\\"=\\u003e\\\"26304\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u0
03e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req
_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u00
3e\\\"(26304) accepted ('10.60.2.83', 38940)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979326\\\\,\
\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\
\\"2017-11-06\\\\ 14:42:06.427\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26304\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\
\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\
\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=
\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\
u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u0
03e\\\\\\\"(26304)\\\\ accepted\\\\ ('10.60.2.83'\\\\,\\\\ 38945)\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-
node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e
\\\"2017-11-06 14:42:06.446\\\", \\\"pid\\\"=\\u003e\\\"26304\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"n
eutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"req-28db3c08-b6f9-4f23-991c-880c27a7416e\\\", \\\"req_user\\\"=\\u003e\\\"4387e8a36f4346a493
f2f51e32fddc10\\\", \\\"req_project\\\"=\\u003e\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\", \\\"req_domain\\\"=\\u003e\\\"-\\\", \\\"req_u
ser_domain\\\"=\\u003e\\\"-\\\", \\\"req_project_domain\\\"=\\u003e\\\"-\\\", \\\"message\\\"=\\u003e\\\"10.60.2.83 - - [06/Nov/2017 14
:42:06] \\\\\\\"GET /v2.0/security-groups.json?id=e0985b55-d85d-4033-81aa-e6eb483cf3fd\\u0026id=a53e0856-72eb-483c-a5e1-737af21e1a7b HT
TP/1.1\\\\\\\" 200 8923 0.017877\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979326\\\\,\\\\ {\\\\\\\
"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-0
6\\\\ 14:42:06.638\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26304\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\
\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u0
03e\\\\\\\"req-3e6302a5-ab9e-43d4-91d6-64a22086e196\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"6ab8310bd78949f1bdac96
cdc723ef30\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"6cddc212e9af4b3e94fa6ec3540aadb9\\\\\\\"\\\\,\\\\ \\\\\\\"re
q_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"re
q_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [0
6/Nov/2017\\\\ 14:42:06]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/ports.json?tenant_id\\\\=5d02edd2865545e29efc557e15dd2071\\u0026device_id\\\\=ac
ceffe5-5c0e-46e1-9616-efe747188f02\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1105\\\\ 0.283914\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\
=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\
\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:06.667\\\", \\\"pid\\\"=\\u003e\\\"26298\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"co
mponent\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\
\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"m
essage\\\"=\\u003e\\\"(26298) accepted ('10.60.2.83', 38950)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[
1509979326\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\
\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:06.888\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26298\\\\\\\"\\\\,\\\\ \\\\\\\"s
everity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\
\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-86e5581d-28c9-45c3-a8a5-9426b535930f\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\
\\\\\"6ab8310bd78949f1bdac96cdc723ef30\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"6cddc212e9af4b3e94fa6ec3540aadb9
\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-
\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"1
0.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:06]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/networks.json?id\\\\=f3324603-6052-49b1-a6ab-d85d007
c3c05\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 609\\\\ 0.220265\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.
lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"20
17-11-06 14:42:06.955\\\", \\\"pid\\\"=\\u003e\\\"26299\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron
.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\
u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26299)
accepted ('10.60.2.83', 38968)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979327\\\\,\\\\ {\\\\\\\"p
ath\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\
\\\ 14:42:07.126\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26299\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\
\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003
e\\\\\\\"req-95c32f49-6c8a-417a-a1a7-1923c6a4c273\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"6ab8310bd78949f1bdac96cd
c723ef30\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"6cddc212e9af4b3e94fa6ec3540aadb9\\\\\\\"\\\\,\\\\ \\\\\\\"req_
domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_
project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/
Nov/2017\\\\ 14:42:07]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/subnets.json?id\\\\=0d815ea3-e2d8-45ec-bc42-a79c4b30ecf7\\\\ HTTP/1.1\\\\\\\\\\\"\
\\\ 200\\\\ 689\\\\ 0.170397\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[
1509979327, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:07.730\\\", \\\"
pid\\\"=\\u003e\\\"26299\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u00
3e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_dom
ain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26299) accepted ('10.60.2.83', 38995)\
\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979327\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"
/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:07.881\\\\\\\"\\\\,\\
\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26299\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\
\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-c049e506-0a9c-4297
-8e64-91c9f73cf6d3\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"4387e8a36f4346a493f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\
\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\
\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u0
03e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:07]\\\\ \\\\
\\\\\\\"GET\\\\ /v2.0/ports.json?device_id\\\\=680767b7-209d-4fcd-aa2e-3622333d9a42\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1055\\\\ 0.150
045\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979329, {\\\"path\\\"
=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:09.099\\\", \\\"pid\\\"=\\u003e\\\"26305\
\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\
\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\",
\\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26305) accepted ('10.60.2.83', 39027)\\\", \\\"hostname\\\"=\\u
003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979329\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-
server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:09.121\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\
\=\\u003e\\\\\\\"26305\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\
=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-8acac77d-0f08-4e0f-abea-ab3213229700\\\\\\\
"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"4387e8a36f4346a493f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\
\=\\u003e\\\\\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\
\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,
\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:09]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/por
ts.json?device_id\\\\=fa85f476-0f93-4ac2-9271-a0f198da5a55\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1126\\\\ 0.021264\\\\\\\"\\\\,\\\\ \\\\
\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979329, {\\\"path\\\"=\\u003e\\\"/var/log/neut
ron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:09.206\\\", \\\"pid\\\"=\\u003e\\\"26268\\\", \\\"severity\\\"=\\u
003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"
req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\
"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26268) accepted ('10.60.2.83', 39030)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab
42b.zzz.xxx.corp\\\"}],[1509979329\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\
\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:09.361\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26268\\\
\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.w
sgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-d3f2eac3-f88b-4c12-b8f8-0515e5a8be2e\\\\\\\"\\\\,\\\\ \\\\\\\"req_us
er\\\\\\\"\\\\=\\u003e\\\\\\\"d7b3e587ba814e9e81a89bc33b595aa8\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"d2abca61
fcef401da69cc7290885502a\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\
\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\
\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:09]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/networks.json?router%3Aexter
nal\\\\=True\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1756\\\\ 0.154021\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"contro
l-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979329, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u00
3e\\\"2017-11-06 14:42:09.437\\\", \\\"pid\\\"=\\u003e\\\"26299\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\
"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domai
n\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"
(26299) accepted ('10.60.2.83', 39050)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979329\\\\,\\\\ {\
\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"201
7-11-06\\\\ 14:42:09.607\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26299\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\
u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\
\=\\u003e\\\\\\\"req-3c48c8da-90cb-4cc9-9123-04738b9caa10\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"4387e8a36f4346a4
93f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\\\\\"\\\\,\\\\ \\\\
\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\
\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\
\\\ [06/Nov/2017\\\\ 14:42:09]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/ports.json?device_id\\\\=1f4552c1-7e1b-4fe6-9015-24195d10d0d2\\\\ HTTP/1.1
\\\\\\\\\\\"\\\\ 200\\\\ 1197\\\\ 0.168345\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.co
rp\\\\\\\"}]=[1509979330, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:10
.140\\\", \\\"pid\\\"=\\u003e\\\"26268\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"re
q_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\
\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26268) accepted ('10.60.
2.83', 39072)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979322': invalid boolean\\nunable to parse '
os_req_metrics,os_service=neutron,tag=metrics.req.openstack.neutron.server path=\\\"/var/log/neutron/neutron-server.log\\\",message_ext
ra=\\\"[SQL: 'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameter
s: {'host': None, 'ip_address': u'10.60.3.151', 'udp_port': 4789}]\\\",[1509979330\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/v
ar/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"statement\\\\\\\"\\\\=\\u003e\\\\\\\"INSERT\\\\ INTO\\\\ ml2_vxlan_endpoints
\\\\ (ip_address\\\\,\\\\ udp_port\\\\,\\\\ host)\\\\ VALUES\\\\ ('10.60.3.150'\\\\,\\\\ 4789\\\\,\\\\ NULL)\\\\\\\"\\\\,\\\\ [15099793
31\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u0
03e\\\\\\\"2017-11-06\\\\ 14:42:11.124\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26252\\\\\\\"\\\\,\\\\ \\\\\\\"severity\
\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req
_id\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\
\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\
"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\
\\\=\\u003e\\\\\\\"(26252)\\\\ accepted\\\\ ('10.60.2.83'\\\\,\\\\ 39111)\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"
control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]\\\\=\\u003e[1509979331\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/n
eutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:11.349\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\
\\\"\\\\=\\u003e\\\\\\\"26252\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\
\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-0bc71c82-f6c0-4ae6-ba0a-04ad71d56486
\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"d7b3e587ba814e9e81a89bc33b595aa8\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\
\\\"\\\\=\\u003e\\\\\\\"d2abca61fcef401da69cc7290885502a\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\
\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\
\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:11]\\\\ \\\\\\\\\\\"GET\\\\ /v
2.0/subnets.json?fields\\\\=id\\u0026id\\\\=792ad441-3531-4675-b735-4fe0601bdb81\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 269\\\\ 0.224254\
\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]}]=[1509979332, {\\\"path\\\"=\
\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:12.075\\\", \\\"pid\\\"=\\u003e\\\"26280\\\
", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\
"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\
\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26280) accepted ('10.60.2.83', 39147)\\\", \\\"hostname\\\"=\\u00
3e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979330': invalid boolean\\nunable to parse 'os_req_metrics,os_service=neutron,tag=met
rics.req.openstack.neutron.server 51=46i,49=53i,48=39i,44=32i,39=117i,100=112i,95=112i,111=114i,116=39i,58=32i,52=55i,56=57i,125=93i,[1
509979366\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\
\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:46.343\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26249\\\\\\\"\\\\,\\\\ \\\\\\\"se
verity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\
\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_proje
ct\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain
\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message\\
\\\\\"\\\\=\\u003e\\\\\\\"(26249)\\\\ accepted\\\\ ('10.60.2.83'\\\\,\\\\ 40230)\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\
\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979366, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"t
ime\\\"=\\u003e\\\"2017-11-06 14:42:46.515\\\", \\\"pid\\\"=\\u003e\\\"26249\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\
\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"req-b9541eff-0203-4963-bcbb-097c8b7b44a8\\\", \\\"req_user\\\"=\\u003e\\\"438
7e8a36f4346a493f2f51e32fddc10\\\", \\\"req_project\\\"=\\u003e\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\", \\\"req_domain\\\"=\\u003e\\\"-
\\\", \\\"req_user_domain\\\"=\\u003e\\\"-\\\", \\\"req_project_domain\\\"=\\u003e\\\"-\\\", \\\"message\\\"=\\u003e\\\"10.60.2.83 - -
[06/Nov/2017 14:42:46] \\\\\\\"GET /v2.0/ports.json?device_id=8d19ea2d-dfcc-46ef-9e92-b1bdecdd4430\\u0026device_id=88ec6012-d6a1-42a7-a
e32-4746752a6090\\u0026device_id=680767b7-209d-4fcd-aa2e-3622333d9a42\\u0026device_id=915cebbd-78e1-4c84-a4f2-f8d4968b6c0a\\u0026device
_id=1f4552c1-7e1b-4fe6-9015-24195d10d0d2\\u0026device_id=fa85f476-0f93-4ac2-9271-a0f198da5a55 HTTP/1.1\\\\\\\" 200 5584 0.171379\\\", \
\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979365': invalid boolean dropped=0\"}\n"
  2017-11-06 14:43:08 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/influxdb-0.5.1/lib/influxdb/client/http.rb:84:in `resolve_error'
2017-11-06 14:43:08 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2017-11-06 14:43:09 +0000 ch[346/9051]
753cc079ba455ed2f2bcd63ad4b" error_class=InfluxDB::Error error="{\"error\":\"partial write: unable to parse 'os_req_metrics,os_service=
neutron,tag=metrics.req.openstack.neutron.server path=\\\"/var/log/neutron/neutron-server.log\\\",statement=\\\"INSERT INTO ml2_vxlan_e
ndpoints (ip_address, udp_port, host) VALUES ('10.60.3.150', 4789, NULL)\\\",[1509979325\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\
\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:05.367\\\\\\\"\\\
\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26307\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"WARNING\\\\\\\"\\\\,\\
\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.plugins.ml2.drivers.type_tunnel\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u
003e\\\\\\\"req-f0d32b0d-3fdc-4eaf-a043-4cefd07e0d30\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\
\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\
\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\
\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"Endpoint\\\\ with\\\\ ip\\\\ 10.60.3.150\\\\ already\\\\ exists:\\\\ DBDuplicateEntry:\\\\
 (psycopg2.IntegrityError)\\\\ duplicate\\\\ key\\\\ value\\\\ violates\\\\ unique\\\\ constraint\\\\ \\\\\\\\\\\"ml2_vxlan_endpoints_p
key\\\\\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message_extra\\\\\\\"\\\\=\\u003e\\\\\\\"[SQL:\\\\ 'INSERT\\\\ INTO\\\\ ml2_vxlan_endpoints\\\
\ (ip_address\\\\,\\\\ udp_port\\\\,\\\\ host)\\\\ VALUES\\\\ (%(ip_address)s\\\\,\\\\ %(udp_port)s\\\\,\\\\ %(host)s)']\\\\ [parameter
s:\\\\ {'host':\\\\ None\\\\,\\\\ 'ip_address':\\\\ u'10.60.3.150'\\\\,\\\\ 'udp_port':\\\\ 4789}]\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\
\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979325, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-ser
ver.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:05.842\\\", \\\"pid\\\"=\\u003e\\\"26304\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\
", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"
=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\
\", \\\"message\\\"=\\u003e\\\"(26304) accepted ('10.60.2.83', 38924)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.cor
p\\\"}],[1509979325\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time
\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:05.990\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26304\\\\\\\"\\\\,\\\\
\\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\
,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-dc540930-32e6-478f-b8ff-741ea47f506d\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=
\\u003e\\\\\\\"4387e8a36f4346a493f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"8668275e0cfa443abd2ff53
21d5ab5a7\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e
\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e
\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:05]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/ports.json?device_id\\\\=88ec6012-d6a1-42a7
-ae32-4746752a6090\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1197\\\\ 0.146809\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"
control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"
=\\u003e\\\"2017-11-06 14:42:06.353\\\", \\\"pid\\\"=\\u003e\\\"26304\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u0
03e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req
_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u00
3e\\\"(26304) accepted ('10.60.2.83', 38940)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979326\\\\,\
\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\
\\"2017-11-06\\\\ 14:42:06.427\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26304\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\
\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\
\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=
\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\
u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u0
03e\\\\\\\"(26304)\\\\ accepted\\\\ ('10.60.2.83'\\\\,\\\\ 38945)\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-
node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e
\\\"2017-11-06 14:42:06.446\\\", \\\"pid\\\"=\\u003e\\\"26304\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"n
eutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"req-28db3c08-b6f9-4f23-991c-880c27a7416e\\\", \\\"req_user\\\"=\\u003e\\\"4387e8a36f4346a493
f2f51e32fddc10\\\", \\\"req_project\\\"=\\u003e\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\", \\\"req_domain\\\"=\\u003e\\\"-\\\", \\\"req_u
ser_domain\\\"=\\u003e\\\"-\\\", \\\"req_project_domain\\\"=\\u003e\\\"-\\\", \\\"message\\\"=\\u003e\\\"10.60.2.83 - - [06/Nov/2017 14
:42:06] \\\\\\\"GET /v2.0/security-groups.json?id=e0985b55-d85d-4033-81aa-e6eb483cf3fd\\u0026id=a53e0856-72eb-483c-a5e1-737af21e1a7b HT
TP/1.1\\\\\\\" 200 8923 0.017877\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979326\\\\,\\\\ {\\\\\\\
"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-0
6\\\\ 14:42:06.638\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26304\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\
\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u0
03e\\\\\\\"req-3e6302a5-ab9e-43d4-91d6-64a22086e196\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"6ab8310bd78949f1bdac96
cdc723ef30\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"6cddc212e9af4b3e94fa6ec3540aadb9\\\\\\\"\\\\,\\\\ \\\\\\\"re
q_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"re
q_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [0
6/Nov/2017\\\\ 14:42:06]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/ports.json?tenant_id\\\\=5d02edd2865545e29efc557e15dd2071\\u0026device_id\\\\=ac
ceffe5-5c0e-46e1-9616-efe747188f02\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1105\\\\ 0.283914\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\
=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\
\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:06.667\\\", \\\"pid\\\"=\\u003e\\\"26298\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"co
mponent\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\
\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"m
essage\\\"=\\u003e\\\"(26298) accepted ('10.60.2.83', 38950)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[
1509979326\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\
\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:06.888\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26298\\\\\\\"\\\\,\\\\ \\\\\\\"s
everity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\
\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-86e5581d-28c9-45c3-a8a5-9426b535930f\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\
\\\\\"6ab8310bd78949f1bdac96cdc723ef30\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"6cddc212e9af4b3e94fa6ec3540aadb9
\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-
\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"1
0.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:06]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/networks.json?id\\\\=f3324603-6052-49b1-a6ab-d85d007
c3c05\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 609\\\\ 0.220265\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.
lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979326, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"20
17-11-06 14:42:06.955\\\", \\\"pid\\\"=\\u003e\\\"26299\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron
.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\
u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26299)
accepted ('10.60.2.83', 38968)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979327\\\\,\\\\ {\\\\\\\"p
ath\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\
\\\ 14:42:07.126\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26299\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\
\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003
e\\\\\\\"req-95c32f49-6c8a-417a-a1a7-1923c6a4c273\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"6ab8310bd78949f1bdac96cd
c723ef30\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"6cddc212e9af4b3e94fa6ec3540aadb9\\\\\\\"\\\\,\\\\ \\\\\\\"req_
domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_
project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/
Nov/2017\\\\ 14:42:07]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/subnets.json?id\\\\=0d815ea3-e2d8-45ec-bc42-a79c4b30ecf7\\\\ HTTP/1.1\\\\\\\\\\\"\
\\\ 200\\\\ 689\\\\ 0.170397\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[
1509979327, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:07.730\\\", \\\"
pid\\\"=\\u003e\\\"26299\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u00
3e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_dom
ain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26299) accepted ('10.60.2.83', 38995)\
\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979327\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"
/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:07.881\\\\\\\"\\\\,\\
\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26299\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\
\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-c049e506-0a9c-4297
-8e64-91c9f73cf6d3\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"4387e8a36f4346a493f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\
\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\
\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u0
03e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:07]\\\\ \\\\
\\\\\\\"GET\\\\ /v2.0/ports.json?device_id\\\\=680767b7-209d-4fcd-aa2e-3622333d9a42\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1055\\\\ 0.150
045\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979329, {\\\"path\\\"
=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:09.099\\\", \\\"pid\\\"=\\u003e\\\"26305\
\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\
\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\",
\\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26305) accepted ('10.60.2.83', 39027)\\\", \\\"hostname\\\"=\\u
003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979329\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-
server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:09.121\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\
\=\\u003e\\\\\\\"26305\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\
=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-8acac77d-0f08-4e0f-abea-ab3213229700\\\\\\\
"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"4387e8a36f4346a493f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\
\=\\u003e\\\\\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\
\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,
\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:09]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/por
ts.json?device_id\\\\=fa85f476-0f93-4ac2-9271-a0f198da5a55\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1126\\\\ 0.021264\\\\\\\"\\\\,\\\\ \\\\
\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979329, {\\\"path\\\"=\\u003e\\\"/var/log/neut
ron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:09.206\\\", \\\"pid\\\"=\\u003e\\\"26268\\\", \\\"severity\\\"=\\u
003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"
req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\
"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26268) accepted ('10.60.2.83', 39030)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab
42b.zzz.xxx.corp\\\"}],[1509979329\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\
\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:09.361\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26268\\\
\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.w
sgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-d3f2eac3-f88b-4c12-b8f8-0515e5a8be2e\\\\\\\"\\\\,\\\\ \\\\\\\"req_us
er\\\\\\\"\\\\=\\u003e\\\\\\\"d7b3e587ba814e9e81a89bc33b595aa8\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"d2abca61
fcef401da69cc7290885502a\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\
\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\
\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:09]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/networks.json?router%3Aexter
nal\\\\=True\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 1756\\\\ 0.154021\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"contro
l-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979329, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u00
3e\\\"2017-11-06 14:42:09.437\\\", \\\"pid\\\"=\\u003e\\\"26299\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\
"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domai
n\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"
(26299) accepted ('10.60.2.83', 39050)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}],[1509979329\\\\,\\\\ {\
\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"201
7-11-06\\\\ 14:42:09.607\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26299\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\
u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\
\=\\u003e\\\\\\\"req-3c48c8da-90cb-4cc9-9123-04738b9caa10\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"4387e8a36f4346a4
93f2f51e32fddc10\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\\\"\\\\=\\u003e\\\\\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\\\\\"\\\\,\\\\ \\\\
\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\
\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\
\\\ [06/Nov/2017\\\\ 14:42:09]\\\\ \\\\\\\\\\\"GET\\\\ /v2.0/ports.json?device_id\\\\=1f4552c1-7e1b-4fe6-9015-24195d10d0d2\\\\ HTTP/1.1
\\\\\\\\\\\"\\\\ 200\\\\ 1197\\\\ 0.168345\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.co
rp\\\\\\\"}]=[1509979330, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:10
.140\\\", \\\"pid\\\"=\\u003e\\\"26268\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"re
q_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\
\"req_user_domain\\\"=\\u003e\\\"\\\", \\\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26268) accepted ('10.60.
2.83', 39072)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979322': invalid boolean\\nunable to parse '
os_req_metrics,os_service=neutron,tag=metrics.req.openstack.neutron.server path=\\\"/var/log/neutron/neutron-server.log\\\",message_ext
ra=\\\"[SQL: 'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameter
s: {'host': None, 'ip_address': u'10.60.3.151', 'udp_port': 4789}]\\\",[1509979330\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/v
ar/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"statement\\\\\\\"\\\\=\\u003e\\\\\\\"INSERT\\\\ INTO\\\\ ml2_vxlan_endpoints
\\\\ (ip_address\\\\,\\\\ udp_port\\\\,\\\\ host)\\\\ VALUES\\\\ ('10.60.3.150'\\\\,\\\\ 4789\\\\,\\\\ NULL)\\\\\\\"\\\\,\\\\ [15099793
31\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u0
03e\\\\\\\"2017-11-06\\\\ 14:42:11.124\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26252\\\\\\\"\\\\,\\\\ \\\\\\\"severity\
\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req
_id\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\\
\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain\\\\\\\
"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\
\\\=\\u003e\\\\\\\"(26252)\\\\ accepted\\\\ ('10.60.2.83'\\\\,\\\\ 39111)\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"
control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]\\\\=\\u003e[1509979331\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/n
eutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:11.349\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\
\\\"\\\\=\\u003e\\\\\\\"26252\\\\\\\"\\\\,\\\\ \\\\\\\"severity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\
\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"req-0bc71c82-f6c0-4ae6-ba0a-04ad71d56486
\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"d7b3e587ba814e9e81a89bc33b595aa8\\\\\\\"\\\\,\\\\ \\\\\\\"req_project\\\\
\\\"\\\\=\\u003e\\\\\\\"d2abca61fcef401da69cc7290885502a\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\
\\\ \\\\\\\"req_user_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"-\\\\\\
\"\\\\,\\\\ \\\\\\\"message\\\\\\\"\\\\=\\u003e\\\\\\\"10.60.2.83\\\\ -\\\\ -\\\\ [06/Nov/2017\\\\ 14:42:11]\\\\ \\\\\\\\\\\"GET\\\\ /v
2.0/subnets.json?fields\\\\=id\\u0026id\\\\=792ad441-3531-4675-b735-4fe0601bdb81\\\\ HTTP/1.1\\\\\\\\\\\"\\\\ 200\\\\ 269\\\\ 0.224254\
\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]}]=[1509979332, {\\\"path\\\"=\
\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"time\\\"=\\u003e\\\"2017-11-06 14:42:12.075\\\", \\\"pid\\\"=\\u003e\\\"26280\\\
", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"\\\", \\\"req_user\\\
"=\\u003e\\\"\\\", \\\"req_project\\\"=\\u003e\\\"\\\", \\\"req_domain\\\"=\\u003e\\\"\\\", \\\"req_user_domain\\\"=\\u003e\\\"\\\", \\
\"req_project_domain\\\"=\\u003e\\\"\\\", \\\"message\\\"=\\u003e\\\"(26280) accepted ('10.60.2.83', 39147)\\\", \\\"hostname\\\"=\\u00
3e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979330': invalid boolean\\nunable to parse 'os_req_metrics,os_service=neutron,tag=met
rics.req.openstack.neutron.server 51=46i,49=53i,48=39i,44=32i,39=117i,100=112i,95=112i,111=114i,116=39i,58=32i,52=55i,56=57i,125=93i,[1
509979366\\\\,\\\\ {\\\\\\\"path\\\\\\\"\\\\=\\u003e\\\\\\\"/var/log/neutron/neutron-server.log\\\\\\\"\\\\,\\\\ \\\\\\\"time\\\\\\\"\\
\\=\\u003e\\\\\\\"2017-11-06\\\\ 14:42:46.343\\\\\\\"\\\\,\\\\ \\\\\\\"pid\\\\\\\"\\\\=\\u003e\\\\\\\"26249\\\\\\\"\\\\,\\\\ \\\\\\\"se
verity\\\\\\\"\\\\=\\u003e\\\\\\\"INFO\\\\\\\"\\\\,\\\\ \\\\\\\"component\\\\\\\"\\\\=\\u003e\\\\\\\"neutron.wsgi\\\\\\\"\\\\,\\\\ \\\\
\\\"req_id\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_proje
ct\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_user_domain
\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"req_project_domain\\\\\\\"\\\\=\\u003e\\\\\\\"\\\\\\\"\\\\,\\\\ \\\\\\\"message\\
\\\\\"\\\\=\\u003e\\\\\\\"(26249)\\\\ accepted\\\\ ('10.60.2.83'\\\\,\\\\ 40230)\\\\\\\"\\\\,\\\\ \\\\\\\"hostname\\\\\\\"\\\\=\\u003e\
\\\\\\"control-node1.lab42b.zzz.xxx.corp\\\\\\\"}]=[1509979366, {\\\"path\\\"=\\u003e\\\"/var/log/neutron/neutron-server.log\\\", \\\"t
ime\\\"=\\u003e\\\"2017-11-06 14:42:46.515\\\", \\\"pid\\\"=\\u003e\\\"26249\\\", \\\"severity\\\"=\\u003e\\\"INFO\\\", \\\"component\\
\"=\\u003e\\\"neutron.wsgi\\\", \\\"req_id\\\"=\\u003e\\\"req-b9541eff-0203-4963-bcbb-097c8b7b44a8\\\", \\\"req_user\\\"=\\u003e\\\"438
7e8a36f4346a493f2f51e32fddc10\\\", \\\"req_project\\\"=\\u003e\\\"8668275e0cfa443abd2ff5321d5ab5a7\\\", \\\"req_domain\\\"=\\u003e\\\"-
\\\", \\\"req_user_domain\\\"=\\u003e\\\"-\\\", \\\"req_project_domain\\\"=\\u003e\\\"-\\\", \\\"message\\\"=\\u003e\\\"10.60.2.83 - -
[06/Nov/2017 14:42:46] \\\\\\\"GET /v2.0/ports.json?device_id=8d19ea2d-dfcc-46ef-9e92-b1bdecdd4430\\u0026device_id=88ec6012-d6a1-42a7-a
e32-4746752a6090\\u0026device_id=680767b7-209d-4fcd-aa2e-3622333d9a42\\u0026device_id=915cebbd-78e1-4c84-a4f2-f8d4968b6c0a\\u0026device
_id=1f4552c1-7e1b-4fe6-9015-24195d10d0d2\\u0026device_id=fa85f476-0f93-4ac2-9271-a0f198da5a55 HTTP/1.1\\\\\\\" 200 5584 0.171379\\\", \
\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979365': invalid boolean dropped=0\"}\n"
  2017-11-06 14:43:08 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/influxdb-0.5.1/lib/influxdb/client/http.rb:84:in `resolve_error'
....

seems to me we don't excape ' inside the SQL statement.

2.83', 39072)\\\", \\\"hostname\\\"=\\u003e\\\"control-node1.lab42b.zzz.xxx.corp\\\"}] 1509979322': invalid boolean\\nunable to parse '
os_req_metrics,os_service=neutron,tag=metrics.req.openstack.neutron.server path=\\\"/var/log/neutron/neutron-server.log\\\",message_ext
ra=\\\"[SQL: 'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameter
s: {'host': None, 'ip_address': u'10.60.3.151', 'udp_port': 4789}]\\\",[1509979330\\\\,\\\\

@epcim
Copy link
Author

epcim commented Nov 6, 2017

This is my config

2017-11-06 14:11:10 +0000 [info]: parsing config file is succeeded path="/etc/fluent/fluent.conf"
2017-11-06 14:11:11 +0000 [info]: using configuration file: <ROOT>
  <system>
    log_level info
    process_name "fluentd_logs"
  </system>
  <source>
    @type forward
    @id input_forward_24224
    @label @MAINSTREAM
    port 24224
  </source>
  <source>
    @type monitor_agent
    bind "0.0.0.0"
    port 24220
  </source>
  <label @OPENSTACKREQMETRICS>
    <filter *>
      @type grep
      <regexp>
        key "req_ttr"
        pattern ^[0-9][.]\d*
      </regexp>
      <exclude>
        key "req_ttr"
        pattern 0
      </exclude>
    </filter>
    <filter *>
      @type record_transformer
      renew_record false
      enable_ruby true
      <record>
        ttimestamp ${time.strftime("%s")}
      </record>
    </filter>
    <filter *>
      @type record_transformer
      renew_record true
      renew_time_key "time"
      keep_keys "[\"tag\",\"hostname\",\"os_service\",\"os_service_component\",\"time\",\"ttimestamp\",\"req_status\",\"req_ttr\",\"req_len\",\"req_method\"]"
    </filter>
    <filter openstack.** openstack.*.*>
      @type record_transformer
      renew_record false
      <record>
        tag metrics.req.${tag}
      </record>
    </filter>
    <match * *.* *.*.* openstack.* openstack.*.* openstack.** metrics.**>
      @type influxdb
      host "influx-cis-influx"
      port 8087
      dbname "cis_os_req_metrics"
      measurement "os_req_metrics"
      use_ssl false
      time_key "ttimestamp"
      time_precision "s"
      tag_keys ["tag","hostname","os_service","os_service_component","req_project","req_project_domain"]
    </match>
  </label>
  <label @OPENSTACK>
    <filter openstack.**>
      @type record_transformer
      <record>
        os_service ${tag_parts[1]}
      </record>
    </filter>
    <filter openstack.* openstack.** openstack.*.* keystone.** nova.** neutron.** cinder.** heat.** glance.** ceilometer.** trove.** ironic.** sahara.**>
      @type parser
      reserve_data true
      emit_invalid_record_to_error false
      inject_key_prefix "req_"
      format /((?<method>GET|POST|PUT|DELETE|CONNECT|OPTIONS|HEAD[^ ]\w+)\s+(?<path>[^ ]+))["'\s+]*((?<protocol>[^ ][hftpsHFTPS\/.\d]+)){0,1}["'\s]*((status:\s*){0,1}(?<status>[^ ]\d+)\s*(len:\s*){0,1}(?<len>[^ ]\d+)\s+(time:\s*){0,1}(?<ttr>[^ ][.\d]+)){0,1}(.*)$/
      key_name "message"
      <parse>
        @type regexp
        expression ((?<method>GET|POST|PUT|DELETE|CONNECT|OPTIONS|HEAD[^ ]\w+)\s+(?<path>[^ ]+))["'\s+]*((?<protocol>[^ ][hftpsHFTPS\/.\d]+)){0,1}["'\s]*((status:\s*){0,1}(?<status>[^ ]\d+)\s*(len:\s*){0,1}(?<len>[^ ]\d+)\s+(time:\s*){0,1}(?<ttr>[^ ][.\d]+)){0,1}(.*)$
      </parse>
    </filter>
    <filter *>
      @type record_modifier
      char_encoding "utf-8"
    </filter>
    <match * *.* *.*.* openstack.* openstack.*.* openstack.**>
      @type copy
      deep_copy true
      <store>
        @type "relabel"
        @label @OPENSTACKREQMETRICS
      </store>
      <store>
        @type "elasticsearch"
        host "elasticsearch-client-cis-elastic"
        port 9200
        index_name "fluent-openstack"
        type_name "fluentd"
        ssl_verify false
        logstash_format true
        flush_interval 10s
        tag_key "tag"
        include_tag_key true
        <buffer tag>
          flush_mode interval
          retry_type exponential_backoff
          flush_interval 10s
        </buffer>
        <inject>
          tag_key tag
        </inject>
      </store>
    </match>
  </label>
  <label @MAINSTREAM>
    <match openstack.*.*  openstack.*  openstack.** keystone.** nova.** neutron.** cinder.** heat.** glance.** ceilometer.** trove.** ironic.** sahara.**>
      @type relabel
      @label @OPENSTACK
    </match>
    <match *>
      @type elasticsearch
      @id output_other
      host "elasticsearch-client-cis-elastic"
      port 9200
      index_name "fluent-other"
      type_name "fluentd"
      ssl_verify false
      logstash_format true
      flush_interval 10s
      tag_key "tag"
      include_tag_key true
      time_as_integer true
      <buffer tag>
        flush_mode interval
        retry_type exponential_backoff
        flush_interval 10s
      </buffer>
      <inject>
        tag_key tag
      </inject>
    </match>
  </label>
  <match *>
    @type elasticsearch
    @id output_nolabel
    host "elasticsearch-client-cis-elastic"
    port 9200
    index_name "fluentbit-other-nolabel"
    type_name "fluentd"
    ssl_verify false
    logstash_format true
    flush_interval 10s
    tag_key "tag"
    include_tag_key true
    time_as_integer true
    <buffer tag>
      flush_mode interval
      retry_type exponential_backoff
      flush_interval 10s
    </buffer>
    <inject>
      tag_key tag
    </inject>
  </match>
</ROOT>
2017-11-06 14:11:11 +0000 [info]: starting fluentd-0.14.20 pid=1
2017-11-06 14:11:11 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "--under-supervisor"]
2017-11-06 14:11:11 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.10.2'
2017-11-06 14:11:11 +0000 [info]: gem 'fluent-plugin-influxdb' version '1.0.0.rc1'
2017-11-06 14:11:11 +0000 [info]: gem 'fluent-plugin-mail' version '0.2.4'
2017-11-06 14:11:11 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2017-11-06 14:11:11 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.0.0.rc1'
2017-11-06 14:11:11 +0000 [info]: gem 'fluentd' version '0.14.20'
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACKREQMETRICS pattern="*" type="grep"
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACKREQMETRICS pattern="*" type="record_transformer"
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACKREQMETRICS pattern="*" type="record_transformer"
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACKREQMETRICS pattern="openstack.** openstack.*.*" type="record_transformer"
2017-11-06 14:11:11 +0000 [info]: adding match in @OPENSTACKREQMETRICS pattern="* *.* *.*.* openstack.* openstack.*.* openstack.** metrics.**" type="influxdb"
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACK pattern="openstack.**" type="record_transformer"
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACK pattern="openstack.* openstack.** openstack.*.* keystone.** nova.** neutron.** cinder.** heat.** glance.** ceilometer.** trove.** ironic.** sahara.**" type="parser"
2017-11-06 14:11:11 +0000 [info]: adding filter in @OPENSTACK pattern="*" type="record_modifier"
2017-11-06 14:11:12 +0000 [info]: adding match in @OPENSTACK pattern="* *.* *.*.* openstack.* openstack.*.* openstack.**" type="copy"
2017-11-06 14:11:12 +0000 [info]: adding match in @MAINSTREAM pattern="openstack.*.*  openstack.*  openstack.** keystone.** nova.** neutron.** cinder.** heat.** glance.** ceilometer.** trove.** ironic.** sahara.**" type="relabel"
2017-11-06 14:11:12 +0000 [info]: adding match in @MAINSTREAM pattern="*" type="elasticsearch"
2017-11-06 14:11:12 +0000 [info]: adding match pattern="*" type="elasticsearch"
2017-11-06 14:11:12 +0000 [info]: adding source type="forward"
2017-11-06 14:11:12 +0000 [info]: adding source type="monitor_agent"
2017-11-06 14:11:12 +0000 [info]: #0 starting fluentd worker pid=10 ppid=1 worker=0
2017-11-06 14:11:12 +0000 [info]: #0 Connecting to database: cis_os_req_metrics, host: influx-cis-influx, port: 8087, username: root, use_ssl = false, verify_ssl = true
2017-11-06 14:11:12 +0000 [info]: #0 [input_forward_24224] listening port port=24224 bind="0.0.0.0"
2017-11-06 14:11:12 +0000 [info]: #0 fluentd worker is now running worker=0
2017-11-06 14:11:50 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:11:50 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:11:50 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:12:00 +0000 [info]: #0 Connection opened to Elasticsearch cluster => {:host=>"elasticsearch-client-cis-elastic", :port=>9200, :scheme=>"http"}
2017-11-06 14:12:10 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:16:55 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:17:10 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:17:10 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2017-11-06 14:38:03 +0000 [warn]: #0 dump an error event: error_class=NoMethodError error="undefined method `merge!' for #<Array:0x000055f65f06a580>" tag="openstack.nova.novncproxy" time=#<Fluent::EventTime:0x000055f65f062628 @sec=1509979083, @nsec=183347484> record=[117, 116, 114, 111, 110, 47, 110, 101, 117, 116, 114, 111, 110, 45, 115, 101, 114, 118, 101, 114, 46, 108, 111, 103, "time", "2017-11-06 14:38:02.858", "pid", "26306", "severity", "INFO", "component"]
...
...

@epcim
Copy link
Author

epcim commented Nov 6, 2017

There is ~30% chance that it appears only if FLUENTD_OPT='-v' it will take me a while to proof as breaking records don't come that frequently.

@okkez
Copy link
Contributor

okkez commented Nov 7, 2017

Some types of errors are found in you log.

  1. unexpected error on reading data host="192.168.237.64" port=47306 error_class=Yajl::ParseError error="lexical error: inside a string
  2. error_class=InfluxDB::Error error="{\"error\":\"partial write: unable to parse ... invalid boolean dropped=0

Both error 1. and 2. may be caused by messy escaped data. But I could not find the reason that such data incoming.

You can insert following snippet to check incoming data :

<filter>
  @stdout
  format none
</filter>

For example, we want to know that incoming data has already been escaped:

# ...
<label @MAINSTREAM>
  <filter>
    @type stdout
    format none
  </filter>
  # ...
</label>
# ...

If incoming data has already been broken, you should check your logging driver.

@epcim
Copy link
Author

epcim commented Nov 7, 2017

Sorry , but I don';t see such option https://docs.fluentd.org/v0.12/articles/out_stdout
I get the error that format none is not know plugin.

@okkez
Copy link
Contributor

okkez commented Nov 7, 2017

Sorry, could you try https://docs.fluentd.org/v0.14/articles/filter_stdout ?

<filter>
  @type stdout
</filter>

Default configuration is enough to debug, I think.

@epcim
Copy link
Author

epcim commented Nov 7, 2017

For now, I can't reproduce. Will provide you the logs. I have them actually. For now I can say that the escapes probably don't come from in_forward. But are somehow created while processing in fluentd

@epcim epcim mentioned this issue Nov 8, 2017
Open
@epcim
Copy link
Author

epcim commented Nov 23, 2017

I did not manage to reproduce it on latest versions of fluentbit/fluentd and used plugins.
No longer a problem.

@epcim epcim closed this as completed Nov 23, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@okkez @epcim