multiline parser could not load

[vijaykumar219]

Bug Report

Describe the bug
Hi
I need to configure multiline parsing for my k8s env application. "Multiline could not load" error getting

Fluent Bit v2.0.6

• Copyright (C) 2015-2022 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

[2022/12/14 04:34:02] [error] [lib] backend failed
[2022/12/14 04:34:02] [ info] [fluent bit] version=2.0.6, commit=211f841fe2, pid=1
[2022/12/14 04:34:02] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/12/14 04:34:02] [ info] [cmetrics] version=0.5.7
[2022/12/14 04:34:02] [ info] [ctraces ] version=0.2.5
[2022/12/14 04:34:02] [ info] [input:tail:tail.0] initializing
[2022/12/14 04:34:02] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2022/12/14 04:34:02] [error] [multiline] parser 'multiline-regex-test' not registered
[2022/12/14 04:34:02] [error] [input:tail:tail.0] could not load multiline parsers
[2022/12/14 04:34:02] [error] failed initialize input tail.0
[2022/12/14 04:34:02] [error] [engine] input initialization failed

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

2022-12-14 04:43:36.266 UTC DEBUG -  -  -  - Compiled stored procedure. Call string is [{call acct_sync.enq_event_append(?, ?, ?, ?)}]
2022-12-14 04:43:36.266 UTC DEBUG -  -  -  - RdbmsOperation with SQL [acct_sync.enq_event_append] compiled
2022-12-14 04:43:36.266 UTC DEBUG -  -  -  - Calling stored procedure [{call acct_sync.deq_event_rc(?, ?, ?, ?, ?)}]
2022-12-14 04:43:36.266 UTC DEBUG -  -  -  - Fetching JDBC Connection from DataSource
2022-12-14 04:43:36.280 UTC DEBUG -  -  -  - Looking up default SQLErrorCodes for DataSource [org.springframework.jdbc.datasource.SingleConnectionDataSource@22b2df2]
2022-12-14 04:43:36.280 UTC DEBUG -  -  -  - Fetching JDBC Connection from DataSource
2022-12-14 04:43:36.280 UTC DEBUG -  -  -  - SQL error codes for 'Oracle' found
2022-12-14 04:43:36.280 UTC DEBUG -  -  -  - Caching SQL error codes for DataSource [org.springframework.jdbc.datasource.SingleConnectionDataSource@22b2df2]: database product name is 'Oracle'
2022-12-14 04:43:36.280 UTC DEBUG -  -  -  - Unable to translate SQLException with Error code '25226', will now try the fallback translator
2022-12-14 04:43:36.280 UTC DEBUG -  -  -  - Extracted SQL state class '99' from value '99999'
2022-12-14 04:43:36.280 UTC ERROR -  -  -  - Error connecting to Database
org.springframework.jdbc.UncategorizedSQLException: CallableStatementCallback; uncategorized SQLException for SQL [{call acct_sync.deq_event_rc(?, ?, ?, ?, ?)}]; SQL state [99999]; error code [25226]; ORA-25226: dequeue failed, queue PIN05.IFW_SYNC_QUEUE is not enabled for dequeue
ORA-06512: at "SYS.DBMS_AQ", line 786
ORA-06512: at "PIN05.ACCT_SYNC", line 255
ORA-06512: at line 1
; nested exception is java.sql.SQLException: ORA-25226: dequeue failed, queue PIN05.IFW_SYNC_QUEUE is not enabled for dequeue
ORA-06512: at "SYS.DBMS_AQ", line 786
ORA-06512: at "PIN05.ACCT_SYNC", line 255
ORA-06512: at line 1

        at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1542) ~[spring-jdbc-5.3.11.jar:5.3.11]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1206) ~[spring-jdbc-5.3.11.jar:5.3.11]
        at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1245) ~[spring-jdbc-5.3.11.jar:5.3.11]
        at org.springframework.jdbc.object.StoredProcedure.execute(StoredProcedure.java:138) ~[spring-jdbc-5.3.11.jar:5.3.11]
        at oracle.communication.brm.charging.migration.customer.updater.internal.aq.BRMEventDequeueProcedure.getMessage(BRMEventDequeueProcedure.java:98) ~[ece-migration-customer.jar:?]
        at oracle.communication.brm.charging.migration.customer.updater.internal.aq.AQClient.receiveMessages(AQClient.java:243) [ece-migration-customer.jar:?]
        at oracle.communication.brm.charging.migration.customer.updater.internal.UpdateEventHandler.run(UpdateEventHandler.java:226) [ece-migration-customer.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
Caused by: java.sql.SQLException: ORA-25226: dequeue failed, queue PIN05.IFW_SYNC_QUEUE is not enabled for dequeue
ORA-06512: at "SYS.DBMS_AQ", line 786
ORA-06512: at "PIN05.ACCT_SYNC", line 255
ORA-06512: at line 1

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079) ~[ojdbc8.jar:12.2.0.1.0]
        at org.springframework.jdbc.core.JdbcTemplate.lambda$call$6(JdbcTemplate.java:1246) ~[spring-jdbc-5.3.11.jar:5.3.11]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1190) ~[spring-jdbc-5.3.11.jar:5.3.11]
        ... 6 more
Caused by: oracle.jdbc.OracleDatabaseException: ORA-25226: dequeue failed, queue PIN05.IFW_SYNC_QUEUE is not enabled for dequeue
ORA-06512: at "SYS.DBMS_AQ", line 786
ORA-06512: at "PIN05.ACCT_SYNC", line 255
ORA-06512: at line 1

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:498) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230) ~[ojdbc8.jar:12.2.0.1.0]
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079) ~[ojdbc8.jar:12.2.0.1.0]
        at org.springframework.jdbc.core.JdbcTemplate.lambda$call$6(JdbcTemplate.java:1246) ~[spring-jdbc-5.3.11.jar:5.3.11]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1190) ~[spring-jdbc-5.3.11.jar:5.3.11]
        ... 6 more
2022-12-14 04:43:36.280 UTC  WARN -  -  -  - An exception occured while trying to BRMUpdateEventList. Reconnecting to database....Attempt 41102

• Steps to reproduce the problem: Configuration of environment below

Expected behavior

A single line output in grafana from multiline to single line.
Screenshots

Your Environment

• Version used: helm chart (fluent/fluent-bit 0.21.5 2.0.6)

• Configuration:

• apiVersion: v1
  data:
  fluent-bit.conf: |
  [SERVICE]
  Daemon Off
  Flush 1
  Log_Level info
  Parsers_File parsers_multiline.conf
  HTTP_Server On
  HTTP_Listen 0.0.0.0
  HTTP_Port 2020
  Health_Check On

  [INPUT]
  Name tail
  Path /var/log/containers/.log
  Tag kube.
  read_from_head true
  multiline.parser docker, cri, multiline-regex-test
  Buffer_Chunk_Size 32k
  Buffer_Max_Size 32k
  Mem_Buf_Limit 50MB
  Skip_Long_Lines off

  [FILTER]
  Name kubernetes
  Match kube.*
  Kube_URL https://kubernetes.default.svc:443
  Merge_Log On
  Keep_Log Off
  K8S-Logging.Parser On
  K8S-Logging.Exclude On

  [OUTPUT]
  Name loki
  Match kube.*
  Host loki-loki-distributed-distributor.loki-grafana.svc.cluster.local
  tenant_id ""
  labels job=fluentbit
  remove_keys kubernetes,stream
  line_format json
  auto_kubernetes_labels off
  parsers_multiline.conf: |
  [MULTILINE_PARSER]
  name multiline-regex-test
  type regex
  flush_timeout 1000

  rules | state name | regex pattern | next state name

  --------|----------------|--------------------------------------------------

  rule "start_state" "/(([\d-]+ [\d:,]+))(.)/" "cont"
  rule "cont" "/^\s+at./" "cont"

• Environment name and version (e.g. Kubernetes? What version?): kubernetes v1.19.15

• Server type and version:

• Operating System and version:

• Filters and plugins: fluentbit ->loki(2.6) ->grafana(9.3.1)

Additional context

I tried to set multiline.parser multiline-regex-test and it provides an error "multiline regex not registered"

[RicardoAAD]

Hello @vijaykumar219,
Your config looks good, but the reason why you are getting this error is due to the missing comments in your multiline parser

it looks like this:

[MULTILINE_PARSER]
name multiline-regex-test
type regex
flush_timeout 1000

rules | state name | regex pattern | next state name
--------|----------------|--------------------------------------------------
rule "start_state" "/(([\d-]+ [\d:,]+))(.)/" "cont"
rule "cont" "/^\s+at./" "cont"

But should be

        name multiline-regex-test
        type regex
        flush_timeout 1000
        # rules    | state name        | regex pattern                | next state name
        #--------|----------------|------------------------|--------------------------
        rule       "start_state"   "/(([\d-]+ [\d:,]+))(.)/"      "cont"
        rule       "cont"          "/^\s+at./"                    "cont"

These two lines are not needed; they were only added to simplify the definition of rules Rules Definition

rules | state name | regex pattern | next state name
--------|----------------|--------------------------------------------------

Please let us know if, after commenting or removing these lines, the multiline parser works for you.

Regards,
Ricardo

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.