Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC5424 structured data parsing #845

Closed
cybermedi opened this issue Jan 6, 2015 · 15 comments
Closed

RFC5424 structured data parsing #845

cybermedi opened this issue Jan 6, 2015 · 15 comments
Assignees
Labels
Milestone

Comments

@cybermedi
Copy link

@cybermedi cybermedi commented Jan 6, 2015

Base on Jochen recommendation https://groups.google.com/forum/#!searchin/graylog2/5424/graylog2/KO91vcZIOXo/vcEPEwAT6e4J

I would like to create issue:

I have an application which creates log in RFC5424 and send them to my central rsyslog server. Logs are resend to graylog2 via syslog protocol. Graylog2 runs local INPUT tcp syslog on 10514.
Configuration of rsyslog resending looks like:

$template GRAYLOGRFC5424,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %STRUCTURED-DATA% %msg%\n"
*.* @@172.100.100.100:10514;GRAYLOGRFC5424

all logs are received by graylog2 without any issue but they are not parsed properly. They are parsed like basic RFC5424 message. Structured data are handled like "message".

the full message looks like:

<190>1 2015-01-06T20:56:33.287Z app-1 app - [mdc@18060 ip="::ffff:132.123.15.30" logger="{c.corp.Handler}" session="4ot7"  user="cybermedi@yahoo.com" user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/7.1.2 Safari/537.85.11"] User page 13 requested

I recieve message fields like:
application_name
app

facility
local7

level
Info [6]

message
[mdc@18060 ip="::ffff:132.123.15.30" logger="{c.corp.Handler}" session="4ot7" user="cybermedi@yahoo.com" user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/7.1.2 Safari/537.85.11"] User page 13 requested

source
app-1

but there should be other ones based on structured data.
Like:
ip, logger, session, user, user-agent
and in the message should be just:
User page 13 requested

@lennartkoopmann
Copy link
Member

@lennartkoopmann lennartkoopmann commented Jan 7, 2015

Which Graylog2 version are you running? @bernd fixed something regarding this recently.

@cybermedi
Copy link
Author

@cybermedi cybermedi commented Jan 7, 2015

Hi Lennart,
I have 0.92.3 (102a59f) (Moose)
Regards,
Martin

    1. 2015 v 2:47, Lennart Koopmann notifications@github.com:

Which Graylog2 version are you running? @bernd fixed something regarding this recently.


Reply to this email directly or view it on GitHub.

@dfch
Copy link

@dfch dfch commented Jan 11, 2015

I can confirm this behaviour (especially when you pass a NILVALUE (-), this is also treated as part of the message) (also running 0.92.3). In addition, the field MSGID seems to be treated as part of the message as well.

Message on the wire:

<128>1 2015-01-11T16:35:21.335797+01:00 s000000.dfch.biz - - - - tralala

... will have Graylog2 recognise the message field as:

- - tralala

See 6.5. Examples for an example.

Note: the VERSION field is not recognised or extracted at all.

I used modified syslog sender for testing.

@bernd
Copy link
Member

@bernd bernd commented Jan 13, 2015

@cybermedi There are several issues unfortunately.

  1. Your rsyslog template is wrong. The %MSGID% is missing. See our docs for a template example. (our example was also wrong until yesterday...)
  2. There are two spaces after the "session" key in your example. Is that a copy & paste error? If not, rsyslog might generate a structured data block that is not valid according to the RFC.
  3. The syslog parser we are using does not support whitespace in structured data values like the "user-agent" one.

I am currently investigating if we can fix the whitespace-in-value parsing issue in our syslog parser.

@bernd bernd added this to the 1.0 milestone Jan 13, 2015
@bernd bernd added the bug label Jan 13, 2015
@bernd
Copy link
Member

@bernd bernd commented Jan 13, 2015

@dfch This also seems to be an issue with the syslog parser. As said in the previous comment, I will investigate.

@bernd bernd added the processing label Jan 13, 2015
@cybermedi
Copy link
Author

@cybermedi cybermedi commented Jan 14, 2015

OK I added msgid to template. And it seems that that there is still something wrong with parsing. Structured data was not parsed, and the message begins with "-" (i think it is the msgid). In my message I have app-name I do not have procid and msgid, that's why raw messages contains:
"app - - [mdc@18060...".
The message beings:
"- [mdc@18060..."
what is interesting that %STRUCTURED-DATA% when they are empty are sometimes replaced by space and sometimes not. But finally the message contains two or three spaces between msgid and msg
ex:

sshd 4988 - Accepted public key for


sudo - - cybermedi : TTY=pts/0

OH I do not how to show it here, the first example has two spaces and the second three. Maybe I will need to have two templates for messages with structured data and one without. Maybe this issue should be fixed in rsyslog.

bernd added a commit to graylog-labs/syslog4j-graylog2 that referenced this issue Jan 14, 2015
bernd added a commit to graylog-labs/syslog4j-graylog2 that referenced this issue Jan 14, 2015
Fixes parsing of structured data values that contain whitespace.
This addresses issues from Graylog2/graylog2-server#845.

Parser and test code from NessComputing/syslog4j@038caf2.
bernd added a commit that referenced this issue Jan 14, 2015
The new syslog4j version can handle whitespace in structured syslog
values.

Ref: #845
bernd added a commit that referenced this issue Jan 14, 2015
If there is structured stat in the syslog message, the structured data
and the message ID should not be in the "message" field.

There is still a problem if the syslog STRUCTURED-DATA field is a
NILVALUE ("-"). Then the message plus the "-" for the nil value will end
up in the "message" field. This needs to be solved in the syslog4j
library.

Refs: #845
@bernd
Copy link
Member

@bernd bernd commented Jan 14, 2015

@cybermedi The next beta (beta.2) will have a fix for the structured data parsing.

It will also remove the structured data from the message field.

@dfch If there is a NILVALUE for the structured data like you posted, the "- -" will still be in the message field. This is a shortcoming in the syslog parser which we cannot fix for 1.0 unfortunately.

See #872 for the pull request.

I am closing this ticket now. Please re-open if you still have issues with the next beta. Thank you for the report!

@cybermedi
Copy link
Author

@cybermedi cybermedi commented Feb 12, 2015

Hi Bernd,
I deployed 1.0.0-9.rc.2.

  1. you mentioned the issue with parsing white space. Is it fixed?
  2. The structured data is correctly removed from "message"
  3. The structured data is still not parsed.
    I use the corrected template configuration:
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%n"
*.* @@10.19.10.1:5514;GRAYLOGRFC5424 # Graylog forwarding
@bernd
Copy link
Member

@bernd bernd commented Feb 12, 2015

Hey @cybermedi, thanks for the feedback!

Did you enable the "Expand structured data?" option on the input?

@cybermedi
Copy link
Author

@cybermedi cybermedi commented Feb 12, 2015

Hi, yes

Martin

    1. 2015 v 16:09, Bernd Ahlers notifications@github.com:

Hey @cybermedi, thanks for the feedback!

Did you enable the "Expand structured data?" option on the input?


Reply to this email directly or view it on GitHub.

@bernd
Copy link
Member

@bernd bernd commented Feb 12, 2015

There is an issue if the "Expand structured data?" option is enabled. Without this, the structured data is parsed.

Thanks for the report!

@bernd bernd reopened this Feb 12, 2015
bernd added a commit that referenced this issue Feb 12, 2015
This fixes structured syslog parsing with the "Expand structured data"
option enabled.

If enabled, the structured syslog message fields are prefixed by the
SD-ID which contains a "@" character.

Fixes #845.
@bernd
Copy link
Member

@bernd bernd commented Feb 12, 2015

I created a pull request in #965.

@cybermedi As a workaround you can disable the "Expand structured data" option.

@bernd
Copy link
Member

@bernd bernd commented Feb 12, 2015

@cybermedi The fix (#965) got merged and will be availablein an upcoming snapshot or RC.

@bernd bernd closed this Feb 12, 2015
@cybermedi
Copy link
Author

@cybermedi cybermedi commented Feb 13, 2015

thanx it works now.

@bernd
Copy link
Member

@bernd bernd commented Feb 13, 2015

Great, thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.