Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help with my log format #1667

Open
rgarrigue opened this issue Feb 10, 2020 · 1 comment
Open

Help with my log format #1667

rgarrigue opened this issue Feb 10, 2020 · 1 comment

Comments

@rgarrigue
Copy link

@rgarrigue rgarrigue commented Feb 10, 2020

Hi there

I'm trying to parse this king of lines, from a python flask service whose log format is %(asctime)s [%(process)d] (%(levelname)s) (%(name)s): %(message)s

2020-02-10 13:58:38,594 [31383] (INFO) (flask.app): request: OPTIONS https://server_hostname/0.1/token/a_big_uuid {'Host': 'server_hostname', 'X-Script-Name': '/api/auth', 'X-Forwarded-For': 'an_IP_address', 'Connection': 'close', 'Accept': '*/*', 'Access-Control-Request-Method': 'DELETE', 'Origin': 'https://client_hostname', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'cross-site', 'Referer': 'https://client_hostname/', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7'}
2020-02-10 13:58:38,601 [31383] (INFO) (flask.app): request: OPTIONS https://server_hostname/0.1/token/another_big_uuid {'Host': 'website_host', 'X-Script-Name': '/api/auth', 'X-Forwarded-For': 'an_IP_address', 'Connection': 'close', 'Accept': '*/*', 'Access-Control-Request-Method': 'DELETE', 'Origin': 'https://client_hostname', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'cross-site', 'Referer': 'https://client_hostname/', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7'}
2020-02-10 13:58:38,609 [31383] (INFO) (flask.app): request: DELETE https://website_host/0.1/token/a_big_uuid {'Host': 'website_host', 'X-Script-Name': '/api/auth', 'X-Forwarded-For': 'an_IP_address', 'Connection': 'close', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36', 'Sec-Fetch-Dest': 'empty', 'Accept': '*/*', 'Origin': 'https://client_hostname', 'Sec-Fetch-Site': 'cross-site', 'Sec-Fetch-Mode': 'cors', 'Referer': 'https://client_hostname/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7'}
2020-02-10 13:58:38,619 [31383] (INFO) (flask.app): request: DELETE https://website_host/0.1/token/another_big_uuid {'Host': 'website_host', 'X-Script-Name': '/api/auth', 'X-Forwarded-For': 'an_IP_address', 'Connection': 'close', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36', 'Sec-Fetch-Dest': 'empty', 'Accept': '*/*', 'Origin': 'https://client_hostname', 'Sec-Fetch-Site': 'cross-site', 'Sec-Fetch-Mode': 'cors', 'Referer': 'https://client_hostname/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7'}

So far I got this

goaccess --date-format "%F" --time-format "%T,%s" --log-format "%d %t [%^] (%^) (%^): request: %m %U {'Host': '%e', 'X-Script-Name': '%^', 'X-Forwarded-For': '~h', 'Connection': '%^', 'Accept': '*/*', 'Access-Control-Request-Method': '%^', 'Origin': '%v', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'cross-site', 'Referer': '%R', 'User-Agent': '%u', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7'}" requests.log

But the answer is Missing braces '{}' and ignore chars for specifier '%h'. I'm a bit at a loss, fiddling with randomly with combinations atm.

If I may, the https://goaccess.io/man#custom-log doc is lacking a complex parsing chain example, so one can grasp the way to skip fields, deal with static values etc .

Best regards,

@rgarrigue

This comment has been minimized.

Copy link
Author

@rgarrigue rgarrigue commented Feb 10, 2020

So, just don't bother for me. The logs are too irregular to be dealt with. But the last remark about the doc lacking a full example still make sense imho.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.