Use url_pattern in web_log

[yasharne]

Hi,
I have a log format like this in my Nginx:

`$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_length $request_time "$http_referer" "$http_user_agent"`

I'm using web_log module (golang version) with this url_pattern to extract metrics per endpoint from the log (from the $request field:

    url_patterns:
      - name: /admin/v2/search
        match: '* /admin/v2/search'

Here is a sample log:

10.10.10.10 - - [30/May/2022:13:54:03 +0430] "GET /admin/v2/search HTTP/1.0" 302 412 847 0.051 "https://my.app/admin/v3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"

But as I look at the log rates and compare them to the metrics coming from the netdata, they are not even close.
I saw the docs and it says the patterns are matched against the URL field:

#  - url_patterns
#    Requests per URL pattern chart. Matches against URL field.
#    Matcher pattern syntax: https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format
#    Syntax:
#      url_patterns:
#        - name:  name1    # Name will be used as dimension.
#          match: pattern  # Pattern syntax: matcher.
#        - name:  name2
#          match: pattern

I'm not sure what URL field exactly is, can someone help me see if I have the correct configurations?

[ilyam8]

Hi, @yasharne.

I'm not sure what URL field exactly is

I believe it is $request_uri which is a part of $request ($request_method $request_uri $server_protocol).

---

match: '* /admin/v2/search'
But as I look at the log rates and compare them to the metrics coming from the netdata

Do all the requests hit /admin/v2/search or it could be /admin/v2/search+ something?

[ilyam8]

match: '* /admin/v2/search'

This internally translates into string full match (str1 == str2). If you need string contains make it * */admin/v2/search* (as u may guess has prefix/ has suffix is * /admin/v2/search* and * */admin/v2/search ).

[yasharne]

Hi @ilyam8 thanks for your reply, I have another question regarding this subject, there is an URL with this pattern:

/admin/v2/search?q={query}

which {query} is a search term, I have this configuration:

      - name: /admin/v2/search?q={query}
        match: '* /admin/v2/search\?q=.*'

Apparently, there is something wrong with my regex as it is not showing any of requests and response charts:

[ilyam8]

@yasharne could be the dot, it is not a regex but Netdata simple pattern. Try q=.* => q=*.

[yasharne]

Thanks @ilyam8 for your response, it is working now as expected

[yasharne]

Hi @ilyam8, I have another question about url_pattern, I have a pattern like this:

      - name: /admin/d/{id}/edit/full
        match: '* /admin/d/*/edit/full'

But it is not matching these lines:

/admin/d/38552/edit/full
/admin/d/23952/edit/full

In general, patterns only match if I have the * at the end of the match, if I use it in the middle of the match, it doesn't match anything

[yasharne]

Hi @ilyam8 , I have another question about url_pattern, here is a configuration for the pattern:

      - name: /admin/d/{id}/edit/full
        match: '* /admin/d/*/edit/full'

But it is not matching the below lines:

/admin/d/3960583/edit/full
/admin/d/3893952/edit/full

When I use * at the end of the match, it works as expected, but when I use it in the middle of the expression, it doesn't match anything

[yasharne]

Hi @ilyam8
Here is the configuration:

autodetection_retry: 600

jobs:
  - name: a.b.c
    path: /var/log/nginx/a.b.c-access.log
    log_type: csv
    csv_config:
      format: '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_length $request_time "$http_referer" "$http_user_agent"'
    url_patterns:
    - name: /admin/d/{id}/edit/full
      match: '* /admin/d/*/edit/full'

here is the output:

[ DEBUG ] main[main] main.go:111 plugin: name=go.d, version=v0.32.2
[ DEBUG ] main[main] main.go:113 current user: name=netdata, uid=998
[ INFO  ] main[main] agent.go:106 instance is started
[ INFO  ] main[main] setup.go:40 loading config file
[ INFO  ] main[main] setup.go:48 looking for 'go.d.conf' in [/opt/netdata/etc/netdata/ /opt/netdata/usr/lib/netdata/conf.d]
[ INFO  ] main[main] setup.go:55 found '/opt/netdata/etc/netdata/go.d.conf
[ INFO  ] main[main] setup.go:62 config successfully loaded
[ INFO  ] main[main] agent.go:110 using config: enabled 'true', default_run 'false', max_procs '0'
[ INFO  ] main[main] setup.go:67 loading modules
[ INFO  ] main[main] setup.go:86 enabled/registered modules: 1/64
[ INFO  ] main[main] setup.go:91 building discovery config
[ INFO  ] main[main] setup.go:121 looking for 'web_log.conf' in [/opt/netdata/etc/netdata/go.d /opt/netdata/usr/lib/netdata/conf.d/go.d]
[ INFO  ] main[main] setup.go:137 found '/opt/netdata/etc/netdata/go.d/web_log.conf
[ INFO  ] main[main] setup.go:142 dummy/read/watch paths: 0/1/0
[ INFO  ] discovery[manager] manager.go:90 registered discoverers: [file discovery: [file reader]]
[ INFO  ] discovery[manager] manager.go:95 instance is started
[ INFO  ] discovery[file manager] discovery.go:71 instance is started
[ INFO  ] build[manager] build.go:106 instance is started
[ INFO  ] run[manager] run.go:30 instance is started
[ INFO  ] discovery[file reader] read.go:39 instance is started
[ INFO  ] discovery[file reader] read.go:40 instance is stopped
[ DEBUG ] build[manager] build.go:153 received config group ('/opt/netdata/etc/netdata/go.d/web_log.conf'): 1 jobs (added: 1, removed: 0)
[ DEBUG ] build[manager] build.go:302 building web_log[a.b.c] job, config: map[__provider__:file reader __source__:/opt/netdata/etc/netdata/go.d/web_log.conf autodetection_retry:600 csv_config:map[format:$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_length $request_time "$http_referer" "$http_user_agent"] log_type:csv module:web_log name:a.b.c path:/var/log/nginx/a.b.c-access.log priority:70000 update_every:1 url_patterns:[map[match:* /admin/d/*/edit/full name:/admin/d/{id}/edit/full]]]
[ DEBUG ] web_log[a.b.c] init.go:34 starting URL patterns creating
[ DEBUG ] web_log[a.b.c] init.go:40 created pattern '/admin/d/{id}/edit/full', type 'matcher.globMatcher', match '* /admin/d/*/edit/full'
[ DEBUG ] web_log[a.b.c] init.go:43 created 1 URL pattern(s)
[ DEBUG ] web_log[a.b.c] init.go:49 skipping custom fields creating, no custom fields provided
[ DEBUG ] web_log[a.b.c] init.go:74 skipping custom time fields creating, no custom time fields provided
[ DEBUG ] web_log[a.b.c] init.go:103 starting log reader creating
[ DEBUG ] web_log[a.b.c] reader.go:71 open log file: /var/log/nginx/a.b.c-access.log
[ DEBUG ] web_log[a.b.c] init.go:108 created log reader, current file '/var/log/nginx/a.b.c-access.log'
[ DEBUG ] web_log[a.b.c] init.go:114 starting parser creating
[ DEBUG ] web_log[a.b.c] init.go:120 last line: '10.19.15.66 - - [18/Jun/2022:13:18:46 +0430] "GET /admin/d/112233 HTTP/1.0" 200 53793 1150 3.675 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Firefox/101.0"'
[ DEBUG ] web_log[a.b.c] parser.go:83 log_type is csv, skipping auto-detection
[ DEBUG ] web_log[a.b.c] parser.go:86 config: {FieldsPerRecord:-1 Delimiter:  TrimLeadingSpace:false Format:$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_length $request_time "$http_referer" "$http_user_agent" CheckField:0xf03e60}
[ DEBUG ] web_log[a.b.c] init.go:126 created parser: csv: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_length $request_time "$http_referer" "$http_user_agent"
[ INFO  ] web_log[a.b.c] job.go:171 check success
[ INFO  ] web_log[a.b.c] job.go:191 started, data collection interval 1s
[ DEBUG ] run[manager] run.go:41 tick 0
CHART 'netdata.execution_time_of_web_log_a.b.c' '' 'Execution time' 'ms' 'go.d' 'netdata.go_plugin_execution_time' '' '145000' '1' '' 'go.d' 'web_log'
DIMENSION 'time' '' '' '1' '1' ''

CHART 'web_log_a.b.c.requests' '' 'Total Requests' 'requests/s' 'requests' 'web_log.requests' '' '70000' '1' '' 'go.d' 'web_log'
DIMENSION 'requests' '' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.requests'
SET 'requests' = 3
END

CHART 'web_log_a.b.c.excluded_requests' '' 'Excluded Requests' 'requests/s' 'requests' 'web_log.excluded_requests' 'stacked' '70001' '1' '' 'go.d' 'web_log'
DIMENSION 'req_unmatched' 'unmatched' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.excluded_requests'
SET 'req_unmatched' = 0
END

CHART 'web_log_a.b.c.requests_by_ip_proto' '' 'Requests By IP Protocol' 'requests/s' 'ip proto' 'web_log.ip_proto_requests' 'stacked' '70021' '1' '' 'go.d' 'web_log'
DIMENSION 'req_ipv4' 'ipv4' 'incremental' '1' '1' ''
DIMENSION 'req_ipv6' 'ipv6' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.requests_by_ip_proto'
SET 'req_ipv4' = 3
SET 'req_ipv6' = 0
END

CHART 'web_log_a.b.c.current_poll_uniq_clients' '' 'Current Poll Unique Clients' 'clients' 'client' 'web_log.current_poll_uniq_clients' 'stacked' '70015' '1' '' 'go.d' 'web_log'
DIMENSION 'uniq_ipv4' 'ipv4' 'absolute' '1' '1' ''
DIMENSION 'uniq_ipv6' 'ipv6' 'absolute' '1' '1' ''

BEGIN 'web_log_a.b.c.current_poll_uniq_clients'
SET 'uniq_ipv4' = 3
SET 'uniq_ipv6' = 0
END

CHART 'web_log_a.b.c.requests_by_http_method' '' 'Requests By HTTP Method' 'requests/s' 'http method' 'web_log.http_method_requests' 'stacked' '70019' '1' '' 'go.d' 'web_log'
DIMENSION 'req_method_GET' 'GET' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.requests_by_http_method'
SET 'req_method_GET' = 3
END

CHART 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_requests_by_http_method' '' 'Requests By HTTP Method' 'requests/s' 'url ptn /admin/d/{id}/edit/full' 'web_log.url_pattern_/admin/d/{id}/edit/full_http_method_requests' 'stacked' '70029' '1' '' 'go.d' 'web_log'

BEGIN 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_requests_by_http_method'
END

CHART 'web_log_a.b.c.requests_by_url_pattern' '' 'URL Field Requests By Pattern' 'requests/s' 'url ptn' 'web_log.url_pattern_requests' 'stacked' '70027' '1' '' 'go.d' 'web_log'
DIMENSION 'req_url_ptn_/admin/d/{id}/edit/full' '/admin/d/{id}/edit/full' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.requests_by_url_pattern'
SET 'req_url_ptn_/admin/d/{id}/edit/full' = 0
END

CHART 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_responses_by_status_code' '' 'Responses By Status Code' 'responses/s' 'url ptn /admin/d/{id}/edit/full' 'web_log.url_pattern_/admin/d/{id}/edit/full_status_code_responses' 'stacked' '70028' '1' '' 'go.d' 'web_log'

BEGIN 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_responses_by_status_code'
END

CHART 'web_log_a.b.c.requests_by_http_version' '' 'Requests By HTTP Version' 'requests/s' 'http version' 'web_log.http_version_requests' 'stacked' '70020' '1' '' 'go.d' 'web_log'
DIMENSION 'req_version_1.0' '1.0' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.requests_by_http_version'
SET 'req_version_1.0' = 3
END

CHART 'web_log_a.b.c.requests_by_type' '' 'Requests By Type' 'requests/s' 'requests' 'web_log.type_requests' 'stacked' '70002' '1' '' 'go.d' 'web_log'
DIMENSION 'req_type_success' 'success' 'incremental' '1' '1' ''
DIMENSION 'req_type_bad' 'bad' 'incremental' '1' '1' ''
DIMENSION 'req_type_redirect' 'redirect' 'incremental' '1' '1' ''
DIMENSION 'req_type_error' 'error' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.requests_by_type'
SET 'req_type_success' = 3
SET 'req_type_bad' = 0
SET 'req_type_redirect' = 0
SET 'req_type_error' = 0
END

CHART 'web_log_a.b.c.responses_by_status_code_class' '' 'Responses By Status Code Class' 'responses/s' 'responses' 'web_log.status_code_class_responses' 'stacked' '70003' '1' '' 'go.d' 'web_log'
DIMENSION 'resp_2xx' '2xx' 'incremental' '1' '1' ''
DIMENSION 'resp_5xx' '5xx' 'incremental' '1' '1' ''
DIMENSION 'resp_3xx' '3xx' 'incremental' '1' '1' ''
DIMENSION 'resp_4xx' '4xx' 'incremental' '1' '1' ''
DIMENSION 'resp_1xx' '1xx' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.responses_by_status_code_class'
SET 'resp_2xx' = 3
SET 'resp_5xx' = 0
SET 'resp_3xx' = 0
SET 'resp_4xx' = 0
SET 'resp_1xx' = 0
END

CHART 'web_log_a.b.c.status_code_class_1xx_responses' '' 'Informational Responses By Status Code' 'responses/s' 'responses' 'web_log.status_code_class_1xx_responses' 'stacked' '70005' '1' '' 'go.d' 'web_log'

BEGIN 'web_log_a.b.c.status_code_class_1xx_responses'
END

CHART 'web_log_a.b.c.status_code_class_2xx_responses' '' 'Successful Responses By Status Code' 'responses/s' 'responses' 'web_log.status_code_class_2xx_responses' 'stacked' '70006' '1' '' 'go.d' 'web_log'
DIMENSION 'resp_code_200' '200' 'incremental' '1' '1' ''

BEGIN 'web_log_a.b.c.status_code_class_2xx_responses'
SET 'resp_code_200' = 3
END

CHART 'web_log_a.b.c.status_code_class_3xx_responses' '' 'Redirects Responses By Status Code' 'responses/s' 'responses' 'web_log.status_code_class_3xx_responses' 'stacked' '70007' '1' '' 'go.d' 'web_log'

BEGIN 'web_log_a.b.c.status_code_class_3xx_responses'
END

CHART 'web_log_a.b.c.status_code_class_4xx_responses' '' 'Client Errors Responses By Status Code' 'responses/s' 'responses' 'web_log.status_code_class_4xx_responses' 'stacked' '70008' '1' '' 'go.d' 'web_log'

BEGIN 'web_log_a.b.c.status_code_class_4xx_responses'
END

CHART 'web_log_a.b.c.status_code_class_5xx_responses' '' 'Server Errors Responses By Status Code' 'responses/s' 'responses' 'web_log.status_code_class_5xx_responses' 'stacked' '70009' '1' '' 'go.d' 'web_log'

BEGIN 'web_log_a.b.c.status_code_class_5xx_responses'
END

CHART 'web_log_a.b.c.bandwidth' '' 'Bandwidth' 'kilobits/s' 'bandwidth' 'web_log.bandwidth' 'area' '70010' '1' '' 'go.d' 'web_log'
DIMENSION 'bytes_received' 'received' 'incremental' '8' '1000' ''
DIMENSION 'bytes_sent' 'sent' 'incremental' '-8' '1000' ''

BEGIN 'web_log_a.b.c.bandwidth'
SET 'bytes_received' = 4406
SET 'bytes_sent' = 90481
END

CHART 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_bandwidth' '' 'Bandwidth' 'kilobits/s' 'url ptn /admin/d/{id}/edit/full' 'web_log.url_pattern_/admin/d/{id}/edit/full_bandwidth' 'area' '70030' '1' '' 'go.d' 'web_log'
DIMENSION 'url_ptn_/admin/d/{id}/edit/full_bytes_received' 'received' 'incremental' '8' '1000' ''
DIMENSION 'url_ptn_/admin/d/{id}/edit/full_bytes_sent' 'sent' 'incremental' '-8' '1000' ''

BEGIN 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_bandwidth'
SET 'url_ptn_/admin/d/{id}/edit/full_bytes_received' = 0
SET 'url_ptn_/admin/d/{id}/edit/full_bytes_sent' = 0
END

CHART 'web_log_a.b.c.request_processing_time' '' 'Request Processing Time' 'milliseconds' 'timings' 'web_log.request_processing_time' '' '70011' '1' '' 'go.d' 'web_log'
DIMENSION 'req_proc_time_min' 'min' '' '1' '1000' ''
DIMENSION 'req_proc_time_max' 'max' '' '1' '1000' ''
DIMENSION 'req_proc_time_avg' 'avg' '' '1' '1000' ''

BEGIN 'web_log_a.b.c.request_processing_time'
SET 'req_proc_time_min' = 1288000
SET 'req_proc_time_max' = 3494000
SET 'req_proc_time_avg' = 2340333
END

CHART 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_request_processing_time' '' 'Request Processing Time' 'milliseconds' 'url ptn /admin/d/{id}/edit/full' 'web_log.url_pattern_/admin/d/{id}/edit/full_request_processing_time' '' '70031' '1' '' 'go.d' 'web_log'
DIMENSION 'url_ptn_/admin/d/{id}/edit/full_req_proc_time_min' 'min' '' '1' '1000' ''
DIMENSION 'url_ptn_/admin/d/{id}/edit/full_req_proc_time_max' 'max' '' '1' '1000' ''
DIMENSION 'url_ptn_/admin/d/{id}/edit/full_req_proc_time_avg' 'avg' '' '1' '1000' ''

BEGIN 'web_log_a.b.c.url_pattern_/admin/d/{id}/edit/full_request_processing_time'
SET 'url_ptn_/admin/d/{id}/edit/full_req_proc_time_min' = 0
SET 'url_ptn_/admin/d/{id}/edit/full_req_proc_time_max' = 0
SET 'url_ptn_/admin/d/{id}/edit/full_req_proc_time_avg' = 0
END

BEGIN 'netdata.execution_time_of_web_log_a.b.c'
SET 'time' = 0
END

[ilyam8]

@yasharne the logged line contains /admin/d/112233, it doesn't match /admin/d/*/edit/full. Can you show a line that should match /admin/d/*/edit/full?

[yasharne]

Here it is @ilyam8 :

1.2.3.4 - - [19/Jun/2022:09:11:13 +0430] "GET /admin/d/275980/edit/full HTTP/1.0" 200 919138 1448 4.928 "https://a.b.c/admin/d/275980" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"

[yasharne]

@ilyam8 can we debug this somehow more?

[ilyam8]

• I created the web_log.conf using your config

  Note: changed '/admin/d/{id}/edit/full' to '/admin/d/ID/edit/full' - the UI didn't like {}

root@deb-work:/opt/logs# cat /opt/netdata/etc/netdata/go.d/web_log.conf
jobs:
  - name: nginx
    path: /opt/logs/access.log
    log_type: csv
    csv_config:
      format: '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_length $request_time "$http_referer" "$http_user_agent"'
    url_patterns:
    - name: /admin/d/ID/edit/full
      match: '* /admin/d/*/edit/full'

• access.log that contains the line you posted

root@deb-work:/opt/logs# tail -n 3 access.log
1.2.3.4 - - [19/Jun/2022:09:11:13 +0430] "GET /admin/d/275980/edit/full HTTP/1.0" 200 919138 1448 4.928 "https://a.b.c/admin/d/275980" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"
1.2.3.4 - - [19/Jun/2022:09:11:13 +0430] "GET /admin/d/275980/edit/full HTTP/1.0" 200 919138 1448 4.928 "https://a.b.c/admin/d/275980" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"
1.2.3.4 - - [19/Jun/2022:09:11:13 +0430] "GET /admin/d/275980/edit/full HTTP/1.0" 200 919138 1448 4.928 "https://a.b.c/admin/d/275980" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"

• I get 4 charts as expected (to add new lines I used cat line >> access.log)