Ip matcher for LogQL #3986
Ip matcher for LogQL #3986
Conversation
pkg/logql/log/ip.go
Outdated
| func (ipf *IPFilter) Process(line []byte, lbs *LabelsBuilder) ([]byte, bool) { | ||
|
|
||
| // make sure the pattern provided was valid, even before trying to match. | ||
| if ipf.patternError != nil { |
There was a problem hiding this comment.
This can be spotted before and so should return an error during query parsing.
There was a problem hiding this comment.
Yea. thought about that actually. My previous implementation had thrown this error during constructor of NewIPFilter() but while writing grammar, there was no way to handle that error while creating the LabelFilter expression!. So I changed it to like this.
May be I'm missing something about handling this error during the parse stage?. Can you point me any example?
There was a problem hiding this comment.
You can use the Stage() method of this expr, it can return an errors and will be considered as 400.
|
@cyriltovena addressed your comments. can you take a look again? thanks |
docs/sources/logql/ip.md
Outdated
| Consider the following logs, | ||
|
|
||
| ``` | ||
| 3.180.71.3 - - [17/May/2015:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)" | ||
| 80.91.33.133 - - [17/May/2015:08:05:14 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.16)" | ||
| 46.4.66.76 - - [17/May/2015:08:05:45 +0000] "GET /downloads/product_1 HTTP/1.1" 404 318 "-" "Debian APT-HTTP/1.3 (1.0.1ubuntu2)" | ||
| 93.180.71.3 - - [17/May/2015:08:05:26 +0000] "GET /downloads/product_1 HTTP/1.1" 404 324 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)" | ||
| ``` | ||
|
|
||
| How would you use LogQL to search for log lines with IP addresses?. Say single IP to start with? That's easy, we can use LogQL line filter (kinda like distributed grep) | ||
|
|
||
| ```logql | ||
| {foo="bar"} |= "3.180.71.3" | ||
| ``` | ||
|
|
||
| will output following log lines. | ||
|
|
||
| ``` | ||
| 3.180.71.3 - - [17/May/2015:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)" | ||
| 93.180.71.3 - - [17/May/2015:08:05:26 +0000] "GET /downloads/product_1 HTTP/1.1" 404 324 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)" | ||
| ``` | ||
|
|
||
| but wait! what `93.180.71.3` is doing here?. Oh yea. it actually matches with what we queried for `3.180.71.3`. It can also match with other log lines which we don't actually want. The right option would be to use regexp to match the IP address(something like `|~"^3.180.71.3"`). | ||
|
|
||
| Now forget about single IP, what about range of IP addresses? What about IP subnet? Or even more interesting IPv6 addresses? (not easy to come up with regexp for all the use cases). | ||
|
|
||
| Luckily, LogQL comes with built-in support for IP matcher. |
There was a problem hiding this comment.
I don't think we need to present the need for the parser; I think that's relatively self-explanatory.
docs/sources/logql/ip.md
Outdated
| - Label Filter | ||
|
|
||
| ```logql | ||
| { foo = "bar" }|logfmt|remote_addr=ip("2001:db8::1-2001:db8::8")|level="error" |
There was a problem hiding this comment.
Let's format these queries to make them more readable:
{ foo = "bar" }
| logfmt
| remote_addr=ip("2001:db8::1-2001:db8::8")
| level="error"
|
@dannykopping @KMiller-Grafana Are we good to go ? |
|
Looks good to me. I've added an extra task to my personal to-do list to correct the weights in the docs files associated with this PR after this PR is merged in. |
|
@kavirajk can up resolve conflicts and regenerate the y.go files. |
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
1. ast tests 2. parse tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
1. support NEQ 2. reduce allocation. 3. Minor tweaks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
1. Add necessary interface implementation 2. More tests for parser and ast Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Its wierd bug. Not handing string() method breaks the parser :( Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
It got out of hands to handle both in same struct. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
1. Indendation fix 2. Remove string() interface for ip label filte
|
@cyriltovena conflicts resolved. Can you merge it? |
What this PR does / why we need it:
This PR adds buiilt-in IP matcher support for LogQL. It adds filter
ip(pattern). It works with both IPv4 and IPv6 and can be used as bothlabel_filterandline_filterpatterncan be one of the following.192.168.0.1,::1192.168.0.1-192.189.10.12,2001:db8::1-2001:db8::8192.168.4.5/16,2001:db8::/32Example queries:
{ foo = "bar" }|logfmt|addr=ip("192.168.4.5/16"){ foo = "bar" }|logfmt|remote_addr=ip("2001:db8::1-2001:db8::8")|level="error"Which issue(s) this PR fixes:
Fixes #2722
Special notes for your reviewer:
Based on the design doc
Checklist