Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support exclamation mark to create non-matching list in tail plugin #8613

Merged
merged 11 commits into from
Feb 16, 2021

Conversation

sspaink
Copy link
Contributor

@sspaink sspaink commented Dec 22, 2020

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

Fixes #7419 by converting [! to [^ which allows both characters to be used.

The Tail plugin depends on the standard Go function filepath.Glob which uses filepath.Match and this function does not support ! only ^, evidently this was an oversight during implementation: golang/go#41394. Hopefully in the second version of the filepath package both characters will be supported natively and we can remove this change.

Also updated the relevant unit test in globpath_test.go to use table driven tests.

Thanks!

@sspaink sspaink changed the title Tail glob Support exclamation mark to create non-matching list in tail plugin Dec 22, 2020
Copy link
Contributor

@ivorybilled ivorybilled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me.

Copy link
Contributor

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would love to use ReplaceAll() instead of the Replace() function with -1 for readability. Furthermore, \\[! is a valid pattern matching literal [! but is now replaced to \\[^ with your change. Even though I think it is unlikely to be used in the wild, the probability is not zero. Could you please check this corner-case!?

@srebhan srebhan self-assigned this Jan 4, 2021
@srebhan srebhan added area/tail feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin labels Jan 4, 2021
@sspaink
Copy link
Contributor Author

sspaink commented Jan 4, 2021

@srebhan thanks for the review! So because Go doesn't support negative lookahead in regular expressions I am having a hard time thinking of a clean way to check for this edge case, do you have any suggestions?

@srebhan
Copy link
Contributor

srebhan commented Jan 5, 2021

Well

	re := regexp.MustCompile(`(?:[^\\])(\[!)`)
	re.ReplaceAllString(s, "[^")

should work according to my test...

@sspaink
Copy link
Contributor Author

sspaink commented Jan 7, 2021

@srebhan So I couldn't get the regex to work exactly like we would want to. I did find another open source project doublestar that supports both double star and !/^ for negation, it does remove the custom logic that globpath has for handling doublestar which is nice. What do you think?

@jagularr the code has changed quite a bit from when you reviewed it after my latest change in case you wanted to look at it again as well.

internal/globpath/globpath_test.go Outdated Show resolved Hide resolved
plugins/inputs/phpfpm/phpfpm_test.go Outdated Show resolved Hide resolved
@ssoroka
Copy link
Contributor

ssoroka commented Jan 14, 2021

replace ** with **/** for backwards compatibility

plugins/inputs/tail/README.md Outdated Show resolved Hide resolved
plugins/inputs/filecount/README.md Outdated Show resolved Hide resolved
@srebhan
Copy link
Contributor

srebhan commented Jan 18, 2021

@sspaink what's wrong with the regexp I posted? Could you give me an example where it fails!? Requiring an external dependency to replace a regexp seems odd...

@sspaink
Copy link
Contributor Author

sspaink commented Jan 19, 2021

@srebhan The two issues I noticed when you run the playground example you provided: lala[!w]ee --> lal[^w]ee the result overrides the 'a' in the second 'la' and then if there isn't any characters trailing the negation class it doesn't update: [!p]laid --> [!p]laid.

@ssoroka provided a regex that solved the second issue, playground but the first problem where the previous character gets replaced still happens.

@ssoroka
Copy link
Contributor

ssoroka commented Jan 19, 2021

re using a library, glob is used in a lot of different places and I'd really love consistent behavior that–ideally–meet some standard that we are not managing ourselves. It seems too easy to accidentally create our own fork of glob behavior, which it seems is exactly what we've done and now we have to support this old broken behavior...

@srebhan
Copy link
Contributor

srebhan commented Jan 22, 2021

@sspaink fixed both with

package main

import (
	"fmt"
	"regexp"
)

func main() {
	test := []string {
		"lala[!w]ee",
		"[!p]laid",
		"foo\\[!wja",
		"\\[!wbar",
		"foo[!abc] and \\[!wbar are wunderbar",
	}

	re := regexp.MustCompile(`([^\\]?)(\[!)`)
	for _, s := range test {
		fmt.Printf("%s --> %s\n", s, re.ReplaceAllString(s, "$1[^"))
	}
}

But if I understand @ssoroka correctly he prefers to use the doublestar-package. Also fine with me. :-)

@sspaink
Copy link
Contributor Author

sspaink commented Feb 9, 2021

@srebhan thank you for the updated regular expression, I've updated the pull request to use the suggested change and it looks like it works without having to change any of the existing tests.
@ssoroka I did like the idea of moving to the library but I couldn't figure out how to cleanly replace **->**/** without causing the issue you describe **/**->**/**/**/**. Maybe we can move this to a library safely for a 2.0 release.

Copy link
Contributor

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Feb 10, 2021
@srebhan srebhan requested a review from ssoroka February 10, 2021 09:45
replace string twice to handle edge case
@sspaink sspaink merged commit 71be90d into influxdata:master Feb 16, 2021
arstercz pushed a commit to arstercz/telegraf that referenced this pull request Mar 5, 2023
…nfluxdata#8613)

* Replace exclamation mark with caret

* Update README and use table driven tests

* Use ReplaceAll instead

* Use doublestar package instead to glob filepath

* Add license

* Fix order of dependencies

* Doc improvement, maybe better then str replace?

* Forgot to remove nil from test

* Use regex instead of library

* Revert unnecessary change

* Go back to using library
replace string twice to handle edge case
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tail feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Glob Pattern at inputs.tail not working properly
4 participants