-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
grok should ignore tilde backup files when processing patterns_dir #2271
Comments
I like the idea, but I"m not sold on the proposed solution. What editor uses '~' suffix for backup files? Emacs? Vim uses Tested on my OSX 10.10 laptop:
I haven't tested any other editors, but this behavior seems hardly uniform :( I'd be open to maybe having the default be more explicit in what filenames are accepted. Perhaps only files matching /[0-9A-Za-z_-]+/ would be ok? This would ignore any dot files and any weird local state files created by the editors I've tested so far. If you use a different editor or an editor tested above with a different behavior, please update this ticket with what backup or local state files are created by your editor so we can try to generalize a pattern here. |
@jordansissel - I was using gedit on Oracle Linux at the time, but a quick Google to see how widespread this practice was suggests that emacs, vi and nano all follow the tilde convention. Edit: not sure about a whitelist; there's a danger it'll just end up frustrating people the other way when their pattern files get unexpectedly ignored. I can't imagine anyone choosing to tilde-suffix their pattern files, but I can definitely imagine people adding file extensions, which would fail your proposed test. Dotfiles proper sound like another good candidate for blacklisting, though. |
@mrec See my previous comment. Neither the "default" (as provided by either OSX or homebrew, I don't know) emacs nor the "default" (same) vim on OSX use tilde-suffix. Additionally doing some additional exploration:
Of vim, emacs, and nano, I can't see any examples of this tilde convention. |
@jordansissel I'm not disputing what you're seeing, but unless a lot of people out there are lying then that's not the whole story. Timing here is unfortunate since I'm not going to have access to any *NIX machines for the next two weeks now, but Google shows many many mentions of tildes, including from OSX and Ubuntu users, although they do fall off after 2011 so maybe some distros changed their default configs around then. Original problem was seen in Oracle Linux 6.4, which was released in 2013. |
@jordansissel Emacs's Emacs has had Ignoring |
For Logstash 1.5.0, we've moved all plugins to individual repositories, so I have moved this issue to logstash-plugins/logstash-filter-grok#33. Let's continue the discussion there! :) |
(This comes from the discussion of #2244)
When testing a config using grok and custom patterns, a user will often be editing pattern definition files in
patterns_dir
between run attempts. Many (most?) Linux-ey text editors create backup files, named as the original filename plus a~
suffix, in the same location as the original; even though they aren't hidden these are often invisible by default in file browsers. When dealing with multiple pattern definition files, and especially when renaming them, it's possible to have a lot of these tilde files lying around after a while.grok
currently reads everything inpatterns_dir
, including any tilde backups. It quite reasonably doesn't define the order in which it reads them, and it doesn't warn if e.g. the definition ofMYPATTERN
in a stalepatterns~
orpreviousfilename~
backup file overrides the definition ofMYPATTERN
inpatterns
. Hilarity ensues. Also hair-tearing, teeth-gnashing, bad language and various other undesirable outcomes.I propose that
grok
should ignore any files inpatterns_dir
ending in a~
. There may be other things it'd be beneficial to blacklist too, but this seems like a good start.The text was updated successfully, but these errors were encountered: