-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NIFI-2859 - Ignore files starting with a dot in ListHDFS #1383
Conversation
@@ -176,7 +176,7 @@ private HDFSListing deserialize(final String serializedState) throws JsonParseEx | |||
|
|||
// Build a sorted map to determine the latest possible entries | |||
for (final FileStatus status : statuses) { | |||
if (status.getPath().getName().endsWith("_COPYING_")) { | |||
if (status.getPath().getName().endsWith("_COPYING_") || status.getPath().getName().startsWith(".")) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should try to be consistent with how ListFile works. It has a property to apply a regex to filer filenames and the default value is anything that doesn't start with a dot:
public static final PropertyDescriptor FILE_FILTER = new PropertyDescriptor.Builder()
.name("File Filter")
.description("Only files whose names match the given regular expression will be picked up")
.required(true)
.defaultValue("[^\\.].*")
.addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
.build();
This way the user can determine if they want dot files or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bbende Yes you're right! Otherwise there is the following in GetHDFS:
public static final PropertyDescriptor IGNORE_DOTTED_FILES = new PropertyDescriptor.Builder()
.name("Ignore Dotted Files")
.description("If true, files whose names begin with a dot (\".\") will be ignored")
.required(true)
.allowableValues("true", "false")
.defaultValue("true")
.build();
But the filter property is much better. I'll update the PR.
@bbende: I added the file filter property as suggested. |
@pvillard31 @bbende LGTM |
+1 merged to master |
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.