The directory should not be repeatly defined in both WatchServiceDirectoryScanner and <int-file:inbound-channel-adapter /> [INT-3989]

[spring-operator]

Simon Wong opened INT-3989 and commented

The WatchServiceDirectoryScanner requires the directory be defined as constructor. If this scanner be injected in <int-file:inbound-channel-adapter />, the directory attribute has to be defined again in <int-file:inbound-channel-adapter /> once more. I think the checking rule should be relaxed such that developer only has to defined once.

---

Affects: 4.2.5

Issue Links:

• WatchService and Java 1.6 [INT-4185] #8122 WatchService and Java 1.6

• Add a Message-Driven File Inbound Channel Adapter [INT-4000] #7943 Add a Message-Driven File Inbound Channel Adapter

Referenced from: pull request #1774

[spring-operator]

Gary Russell commented

Perhaps we can add a marker interface to detect whether or not the scanner already has a directory and relax the check in the namespace.

[spring-operator]

Artem Bilan commented

H-m-m. Looking to the inconsistent implementation of protected File[] listEligibleFiles(File directory) { I'd say that logic around WatchService should be encapsulated directly in the FileReadingMessageSource to avoid such a directory confuse.
Since WatchServiceDirectoryScanner has only one top-level option like directory and all other can be delegated from the FileReadingMessageSource that should not be so difficult to move the WatchServiceDirectoryScanner into the inner class and hide that inconsistent logic.
With that the #7931 will go by itself.

[spring-operator]

Simon Wong commented

If the WatchServiceDirectoryScanner is moved to the inner class and not to expose to outside, will it lose the flexibility for extension? e.g. currently the WatchServiceDirectoryScanner doesn't support the monitoring file changed (only file creation is supported), we might have to extends the WatchServiceDirectoryScanner class to support this feature.

[spring-operator]

Artem Bilan commented

Well, I think we just should support the StandardWatchEventKinds.ENTRY_MODIFY as well. The target FileListFilter should do the proper filtering logic on the matter.

I even think that with the StandardWatchEventKinds.ENTRY_DELETE we can get a gain from the inner class when ResettableFileListFilter is presented for the FileReadingMessageSource.

If there is no any objection, I'll go ahead and implement this feature and an inconsistency will be deprecated.
All other JIRAs around WatchServiceDirectoryScanner will be closed automatically because they won't make sense any more.

[spring-operator]

Gary Russell commented

Thanks, Artem Bilan I think that's the best way forward.

[spring-operator]

Simon Wong commented

Hi Artem, with the new modification. could I know how to distinguish the file being listed is newly created or modified?

[spring-operator]

Artem Bilan commented

Not sure why you need to know that distinguishes, because the downstream flow deals just only with file.

In the test-case of the PR (#1774) for this JIRA I use FileSystemPersistentAcceptOnceFileListFilter which has ability to check the file.lastModified() against existing entry and, therefore, accepts it and emits for the downstream flow.

Now, please, share your experience how you do that in your application.
We may reconsider some logic in the fix and may introduce some callback to react to those WatchService event outside of the FileReadingMessageSource.
Just need to know what we are missing with the current solution.

Thanks

[spring-operator]

Simon Wong commented

Not my current use case, but other scenarios that I could think of.

e.g. - I have a file inbound adapter that monitor one directory.

• Some new files will be created in that directory.
• Some existing files might be updated.
• Different actions will be taken by downstream components based on whether the file is created or updated.
  That's the reason why it would be like to distinguish them.

One solution I could think of is using 2 different FileReadingMessageSources and points to the same directory. One is monitor file creation. The other one is monitor file modification.

And a new method setWatchOptions could be added in the FileReadingMessageSourceFactoryBean to control which StandardWatchEventKinds events (ENTRY_CREATE or ENTRY_MODIFIED) are interested.
i.e.

FileReadingMessageSourceFactoryBean factoryBean = new FileReadingMessageSourceFactoryBean();
factoryBean.setUseWatchService(true);
factoryBean.setOptions(Arrays.asList(EVENT_CREATE));   // e.g. I only concern the file creation event but not file modification event

[spring-operator]

Artem Bilan commented

OK, Simon Wong,

That all makes sense.
For the ENTRY_MODIFY case we can use AcceptAllFileListFilter to pass all the modified files to the downstream.

Will add the setWatchEvents to the PR soon.

Feel free to comment there.

Thank you!

[spring-operator]

Simon Wong commented

Is there any plan to support event driven based file notification for WatchService? Currently I have to set a poller explicitly. I think we could take advantage of using WatchService.take() to block when no watch key in queue. The downstream subscriber could be notified automatically when some watch keys be added in the queue.

[spring-operator]

Artem Bilan commented

Simon Wong,

That sounds great. Thank you for pointing that out!

Please, raise a new JIRA on the matter.
In addition consider the Contribution.
That new component should be an extension of MessageProducerSupport since it will be already event-driven.

I need to think more about the future in case of there are some cornerstones, which were as a reason do not implement the WatchService logic that way from the beginning...

But you feel free to start your solution and we will appreciate it in the Pull Request.

[spring-operator]

Gary Russell commented

I opened #7943 - I think a message-driven file adapter would be a good addition to the framework.