FeedEntryMessageSource Improperly Calculates Proper Last Entry Processed Date [INT-3700]

[spring-operator]

Aaron Loes opened INT-3700 and commented

This is different from issue #5797

I'm using SpringXD and i need to process information from RSS feeds. Since SpringXD doesn't come with the feed source, i've compiled the example from the documentation (http://docs.spring.io/spring-xd/docs/1.1.1.RELEASE/reference/html/#creating-a-source-module). when testing, i noticed that after some time, it stops honoring my polling interval and keeps polling in the last feed entry continually.

After some debugging, i noticed that when the FeedEntryMessageSource reaches the end of the entry set and if it has not meet the maximum number of messages to poll, that it will attempt to fetch the feed again (presuming that it may have updated since) and load any NEW feeds in to the collection for processing. In my case, it always loads the last feed as the feed publish/update date is greater than the feeds publish date or the last processed entries publish date.

I feel that the setting of the lastTime variable in FeedEntryMessageSource#getNextEntry should use the same logic that is used for comparison. Namely the logic in the method FeedEntryMessageSource#getLastModifiedDate.

I feel that this is a critical issue as the message source will never return control back to and continually poll the same feed until the feed entries have been added to reach the maximum number of messages to poll count.

One work around would be to tailor the polling interval and the maximum number of messages to poll count to the update frequency of the feed being read. This is not suitable for me as I do not have control of which feeds are added to my system. The other work around would be to copy the existing class, fix the issues, and wire it in as a source to an inbound channel adapter.

I have done the second to verify this works; at least on the feed I've been using.

---

Affects: 4.1.2

Referenced from: pull request #1426

Backported to: 4.0.8, 3.0.8

[spring-operator]

Aaron Loes commented

Title probably should be renamed to "FeedEntryMessageSource improperly calculates proper last entry processed date"

[spring-operator]

Artem Bilan commented

Aaron Loes, thank you for the feedback and the good investigation!

Since you already have a fix, would you mind to Pull Request it as a Contribution ?

[spring-operator]

Aaron Loes commented

Added pull request to replace one marked. This one hopefully follows the contribution guide and also adds a unit test.

#1426