adding a dateLastProcessed field?

[owenrh]

Hi,

We're looking to add some additional diagnostic capability to the status index by adding a dateLastProcessed field.

We could push it in as a metadata field in our own crawler code, but we were wondering if it would make sense as a top-level field that most storm-crawler users would benefit from?

[jnioche]

Thanks @owenrh (and great to see that you joined @mattburns and his gang)

Would that value be different from the fetch date? If so it could be added by the AbstractStatusUpdaterBolt, if not then it would be created by the Fetcher bolt implementations and be called e.g. lastFetchDate to mirror nextFetchDate. Having this could be useful for implementing an adaptive scheduling and it would be good for diagnostics indeed.

[owenrh]

Thanks for getting back to me so quickly Julien.

Re: fetch date - it would be the timestamp of the last attempted fetch, so either lastFetchDate or lastProcessedDate would make sense. I think I suggested the latter just because I felt it didn't imply a successful fetch, which I thought lastFetchDate might. (But maybe that's just a matter of documentation).

If you think it would be useful in the core then I'm happy to create a PR for it, and mod the AbstractStatusUpdaterBolt.