Add option to track multiple pages in one stream #2
Conversation
erikschlegel
left a comment
erikschlegel
left a comment
There was a problem hiding this comment.
The pageIds PR looks good. Perhaps we can create a separate PR to address the enhancement to bifurcate FB posts and comments.
| override def fetchFacebookResponse(after: Date): ResponseList[Post] = { | ||
| protected def fetchFacebookResponse(after: Date): ResponseList[Post] = { | ||
| logDebug(s"Fetching posts for $pageId since $after") | ||
| facebook.getPosts(pageId, createReading(after)) |
There was a problem hiding this comment.
What happens to the comments that were posted since after? Keep in mind that FB comments are the most useful piece of data for trend detection in fortis. These posts can have comments that carry across for a very long time.
There was a problem hiding this comment.
Posts and Comments are linked in the Facebook API. As such, you can get a post for date T and then look up its linked comments for it that may have been posted at a date after T.
| ): ReceiverInputDStream[FacebookPost] = { | ||
| createPageStreams(ssc, auth, Set(pageId), fields, pollingSchedule, pollingWorkers, storageLevel) | ||
| } | ||
| } |
There was a problem hiding this comment.
we're going to need to expose a createPageCommentsStream utility to focus on newly posted comments since the previous processed time. This stream should crawl facebook4j getPostComments http://facebook4j.github.io/javadoc/facebook4j/api/PostMethods.html#getPostComments(java.lang.String, facebook4j.Reading)
There was a problem hiding this comment.
Okay. Will do in a follow-up PR, see #3 for tracking.
| auth: FacebookAuth, | ||
| pageId: String, | ||
| pageIds: Set[String], | ||
| fields: Set[String] = Set("message", "place", "caption", "from", "name", "comments"), |
There was a problem hiding this comment.
let's remove comments as this should be populated from createPageCommentsStream
Resolves #1