get_connections("me", "home") with paging returns unpredictable results

[alecnmk]

Hi,

I'm receiving unpredictable results while using "home" connection for user which granted necessary permissions to my app. Here is a link to a rake task I've used to demonstrate it: https://gist.github.com/1725794

I was using Koala v1.2.1 and FbGraph v2.3.1 for comparison.

With FbGraph I receive stable(can be reproduced multiple times in a row) results (still appearing strange to me, but more promising than Koala provides) Output from rake task:

starting fetching user's home with FbGraph v2.3.1
Initial feed size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 24
Next page size: 21
Next page size: 18
Next page size: 18
the end

But when I use Koala to do the same thing, I receive:

starting fetching user's home with Koala v1.2.1
Initial feed size: 25
Next page size: 25
Next page size: 2
Next page size: 0
the end

And results above cannot be reproduced, they are differ each time, but my main point is that I'm not able to get all posts. If that is an issue with FB API than why results provided by different libraries are so different?

[alecnmk]

same is reproducible with Koala v1.3.0rc2

[alecnmk]

I believe that post brings some light on the issue https://developers.facebook.com/blog/post/478/

[arsduo]

Thanks Alexander! I'd seen this issue before, but never thought to compare it with another library to see if it was Koala's implementation or a Facebook issue. This is great data.

I'm reading through the blog post and will figure out what the problem is and get a fix out ASAP.

[alecnmk]

I've figure out the problem for myself, and there is no good solution to implement in Koala, but I think it's worth to include the link to that post to one of your wiki pages regarding paged requests.

As for myself, I just setting higher limit for paging and receiving more than enough objects to process. But it's still interesting why results with Koala differ from FbGraph. I suppose it's more about tweaking HttpService.

Looking forward for your expertise on that matter.

[arsduo]

Yeah, it is really interesting -- Koala's implementation literally just parses the next and previous URLs provided by Facebook and uses them to fetch more data. Once I see how FbGraph implements it, perhaps the difference will be clearer. In the meanwhile, I'm glad you were able to work around it by simply requesting more information per page.

I'll let you know what I find.

[arsduo]

Unfortunately, I can't reproduce this on my own stream or on a random public stream (ContextOptional). Are you by any chance able to reproduce it on a publicly visible feed?

The only real difference I can see between Koala's implementation and fb_graph's is that the latter URI.unescapes the limit, since, etc. parameters. I've never seen any that require unescaping, but it's worth a try.

[arsduo]

Actually, could you just have your rake task print out the paging URLs? (feed.paging.inspect) -- that might be enough.

Thanks!

Alex

[alecnmk]

Hope that would help. Pls, let me know if you need anything else.

starting fetching user's home with Koala v1.2.1
Initial feed size: 25
Feed.paging.inspect: {"previous"=>"https://graph.facebook.com/me/home?access_token=XXX&limit=25&since=1328438383&__previous=1", "next"=>"https://graph.facebook.com/me/home?access_token=XXX&limit=25&until=1328382855"}
Next page params: ["me/home", {"access_token"=>"XXX", "limit"=>"25", "until"=>"1328382855"}] (until 2012-02-04T19:14:15+00:00)
Feed.paging.inspect: {"previous"=>"https://graph.facebook.com/me/home?access_token=XXX&limit=25&since=1328382150&__previous=1", "next"=>"https://graph.facebook.com/me/home?access_token=XXX&limit=25&until=1328351710"}
Next page params: ["me/home", {"access_token"=>"XXX", "limit"=>"25", "until"=>"1328351710"}] (until 2012-02-04T10:35:10+00:00)
Next page size: 22
Feed.paging.inspect: {"previous"=>"https://graph.facebook.com/me/home?access_token=XXX&limit=25&since=1328350979&__previous=1", "next"=>"https://graph.facebook.com/me/home?access_token=XXX&limit=25&until=1328350867"}
Next page params: ["me/home", {"access_token"=>"XXX", "limit"=>"25", "until"=>"1328350867"}] (until 2012-02-04T10:21:07+00:00)
Next page size: 3
Next page size: 0
the end, 50 total

And consecutive execution with fb_graph:

starting fetching user's home with FbGraph v2.3.1
Initial feed size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 25
Next page size: 21
Next page size: 20
Next page size: 22
the end

[arsduo]

I was getting this problem earlier with test users, but now it seems to be working properly. I'm going to release Koala 1.4rc1 tonight, and will ask if anyone else has seen this and can reproduce it on a publicly-visible object.

[bnorton]

The last thing to note is that when facebook grabs the limit number of object from the server, it then filters out any that don't match some criteria such as the status of the post, etc. This is expected behavior.