Unable to collect posts beyond a certain number

[chenxingyuzealken]

Hi wondering how can i solve this issue where I get only 2000+ posts from doing this. I know there are more posts, but I'm not getting them.

opt = {}
opt['daterange'] = False #set to True if you want to limit your search by startDate and endDate
opt['startDate'] = datetime.strptime('01/05/19 00:00:00', '%m/%d/%y %H:%M:%S') #change the time ranges to what you want
opt['endDate'] = datetime.strptime('12/06/22 00:00:00', '%m/%d/%y %H:%M:%S') #change the time ranges to what you want

page_name = 'ChannelNewsAsia'
fbcookies = { cookie details}
lst = []

for post in get_posts(page_name, cookies=fbcookies, pages=1000000,options={"allow_extra_requests": False}):

if opt['daterange'] == True:    
    #opt['demo'] = False
    
    if post['time'] < opt['startDate']:
        print('post collection is earlier than',opt['startDate'],'stopping collection' )
        if opt['demo'] == False:
            break
    
    if post['time'] > opt['endDate']:
        print('post collection is after',opt['endDate'],'stopping collection' )
        if opt['demo'] == False:
            break
    lst.append(post)

It stopped collecting stuff at 2020, which is odd for me

[neon-ninja]

Try increasing the posts_per_page option

[neon-ninja]

I added some code to retry pagination requests on error (1cc8064), and with that, and this test code:

start = time.time()
posts = []
try:
    for post in get_posts("ChannelNewsAsia", cookies="cookies.txt", pages=200, timeout=60, options={"allow_extra_requests": False, "posts_per_page": 200}):
        posts.append(post)
except:
    print(f"{len(posts)} posts retrieved in {round(time.time() - start)}s. Oldest post: {posts[-1].get('time')}")

I get

14201 posts retrieved in 910s. Oldest post: 2013-12-12 13:05:00

[neon-ninja]

717d522 might also be useful to resume from the last cursor that errored out, see #287 (comment) for usage

[chenxingyuzealken]

Thanks! I think the combination of:

from facebook_scraper import *
import pandas as pd
import ast
import time
from datetime import datetime

import requests
import logging
enable_logging(logging.DEBUG)

start = time.time()
posts = []
try:
for post in get_posts("ChannelNewsAsia", cookies="cookies.txt", pages=200, timeout=60, options={"allow_extra_requests": False, "posts_per_page": 200}):
posts.append(post)
except:
print(f"{len(posts)} posts retrieved in {round(time.time() - start)}s. Oldest post: {posts[-1].get('time')}")

and this:

cursor = " some url from the loggin output."

posts = []
try:
for post in get_posts("ChannelNewsAsia", cookies="cookies.txt", pages=200, timeout=60, options={"allow_extra_requests": False, start_url=cursor, "posts_per_page": 200}):
posts.append(post)

have helped in making the process more robust.

Thanks for the help! Your project is amazing!

[neon-ninja]

#291 might be useful