Switching to the desktop layout (done)

[iMrDJAi]

The mobile layout of Facebook provides limited data and low quality media, because of that the project should switch entirely to the desktop layout.

Todo list:

• Desktop layout selectors:
  • I'm currently trying to support posts that contain text, images and videos, as a result posts with embeds, 360 images, backgrounds and shared posts may not be fully/correctly scraped, and this is due to the lack of selectors which will be added later, or the workarounds that will be used to scrape them (Especially 360 images, I have no idea how to deal with them).
    These are all the selectors explained:
    • group_name: This is the title element of the document, it contains the group name.
    • group_feed_container: This element contains the submissions.
    • post_element: This is the submission element, it's not necessary a direct child of the group feed container, but it's guaranteed to contain all the data that we're targeting.
    • post_author: This is an "a" element, we can extract the author name and profile url from it.
    • post_author_avatar: A small profile picture of the author.
    • post_link: An "a" element contains the post permalink and the time when it was created (but since the innerText is obfuscated we can't simply extract it).
    • post_content: This contains the post text. We can convert the innerHTML of this element to markdown with the turndown library.
    • post_content_expand_button: This is the "See More" button, it must be clicked before we start scraping the post content.
    • post_attachment: This one contains attachments, it's the next element after the post content wrapper.
    • post_video: A selector for all the videos inside the post attachments wrapper, these can be scraped with the following workaround.
    • post_img: A selector for all the images inside the post attachments wrapper, these don't necessary have the best quality since you should click on them to show their best resolutions, but they are far better than the images on the mobile layout.
• Update the scraper code.

[kaanyagci]

HI,

Thank you for your contribution. I changed the base branch from master to development as there are still things uncompleted. I'll try to take a closer look probably this weekend.

The markdown on post contents using another npm module could be an overkill also may be a maintainability problem. But we can implement something which removes all HTML tags around the String content not necessarily converting to a markdown input.

[iMrDJAi]

HI,

Thank you for your contribution. I changed the base branch from master to development as there are still things uncompleted. I'll try to take a closer look probably this weekend.

The markdown on post contents using another npm module could be an overkill also may be a maintainability problem. But we can implement something which removes all HTML tags around the String content not necessarily converting to a markdown input.

@kaanyagci The point of the markdown format is actually providing users a minimal output can be used to re-visualize posts content in the exact same way as the original, I only suggested that to avoid including the HTML format as it may be large and not human readable.
But I think you're right, we should reduce the number of the third party modules, also it's a good idea to let users handle the output by themselves, and we can provide them examples on how to do that.
In this case, we have to include the innerHTML along with the innerText in the output.

[iMrDJAi]

After some testing, I've noticed that when you start scraping without authentication, some posts won't provide the author profile url, in this case the selector group_post_author won't work.

Also, it's quite different how elements are being loaded in the desktop layout, in fact they won't until they show up on the viewport, and for that we should start scrolling before scraping.

[iMrDJAi]

@kaanyagci So yeah, I did it! The scraper works perfectly now with the new desktop layout of Facebook, and it has the same functionality as the one from the master branch. I think it's time you merge this to the development branch (after reviewing and testing it of course). Other features and new fields for the GroupPost interface should be added in a separate pull request to make it easier to organize things up!

[kaanyagci]

@all-contributors please add @iMrDJAi for code

[allcontributors]

@kaanyagci

I've put up a pull request to add @iMrDJAi! 🎉

[kaanyagci]

@iMrDJAi This is excellent news! I was really busy with other stuff today. I'll test this first thing tomorrow! Great job! 💯

[iMrDJAi]

@kaanyagci Any updates? Have you tested it? Any issues?

[kaanyagci]

Sorry for the delay. I was still a little busy :( I'll look ASAP.

[kaanyagci]

Just checked. Sadly I can not get it to work.

• The first error I've faced with log in with the following reproduction route:

import { FB } from './index';

async function main() {
  const f = await FB.init({
    debug: true,
    output: 'test.json',
    headless: false,
    groupIds: ['774278349295443'],
    useCookies: true,
    disableAssets: true,
  });
  f.login('<redacted>', '<redacted>');
  await f.getGroupPosts(774278349295443, 'groupOutput');
}

main().then(() => {
  console.log('Done');
});

Gives the following output:

/Users/kaanyagci/Documents/makepad/fbjs/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:115
                    ? new Error(`${response.errorText} at ${url}`)
                      ^

Error: net::ERR_ABORTED at https://facebook.com
    at navigate (/Users/kaanyagci/Documents/makepad/fbjs/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:115:23)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async FrameManager.navigateFrame (/Users/kaanyagci/Documents/makepad/fbjs/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:90:21)
    at async Frame.goto (/Users/kaanyagci/Documents/makepad/fbjs/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:416:16)
    at async Page.goto (/Users/kaanyagci/Documents/makepad/fbjs/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:819:16)
    at async Facebook.login (/Users/kaanyagci/Documents/makepad/fbjs/dist/lib/models/fb.js:113:9)

Note: The output is the same for both headless and not headless modes.

• The second error I've faced without login the web page used for group details is still the mobile page m.facebook.com

I'll try to investigate these issues as soon as possible this week

[iMrDJAi]

@kaanyagci Interesting. in fact I haven't tried logging in, I been always testing in userless mode, I'll try that later and check what's going on.
For now you can try this:

;(async () => {

    const { FB } = require("@makepad/fbjs")

    const fb = await FB.init({
        headless: true,
        useCookies: false,
        output: ''
    })

    //await fb.getGroupPosts("319144912641926", "./output.json")
    
    await fb.getGroupPosts("319144912641926")

})()

[iMrDJAi]

The second error I've faced without login the web page used for group details is still the mobile page m.facebook.com

@kaanyagci That doesn't make sense, I'm 100% sure that I totally removed the mobile website. Fork my master branch again.

[kaanyagci]

My bad, I was trying on another branch 🤦

[kaanyagci]

This looks great actually. For the first issue, I've added the userAgent as Facebook rejects connections from headless browsers. I'll add this line once it's merged on development branch! Anyway great work @iMrDJAi !