please provide a scrape endpoint which can take care of MFA, authentication, sharepoint, captcha solving, support for different extensions like pdf, pptx, docx.

[rostwal95]

Thank you for this amazing project, can we add below functionalities as well. -

For a given url we should be able to scrape the website, This can include support for SSO, multi factor authentication, sharepoint handling and ability to handle captcha.
It should also be able to scrape and extract information from different extensions like pdf, docx, ppt and other well known extensions.

[unclecode]

Hi @rostwal95 The second part of your request is on the Todo list. As for the first part, there is an internal debate among us about whether we should proceed with it or not.

[rostwal95]

As long as we open up a session with the user after the detection of SSO or MFA where he is able to authenticate and upon successful authentication we get the cookies and scrape the endpoint as per the depth provided by the user after reading the robots.txt and once the scraping is done clear all the cookies. we should be good right ?

[unclecode]

@rostwal95 To address the authentication and similar tasks, I have created a new branch called hooks which includes hooks (callback) functionality. This allows you to handle custom authentication workflows externally and pass the authenticated drivers to the crawler.

You can check out the hooks branch and look at the example in the quickstart.py file, particularly the last function which demonstrates one possibility for using these hooks.

Right now there are 4 hooks:

• on_driver_created
• before_get_url
• after_get_url
• before_return_html

Please review the hooks branch and I plan to merge this branch into the main branch soon.

[unclecode]

These new hooks (events) are added and now available. Also check quickstart.py for examples. I close this issue, and thx for suggestion.

[rostwal95]

Thank you for such a quick response, I will check it out. Just one thing around scraping for a provided url wouldn't it be a good idea to get the depth from the user (max depth defined in the backend) and we should start scraping using that, open up a storage interface which anyone can implement based on their needs to store these documents before processing it. it will start solving lot of use cases and will provide customization as well.