Firecrawl - Improvements to the scraping

[AndyMik90]

Suggested features to make the scraping more robust:

• use firecrawl pageOptions.removetags so users can enter classes/scripts to exclude from the scraping
• Stop functionality to stop the webscraping process
• Scheduling, functionality so the user can set a schedule for when the scraping should run again to update the information in the vector db.

[perzeuss]

+1 for remove tags option, it's the most important one to improve results

[crazywoola]

@VincePotato

[AndyMik90]

Btw, the request should be configured like this, with crawlerOptions and pageOptions seperated:

{
  "url": "https://example.com",
  "crawlerOptions": {
    "includes": [],
    "excludes": [],
    "generateImgAltText": false,
    "limit": 1,
    "returnOnlyUrls": false
  },
  "pageOptions": {
    "onlyMainContent": true,
    "includeHtml": false,
    "removeTags": [
        "*cmplz*",
        "*menu*"
      ]
  },
  "maxDepth": 0
}

According to FireCrawl doc: https://docs.firecrawl.dev/advanced-scraping-guide

Note that the above code has regex for tags, that can be used aswell.