WebCrawler: improvements

[eolivelli]

Summary:
Add a couple of improvements to webcrawler-source:

• Take into account the robots.txt file
• Download the site maps
• Add a configuration parameter to limit the overall number of urls considered by the crawler
• Add a configuration parameter to limit the depth of the crawling
• Add a configuration parameter to not scan for Links inside the documents
• Removed the "idle-time" parameter

The new parameters are:

• max-urls: the maximum number of urls visited by the Web Crawler (defaults to 1000)
• max-depth: the maximum depth (defaults to 10)
• handle-robots-file: look for robots.txt (and then sitemaps) (defaults to true)
• scan-html-documents: scan the HTML documents in order to find links to other pages (defaults to true)

Full configuration:

  - name: "Crawl the WebSite"
    type: "webcrawler-source"
    configuration:
      seed-urls: ["https://docs.langstream.ai/"]
      allowed-domains: ["https://docs.langstream.ai"]
      forbidden-paths: []
      min-time-between-requests: 500
      reindex-interval-seconds: 3600
      max-error-count: 5
      max-urls: 1000
      max-depth: 10
      handle-robots-file: true
      user-agent: "" # this is computed automatically, but you can override it
      scan-html-documents: true
      http-timeout: 10000
      handle-cookies: true
      max-unflushed-pages: 100
      bucketName: "{{{secrets.s3.bucket-name}}}"
      endpoint: "{{{secrets.s3.endpoint}}}"
      access-key: "{{{secrets.s3.access-key}}}"
      secret-key: "{{{secrets.s3.secret}}}"
      region: "{{{secrets.s3.region}}}"