Skip to content

Latest commit

 

History

History
75 lines (53 loc) · 2.49 KB

README.md

File metadata and controls

75 lines (53 loc) · 2.49 KB
Raw
title markmap
ScrapGraphAI Roadmap
colorFreezeLevel maxWidth
2
500

ScrapGraphAI Roadmap

Short-Term Goals

  • Integration with more llm APIs

  • Test proxy rotation implementation

  • Add more search engines inside the SearchInternetNode

  • Improve the documentation (ReadTheDocs)

  • Create tutorials for the library

Medium-Term Goals

  • Node for handling API requests

  • Improve SearchGraph to look into the first 5 results of the search engine

  • Make scraping more deterministic

    • Create DOM tree of the website
    • HTML tag text embeddings with tags metadata
    • Study tree forks from root node
    • How do we use the tags parameters?

  • Create scraping folder with report

    • Folder contains .scrape files, DOM tree files, report
    • Report could be a HTML page with scraping speed, costs, LLM info, scraped content and DOM tree visualization
    • We can use pyecharts with R-markdown

  • Scrape multiple pages of the same website

    • Create new node that instantiate multiple graphs at the same time
    • Make graphs run in parallel
    • Scrape only relevant URLs from user prompt
    • Use the multi dimensional DOM tree of the website for retrieval
    • Issue #112

  • Crawler graph

    • Scrape all the URLs with the same domain in all the pages
    • Build many DOM trees and link them together
    • Save the multi dimensional tree in a file

  • Compare two DOM trees to assess the similarity

    • Save the DOM tree of the scraped website in a file as a sort of cache to be used to compare with future website structure
    • Create similarity metrics with multiple DOM trees (overall tree? only relevant tags structure?)

  • Nodes for handling authentication

    • Use Selenium or Playwright to handle authentication
    • Passes the cookies to the other nodes

  • Nodes that attaches to an open browser

    • Use Selenium or Playwright to attach to an open browser
    • Navigate inside the browser and scrape the content

  • Nodes for taking screenshots and understanding the page layout

    • Use Selenium or Playwright to take screenshots
    • Use LLM to asses if it is a block-like page, paragraph-like page, etc.
    • Issue #88

Long-Term Goals

  • Automatic generation of scraping pipelines from a given prompt

  • Create API for the library

  • Finetune a LLM for html content