Adding filter to crawl

Filtered Crawl Process

The crawl process has been refined to ensure that only relevant content is retained. Here's a detailed breakdown of the steps involved:

Initial Crawl:

The crawler begins by fetching web pages based on the initial set of URLs.
Content Extraction:

Extract the main content from each page, focusing on textual information.
Maintain the structure of the content, including headings, paragraphs, and lists.
Link Removal:

Identify and remove all hyperlinks from the extracted content.
Ensure that the removal of links does not disrupt the overall structure or readability of the text.
Code Structure Preservation:

Preserve the Python indentation and code structure within the markdown.
Ensure that code blocks are properly formatted and indented for clarity.
Final Markdown Generation:

Compile the filtered content into a markdown format.
The final markdown will be concise, with a focus on readability and structure, free from extraneous links.