Extract text from Html

[MihirModi1421]

Hi there @j-mendez initially thanks a lot for this spider awesome crate

i am using you example scrape.rs to scrape website and along with sublinks of that domain present in it
but it gives me raw html

i want to extract text from raw html provided via scrape any idea or advice is accepted

[j-mendez]

Hi @MihirModi1421 take a look at this example here to extract content as text https://github.com/seoderapp/seoder/blob/main/seoder_lib/src/packages/spider/fs.rs#L171C6-L171C6.

The main gist is to use Html::parse_document(&my_html) and element.text().map(|s| s.chars()).flatten().collect() for the element nodes to get the text content. You can use the selector crate to use CSS selectors to get exactly what you want or just iterate over all the nodes of Html::parse_document.