StringError("stream did not contain valid UTF-8")

[jicee13]

I'm very new to rust so forgive me if this is a poor question. However all I am trying to do is get the data from a url and read it into a Document however I keep getting this error

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") }'

I took a look at this issue however my code still isn't too happy. All I'm doing is this..

let url = "http://www.google.com";
let resp = reqwest::get(url).unwrap();
let document = Document::from_read(resp).unwrap();

It works on other websites, but no all (including google). Am I doing something wrong? Could someone point me in the right direction?

[seanmonstar]

Sometimes websites don't send the bytes of their body in UTF-8 encoding, but some other encoding. I'm not certain what Document is exactly, but likely when you call Document::from_read, it reads the response body and tries to decode it as UTF-8. You could probably do a few different things to fix your issue:

• If the pages are never that big, or you don't mind reading the full page into memory, you could make use of Response::text to try to auto-decode the body into UTF-8. Then you would feed that new string into Document.

  let mut resp = reqwest::get(url).expect("reqwest GET");
  let text = resp.text().expect("response text");
  let doc = Document::from_read(::std::io::Cursor::new(text.into_bytes())).expect("Document from_read");

  You'd need to check the documentation for Document, but there may be other constructors that don't require an impl Read, so you may not need to wrap the string in a Cursor.

• Look into using something like the encoding_rs crate to decode the body while reading. This might result in reducing a copy, and may help not read the entire body into memory, depending on what Document::from_read actually does.

[jicee13]

Awesome, that first bullet point worked perfectly. Thank you very much.

For anyone interested, this is all I do to grab all the images from a page.

let mut test_resp = reqwest::get(url).expect("reqwest GET");
let text = test_resp.text().expect("response text");
let doc = Document::from_read(::std::io::Cursor::new(text.into_bytes())).expect("Document from_read");
let nodes = doc.find(Name("img"));
for node in nodes {
    let img_info = node.attr("src");
    println!("{}", img_info.unwrap());
}