Efficient `from_utf8_lossy` for `SmolStr`?

[Jason5Lee]

What's the best way to convert bytes or OsStr to a SmolStr using lossy UTF-8 decoding, avoiding the extra allocation of to_string_lossy().to_smolstr()?

[alexheretic]

You can port String::from_utf8_lossy to SmolStr externally using SmolStrBuilder. This could also be added as a fn to this crate.

However, to be efficient it would also have to add a equivalent Cow concept to handle valid long utf8 slices without allocating similarly to std.

pub enum CowSmolStr<'a> {
    Borrowed(&'a str),
    Owned(SmolStr),
}

fn smolstr_from_utf8_lossy(v: &[u8]) -> CowSmolStr<'_> {
    let mut iter = v.utf8_chunks();

    let first_valid = if let Some(chunk) = iter.next() {
        let valid = chunk.valid();
        if chunk.invalid().is_empty() {
            debug_assert_eq!(valid.len(), v.len());
            return CowSmolStr::Borrowed(valid);
        }
        valid
    } else {
        return CowSmolStr::Borrowed("");
    };

    const REPLACEMENT: &str = "\u{FFFD}";

    let mut res = SmolStrBuilder::new();
    res.push_str(first_valid);
    res.push_str(REPLACEMENT);

    for chunk in iter {
        res.push_str(chunk.valid());
        if !chunk.invalid().is_empty() {
            res.push_str(REPLACEMENT);
        }
    }

    CowSmolStr::Owned(res.finish())
}

[Jason5Lee]

You can port String::from_utf8_lossy to SmolStr externally using SmolStrBuilder. This could also be added as a fn to this crate.

However, to be efficient it would also have to add a equivalent Cow concept to handle valid long utf8 slices without allocating similarly to std.

pub enum CowSmolStr<'a> {
Borrowed(&'a str),
Owned(SmolStr),
}

fn smolstr_from_utf8_lossy(v: &[u8]) -> CowSmolStr<'_> {
let mut iter = v.utf8_chunks();

let first_valid = if let Some(chunk) = iter.next() {
    let valid = chunk.valid();
    if chunk.invalid().is_empty() {
        debug_assert_eq!(valid.len(), v.len());
        return CowSmolStr::Borrowed(valid);
    }
    valid
} else {
    return CowSmolStr::Borrowed("");
};

const REPLACEMENT: &str = "\u{FFFD}";

let mut res = SmolStrBuilder::new();
res.push_str(first_valid);
res.push_str(REPLACEMENT);

for chunk in iter {
    res.push_str(chunk.valid());
    if !chunk.invalid().is_empty() {
        res.push_str(REPLACEMENT);
    }
}

CowSmolStr::Owned(res.finish())

}

Thanks. I didn't expect the implementation to be so simple. For me, implementing it myself is already enough.

Regarding CowSmolStr, I have another idea and opened a separate issue. #95