Why Disable 'preserve_accents'?

[dcampbell24]

It looks like you had the option to preserve the accents (and diacritics). Why did you comment this out? What if the application is intended for English and non-English speakers?

[finnbear]

Hello!

Why did you comment this out?

Because I wanted to implement but it wasn't possible to do so at the time.

Since then, I've tried multiple ground up rebuilds of the filter to support features like keeping accents, but they either didn't work or ran at speeds like 0.01MB/s.

What if the application is intended for English and non-English speakers?

One solution is to let the original message pass through if the filter's analysis function returns an acceptable Type. This is not totally safe, though, as diacritics can significantly alter how text is displayed and filtered.

[dcampbell24]

0.01 MB/s is plenty fast for my application. Text messages are limited to 128 characters and I doubt they will be that long in practice. I don't think a high number of messages will ever be sent. I am guessing 0.01 MB/s is fast enough for a lot of applications, but you can give a warning that enabling it will slow down censoring significantly.

[finnbear]

Unfortunately, at 0.01 MB/s, I would struggle to develop and test the filter, as my test cases are extensive and slow even at the current 3MB/s speed.

I will continue to think of ways to support features like this, while keeping the speed high, but I don't have a time estimate for when I'll succeed.

[dcampbell24]

I can improve the situation a little by doing:

fn censor(text: String) -> String {
    let (censored, analysis) = Censor::from_str(&text)
        .with_censor_threshold(Type::PROFANE | Type::SEXUAL)
        .with_censor_first_character_threshold(Type::ANY)
        .censor_and_analyze();

    if analysis == Type::NONE {
        text
    } else {
        censored
    }
}

and it would be a even a little better if censor returned the original string if it didn't do any censoring. So, in the example above, ä nazi would return ä nazi instead of a nazi.

If you are busy and you agree with the change, I can make a stab at the change.

[finnbear]

it would be a even a little better if censor returned the original string if it didn't do any censoring

This would undermine the filter in certain cases. Consider the text ͨ ͬ ͣ p which is mostly composed of diacritics that look like Latin letters spelling "cr*p". The filter wouldn't detect profanity, and would return the string as-is, instead of an innocent, nearly-empty string.

(this example may not work in your browser, so here is a screenshot in Firefox:
)

If you are fine with this exploit, the code snippet you wrote is perfect!

[dcampbell24]

I think I will allow it. I am really trying to go by what OARS allows, which is not at all clear.