[AiLab] prevent save of trained models if they don't pass share filtering #40915
Conversation
| return head :bad_request if model_data.nil? || model_data == "" | ||
| profanity_or_pii = ShareFiltering.find_failure(model_data.to_s, request.locale) | ||
| if profanity_or_pii | ||
| render json: {id: model_id, status: "failure", details: profanity_or_pii.type} |
There was a problem hiding this comment.
Out of curiosity, what are the possible values of profanity_or_pii.true? We want to make sure all possible values are unique to profanity filtering failures, so that the client code doesn't have to make too many assumptions about what type of error this is. Otherwise, we could return a more explicit error indicating that it relates to profanity filtering...
There was a problem hiding this comment.
The possible values for profanity_or_pii.type are email, phone, address and profanity found here.
There was a problem hiding this comment.
I think I see what you mean, I updated details to be more specific.
There was a problem hiding this comment.
We decided that it wasn't important to specify the type of error so I modified to render a new status: piiProfanity that the code in AI Lab now handles to display the correct, generic message.
…ring::FailureType
We don't want students to be able to save trained machine learning models if they contain profanity or personally identifying information (pii). Saved models can be imported into App Lab apps, which can be published or remixed and we don't want indecent or unsafe information shared. Prior to saving a model, we now run its data through the share filter, which will check for profanity, emails, phone numbers or street addresses and prevent the model from saving if any are found.
We'll show an alternate fail message if profanity or pii is found:
code-dot-org/ml-playground#218