Define general rules for all sentences

[mone27]

I think it is important to define some rules for processing sentences from all importers.
This checks can either be done in the wrapper script or in sanitize.py (this can be more efficient)
My proposal is:

• everything should be converted to lowercase
• [^\s'abcdefghijklmnopqrstuvwxyzàèéìíòóôùú,\.!?:;] if a sentence match this regex it contains not valid chars so should be discarded

The discard will be done after trying to clean sentences (like removing trailing dashes or unescaping html)

[Mte90]

For the first case we need to see what happens to various exporter.
My idea is to keep those in this way the corpus can be used also elsewhere.

About the second, for the symbols I think that we need to add more like `-\°()[]€$ but it is something that we can improve.

[Mte90]

We can add it to https://github.com/MozillaItalia/DeepSpeech-Italian-Model/tree/master/MITADS

[nefastosaturo]

Right now, in merge_text.sh there is a final regex that filters each sentence. The final regex is:

^[ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzèÈ][ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzàÀèÈéÉìÌòÒùÙ' ]+$

so, right now there are no special needs for this task so I'm closing this issue