Punctuation in transcriptions

Punctuation in transcriptions can cause a lot of out-of-dictionary errors since the text transcriptions are tokenized based on whitespace[^roman]. While basic functionality for punctuation removal may be useful to some users, the extra complexity adds more areas where the system can break down. It is also an NLP project in its own right to remove punctuation precisely, which can make it difficult to Instead, some basic techniques using external solutions are offered as inspiration. This means that handling punctuation in transcriptions is left up to the user, with the exception that MAPS will uppercase all text, at the time of writing.

Source file

Assume we have a text file with the first sentence of Joyce's Ulysses contents saved as "ulysses.txt".

Stately, plump Buck Mulligan came from the stairhead, bearing a bowl of lather on which a mirror and a razor lay crossed.

Removing punctuation with sed in Bash

Punctuation can be removed from text using the sed command.

sed "s/[.,;]//g" ulysses.txt

which produces

Ulysses without punctuation

Stately plump Buck Mulligan came from the stairhead bearing a bowl of lather on which a mirror and a razor lay crossed

Additional punctuation symbols can be added to the RegEx set so that sed will replace more symbols.

Looping over files

This process can be applied to a number of files in a directory. Assume we have a second file now which contains the first sentence of Don Quijote saved as "quijote.txt". The file has the following contents:

En un lugar de la Mancha, de cuyo nombre no quiero acordarme, no ha mucho tiempo que vivía un hidalgo de los de lanza en astillero, adarga antigua, rocín flaco y galgo corredor.

The following bash command will make a folder called nopunct and save a version of each txt file in the current directory to the nopunct directory, sans the period and comma symbols.

mkdir nopunct; for x in *.txt; do sed "s/[.,]//g" $x > nopunct/$x; done

Ulysses without punctuation

> Stately plump Buck Mulligan came from the stairhead bearing a bowl of lather on which a mirror and a razor lay crossed

Quijote without punctuation

En un lugar de la Mancha de cuyo nombre no quiero acordarme no ha mucho tiempo que vivía un hidalgo de los de lanza en astillero adarga antigua rocín flaco y galgo corredor

Adding words with punctuation to the dictionary

It is also possible to simply add words with punctuation to the dictionary. This process may prove unwieldy for larger amounts of text, but it may be worthwhile for small amounts. In the "ulysses.txt" example, "Stately," could be rendered in an pronunciation dictionary as:[^format]

Arpabet

STATELY, S T EY1 T L IY0

[^roman]: This strategy is known to be an issue for orthographic systems that do not use spaces to separate wordlike units. Future development may address this shortcoming.

[^format]: Note that MAPS currently assumes Arpabet style transcriptions for English. This may be addressed in future releases.