Punctuation in transcriptions

Punctuation in transcriptions can cause a lot of out-of-dictionary errors since the text transcriptions are tokenized based on whitespace. (This strategy is known to be an issue for orthographic systems that do not use spaces to separate wordlike units. Future development may address this shortcoming.)

While basic functionality for punctuation removal may be useful to some users, the extra complexity adds more areas where the MAPS code can break down. It is also an NLP project in its own right to remove punctuation precisely, which can make it difficult to define the scope of the project. Instead, some basic techniques using external solutions are offered as inspiration. This means that handling punctuation in transcriptions is left up to the user, with the exception that MAPS will uppercase all text, at the time of writing.

Source file

Assume we have a text file with the first sentence of Joyce's Ulysses contents saved as "ulysses.txt".

Stately, plump Buck Mulligan came from the stairhead, bearing a bowl of lather on which a mirror and a razor lay crossed.

Removing punctuation with sed in Bash

Punctuation can be removed from text using the sed command.

sed "s/[.,;]//g" ulysses.txt

which produces

Ulysses without punctuation

Stately plump Buck Mulligan came from the stairhead bearing a bowl of lather on which a mirror and a razor lay crossed

Additional punctuation symbols can be added to the RegEx set so that sed will replace more symbols.

Looping over files

This process can be applied to a number of files in a directory. Assume we have a second file now which contains the first sentence of Don Quijote saved as "quijote.txt". The file has the following contents:

En un lugar de la Mancha, de cuyo nombre no quiero acordarme, no ha mucho tiempo que vivía un hidalgo de los de lanza en astillero, adarga antigua, rocín flaco y galgo corredor.

The following bash command will make a folder called nopunct and save a version of each txt file in the current directory to the nopunct directory, sans the period and comma symbols.

mkdir nopunct; for x in *.txt; do sed "s/[.,]//g" $x > nopunct/$x; done

Ulysses without punctuation

Stately plump Buck Mulligan came from the stairhead bearing a bowl of lather on which a mirror and a razor lay crossed

Quijote without punctuation

En un lugar de la Mancha de cuyo nombre no quiero acordarme no ha mucho tiempo que vivía un hidalgo de los de lanza en astillero adarga antigua rocín flaco y galgo corredor

Using an NLP library

NLP libraries offer a lot of functionality for working with natural language in a text format. More complicated or sophisticated projects may need to use such libraries as NLTK or spaCy. A simple example for the Ulysses text is shown below, but users are encouraged to explore software as needed for their own projects, such as with the NLTK book.

from nltk.tokenize import RegexpTokenizer
rt = RegexpTokenizer('\w+')
with open('ulysses.txt', 'r') as f:
    s = f.read()
    s_nopunct = ' '.join(rt.tokenize(s))
with open('ulysses_nopunct.txt', 'w') as w:
    w.write(s_nopunct)

Result of NLTK tokenization

Stately plump Buck Mulligan came from the stairhead bearing a bowl of lather on which a mirror and a razor lay crossed

Adding words with punctuation to the dictionary

It is also possible to simply add words with punctuation to the dictionary. This process may prove unwieldy for larger amounts of text, but it may be worthwhile for small amounts. In the "ulysses.txt" example, "Stately," could be rendered in a pronunciation dictionary as follows. (Note that MAPS currently assumes Arpabet style transcriptions for English; this may be addressed in future releases.)

Arpabet

STATELY, S T EY1 T L IY0