NLP-Data-Cleaning-Task

We explore the use of regular expressions to clean input text in support of a downstream task found here.

Data

Input data: Child Language Data Exchange System (CHILDES Talkbank)

The data contains transcripts from oral language that represent the language that young children are exposed to in everyday settings. The data is stored in .cha files.

NLP-Data-Cleaning-Task/
├─ Data/
│  ├─ Bates
│  │  ├─ Free20
│  │  │  ├─ amy.cha
│  │  │  ├─ betty.cha
│  │  │  ├─ (more files)
│  │  ├─ (more directories)
│  ├─ (more directories)

Tasks

Task 1: clean the files:

• Convert the files to raw text and remove both the header and extraneous information. Extraneous information is any information that is not needed to support the downstream task.

Task 2: transform the files:

• You will transform the data so that it uses ArpaBET to represent the sounds that the text maps to. We use CMU's Pronunciation Dictionary to inform this transformation.

Report

Further details and justifications can be found here.

Contributors

Leen Alzebdeh @Leen-Alzebdeh

Sukhnoor Khehra @Sukhnoor-K

Resources Consulted

https://www.w3schools.com/python/python_regex.asp

Jurafsky, D., & Martin, J. H. (2009). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall.

CMU’s Pronunciation Dictionary

ARPABET

Child Language Data Exchange System (CHILDES Talkbank)

https://stackoverflow.com/questions/16510017/how-to-use-regular-expressions-do-reverse-search

https://stackoverflow.com/questions/3114252/one-liner-to-check-whether-an-iterator-yields-at-least-one-element

GitHub Copilot

Libraries

We run this project using standard Python libraries re (regex), random, and os.

Instructions to execute code

To run this project:

1. Ensure Python is installed, as well as the Python Standard Library.
2. Clone the repository.
3. Ensure you have CHILDES input data in the format outlined above and in a directory 'Data/'
4. Run the main.py file (no parameters needed). Run this command

   python3 src/main.py

Clean files can be found within the clean/ directory found in root.

Transformed files can be found within the transformed/ directory found in root.