Usage, Testing & Evaluation

Usage

Use the run.py script to process Reddit comment data with the RedTEI pipeline:

Basic Execution:

To run the main pipeline script run.py, you need to provide the path to one or more Reddit comment data files in .zst format.

• Grouped Mode (default):

  python run.py path/to/subreddit_comments.zst

  In this default mode, comments will be grouped by thread.

• No-Group Mode: To process each comment individually (no thread grouping), use the --no-group flag:

  python run.py --no-group path/to/subreddit_comments.zst

For processing multiple files, just provide multiple .zst files as input to run.py.

→ Right after the XML validation, an additional check compares the processed and stored JSON objects with those in the filtered .zst file to confirm that the conversion was likely successful.

Examples

In the examples/demo directory, you’ll find sample JSON files in two folders: grouped and ungrouped. To see the conversion process, run:

python -m extractor.json2xml

The JSON samples will be processed and converted into XML files in their respective folders (grouped or ungrouped).

Testing

To execute the tests, navigate to the root directory of the repository and run:

pytest

This will show and run all tests located in the project's test directories (if any).

Code Coverage Analysis

coverage.py to measure code coverage, helping to identify parts of the codebase that are not adequately tested:

1. Run tests with coverage measurement:

   coverage run -m pytest

2. Generate coverage report:

   coverage report -m --include="extractor/*"

The coverage report will be displayed in your terminal, showing the coverage percentage for each file and indicating any lines that were not executed during testing. Currently 80% code coverage.

Adding New Tests

Contributing new tests is highly encouraged!