Processing Pipeline

Processing Pipeline

RedTEI processes Reddit comments in three stages: filtering, extraction, conversion. It offers two main processing modes that affect how comments are saved.

Processing Modes

• Grouped Mode (default): Comments are grouped by Reddit threads. All comments belonging to the same thread are processed and stored together in a single output file (both JSON and XML).

• No-Group Mode (--no-group): When the --no-group flag is activated, the pipeline processes each comment individually. Each comment, regardless of its thread, is processed and stored as a separate file (both JSON and XML).

---

1. Filtering JSON Objects

Script: trim_username_comments.py

The trim_username_comments.py script is responsible for filtering and modifying JSON objects directly within the .zst archives before further processing. This step is important for cleaning and preparing the comment data.

JSON objects are excluded from further processing if they meet any of the following criteria:

• Deleted/Removed Comments: Comments where the "body" key value is "[removed]", "[deleted]", or "[removed by reddit]" are excluded. These are comments that were removed by moderators or Reddit itself.

• Bot Authors: Comments authored by known bots are excluded. The script uses a list of bot usernames defined in src/config/botlist.txt. If the value of the "author" key matches a bot name in this list, the comment is excluded.

• RemindMeBot Requests: Comments that are identified as requests to the RemindMeBot (e.g., !RemindMe 2 days) are filtered out.

• URL-Only Comments: Comments that consist solely of a plaintext URL, or after URL removal, contain only [URL] placeholders (optionally followed by punctuation or whitespace), are excluded. This filter aims to remove low-content comments that primarily serve as link dropping.

In addition to exclusion, the script applies the following modifications to the comment text ("body"):

• Quote Removal: Quotations within comments are removed to clean up the text and focus on original content.

• URL Removal and Replacement:

  • Plaintext URLs (e.g., http://example.com) are replaced with the placeholder [URL].
  • Markdown URLs (e.g., [Example](https://example.com)) are processed to keep the link text (e.g., Example), unless the link text itself is also a URL, in which case it is replaced with [URL].

• Inline Formatting Removal:

  • Bold Text: Double asterisks ** surrounding text are removed, preserving the text (e.g., **Text** becomes Text).
  • Italic Text: Single asterisks * surrounding text are removed, preserving the text (e.g., *Text* becomes Text).
  • Strikethrough Text: Tildes ~~ and the enclosed strikethrough text are completely removed.

• Zero-Width Space Removal: All Zero-Width Space characters (\u200B, ​, ​) are removed to ensure cleaner text.

• Newline Reduction: Multiple consecutive newline characters are reduced to a single newline character to normalize spacing.

• Empty Comment Discarding: After all modifications, comments that are empty or consist only of whitespace are discarded and not processed further.

All filtering and modification actions (except inline formatting removals) are logged in a text file named filtered_log_{input_filename}.txt. This log documents the original content of comments that were filtered or modified, providing transparency and traceability to the filtering process. A single comment might appear multiple times in the log file if it was subject to multiple filtering actions (e.g., quote removal and URL removal).

---

2. Extraction to JSON Files

Scripts: comment_tree.py & comment_processing.py

The scripts comment_tree.py and comment_processing.py work together to extract comments from the filtered .zst files and store them as JSON files. The format and structure of these JSON files depend on the chosen processing mode (grouped or no-group).

Grouped Mode (default):

• Thread-Based Grouping: Comments are grouped into threads based on their link_id. All comments sharing the same link_id are part of the same thread.
• Single JSON File per Thread: For each thread, all its comments are stored in a single JSON file.
• Flat List of Comments: Within each JSON file, comments are stored as a flat list of JSON objects. The original tree-like structure of replies and nested comments within the thread is not preserved.
• Filename Convention: JSON files are named using the link_id of the thread, followed by _flat.json. For example, a file for thread ID 10ax890 would be named 10ax890_flat.json.

No-Group Mode (--no-group):

• Individual Comment Processing: Each comment is processed and extracted independently, without considering thread groupings.
• Separate JSON File per Comment: Each comment is stored in its own JSON file.
• Single JSON Object per File: Each JSON file contains only one JSON object (= a single comment).
• Filename Convention: JSON files are named using a combination of the link_id and the id of the comment, separated by an underscore and with the .json extension. For example, a comment with link_id 10wugax and id jepcf1r would be named 10wugax_jepcf1r.json.

---

3. Conversion to XML

Script: json2xml.py

The json2xml.py script performs the final step of the pipeline: converting the JSON files generated in the previous step into TEI-XML format. The structure of the resulting XML files is influenced by the processing mode selected (grouped or no-group).

Grouped Mode (default):

• One XML File per Thread: For each thread (represented by a JSON file), the script creates one corresponding XML file.
• TEI Header: Each XML file starts with a TEI header (<teiHeader>) that contains metadata about the thread. This metadata is extracted from the first comment in the JSON file and includes:
  • Title of the thread (derived from link_id)
  • Subreddit name
  • Date of the last comment in the thread
  • URL of the Reddit thread
• XML Body Structure: The XML body (<text><body/>) contains the comment data.
  • Comments are enclosed within a <div type="comments"><list>...</list></div> structure.
  • Each individual comment is represented as an <item> element within the <list>.
  • Each <item> element contains:
    • The comment text itself.
    • A <date> element indicating the comment's creation date.
    • A <name> element specifying the author's username.
    • A source attribute on the <item> element, containing the URL of the comment on Reddit.
    • Line breaks within the comment text are encoded using <lb/> elements in the XML.

No-Group Mode (--no-group):

• One XML File per Comment: For each individual comment (represented by a JSON file), the script generates a separate XML file.
• TEI Header: The TEI header in "no-group" mode contains more detailed metadata compared to the grouped mode. This includes:
  • Title of the thread (derived from link_id)
  • Subreddit name
  • Date of the comment
  • URL of the Reddit thread
  • URL of the specific comment
• XML Body Structure: The XML body in "no-group" mode is simpler.
  • Each comment is placed directly within the <body> as a <p> (paragraph) element.
  • The <p> element contains only the comment text. It does not include <item>, <list>, <date>, or <name> elements within the comment body itself.
  • Line breaks in the comment text are again encoded as <lb/> elements.

XML Validation:

After the conversion to XML, the validate.py script is used to check the validity of the generated XML files against a defined TEI schema (tei_corpus.dtd). This validation step ensures that the XML output is well-formed and conforms to the TEI guidelines.