Skip to content

Preprocessing

Jump to bottom
Tang ZhaoSheng edited this page Nov 5, 2024 · 13 revisions

This Python script processes PDFs, extracting both text and tabular data, and cleans it for easier analysis. It's designed for structured reports, particularly ESG (Environmental, Social, and Governance) documents. The code combines natural language processing (NLP) and table extraction.

Libraries Used:

  • PyPDF2: Extracts text from PDF pages.

  • nltk (Natural Language Toolkit): Tokenizes, removes stopwords, and lemmatizes text.

  • re (regular expressions): Cleans and formats text.

  • pandas: Structures extracted tables in DataFrames.

  • gmft.auto (CroppedTable, AutoTableDetector, AutoTableFormatter): Detects and formats tables.

  • gmft.pdf_bindings (PyPDFium2Document): Reads PDF data for table detection.

Workflow:

  1. SSL Workaround and NLTK Downloads:

  • Ensures NLTK data (tokenizers, stopwords, lemmatizers) can download without SSL issues.

  1. PDF Text Extraction (extract_text_from_pdf):

  • Uses PyPDF2 to read and extract text from PDF pages, storing it as a single string.

  1. Text Cleaning (clean_text):

  • Removes extra spaces.

  • Preserves punctuation (e.g., periods, commas) and numbers.

  • Simplifies text for further processing.

  1. Tokenization and Preprocessing (tokenize_text, remove_stopwords, lemmatize_tokens):

  • Tokenization (tokenize_text): Splits text into sentences and words.

  • Stopword Removal (remove_stopwords): Filters out common, less meaningful words.

  • Lemmatization (lemmatize_tokens): Reduces words to their base forms for consistency.

  1. Preprocessing Text (preprocess_text):

  • Combines cleaning, tokenization, stopword removal, and lemmatization into one function.

  1. Formatting Text for AI (format_for_ai):

  • Structures processed sentences for compatibility with AI models.

  1. PDF Table Extraction (extract_tables_from_pdf):

  • Uses gmft’s tools to detect tables with a minimum of two columns and two rows.

  • Extracts tables from each page of the PDF.

  1. Formatting Tables (format_table):

  • Converts detected tables into DataFrames, fills missing values, and formats them as Markdown tables.

  1. Process PDF (process_pdf):

  • Extracts and preprocesses text.
  • Extracts and formats tables.
  • Outputs cleaned text and formatted tables.

  1. Saving the Output:

  • Saves the cleaned text and tables as a .txt file in the ../txt_files directory.

Use Case:

Ideal for structured documents like ESG reports, combining narrative text and statistical tables. The output is suitable for machine learning tasks.

Example Output:

  • Preprocessed Text: Cleaned and structured text for AI analysis.

  • Extracted Tables: Detected tables with headers and rows formatted for readability.

Example Table Output:

================================================================================
Table 1
================================================================================
Column1  | Column2  | Column3
---------|----------|---------
123      | 45%      | 2023
================================================================================

Python Code Documentation: report_convert.py

This script is similar but adds directory and SSL management.

Key Libraries:

  • PyPDF2: Reads and extracts text from PDFs.

  • nltk (Natural Language Toolkit): Provides tools for text tokenization, stopword removal, and lemmatization.

  • re (regular expressions): Cleans text.

  • ssl: Manages secure connections for NLTK downloads.

  • os: Manages files and directories.

Workflow:

  1. SSL Workaround for NLTK Downloads:

  • Ensures SSL downloads work for required NLTK corpora.

  1. NLTK Data Downloads:

  • Downloads essential NLTK tools (e.g., punkt, stopwords, wordnet).

  1. Extract Text from PDF (extract_text_from_pdf):

  • Reads and combines text from each PDF page.

  1. Clean Text (clean_text):

  • Replaces spaces and special characters to simplify text.

  1. Tokenize Text (tokenize_text):

  • Splits text into sentences and words.

  1. Remove Stopwords (remove_stopwords):

  • Filters out common, less meaningful words.

  1. Lemmatize Tokens (lemmatize_tokens):

  • Standardizes words to base forms.

  1. Preprocess Text (preprocess_text):

  • Combines cleaning, tokenization, stopword removal, and lemmatization.

  1. Detect and Format Tables (detect_and_format_tables):

  • Identifies and structures tabular data in Markdown format.

  1. Format for AI (format_for_ai):

  • Structures preprocessed text and formatted tables for AI models.

  1. Main Function (extract_and_preprocess_esg_data):

  • Extracts and preprocesses text and saves it as a Markdown .txt file.

Use Case

Processes ESG or other reports with mixed text and table data for easy machine learning use.

Example Output

The text file would contain clean, preprocessed text with tables:

Table header 1 | Table header 2 | Table header 3
---------------|----------------|---------------
Data1          | Data2          | Data3

More text follows here...

The structure ensures that the data is neatly organized and ready for analysis or modeling.

Clone this wiki locally