Markdown Processor and Embedder

md-embed processes markdown files, cleans and prepares the data, splits the text into manageable chunks, and creates embeddings for use in vector databases (specifically ChromaDB). It supports multiple input methods and provides options for customizing the splitting and embedding process.

Features

• Multiple Input Methods:
  
  • JSON file containing URLs and markdown data
    
  • Folder of markdown files
    
  • Single markdown file
    
• Data Cleaning:
  
  • Removes duplicate entries based on URL section titles
    
  • Handles encoding issues
    
  • Sanitizes filenames for safe saving
    
  • Optionally filters out files containing "404" and "page not found" (can be disabled)
    
  • Removes lines containing the copyright symbol "©"
    
• Text Splitting:
  
  • Markdown Header Splitting: Splits text based on specified markdown header levels (e.g., #, ##). Allows for custom header level selection. Preserves header hierarchy in metadata
    
  • Recursive Character Text Splitting: Splits text into chunks of specified size and overlap
    
  • Link Removal: Optionally removes markdown links, keeping only the link text
    
• **Embedding Generation:*
  
  • Supports Hugging Face embeddings (using langchain_huggingface). Defaults to all-MiniLM-L6-v2
    
  • Supports Ollama embeddings (using langchain_community). Defaults to nomic-embed-text, requires a local Ollama server running at http://localhost:11434
    
• Vector Database Integration:
  
  • Uses ChromaDB (langchain_chroma) to store embeddings and associated metadata
    
  • Allows specifying the collection name and persistence directory
    
  • Handles large datasets by processing in batches
    
• Logging:
  
  • Comprehensive logging through the logging module
    
• Duplicate Logs:
  
  • Writes URLs with duplicate sections to a log
    
• Removed Files Logs
  
  • Write to a log files that have been removed due to filters
    

Requirements

• Python 3.7+
  
• langchain (various components - see import statements)
  
• chromadb
  
• tqdm
  
• beautifulsoup4 (if you were scraping, but this script doesn't actually use it)
  
• requests (if you were scraping, but this script doesn't actually use it)
  

To install the required packages, run:

pip install langchain langchain-chroma langchain-huggingface tqdm

If you are planning to use Ollama, you need to:
Install Ollama by following the instructions provided at Ollama's official website.
Run an Ollama server locally on port 11434

md-embed can be run from the command line. It provides a command-line interface using argparse with the following option:
--filters-off: Disables the "404" and "©" filters
The script will then guide you through a series of interactive prompts to configure the processing:
Input Method Selection: Choose between JSON input, a folder of markdown files, or a single markdown file
Input File/Folder/URL: Provide the path to the input file or folder, as appropriate
Output Folder (for JSON input): Specify the directory where cleaned markdown files will be saved
Data Cleaning Options: The script will show total entires and total duplicates
Language: Specify the primary language of the input files (e.g., "TypeScript", "Python")
Splitting Method: Choose between "markdown" (header-based splitting) and "recursive" (chunk size and overlap)
Markdown Splitting Options (if applicable):
Remove Links: Choose whether to remove markdown links
Header Levels: Specify which header levels to split on (e.g., "1,2,3" for #, ##, and ###). Enter "all" for all header levels
Recursive Splitting Options (if applicable):
Remove Links: Choose whether to remove markdown links
Chunk Size: Specify the desired chunk size (in characters)
Chunk Overlap: Specify the desired chunk overlap (in characters)
Preview Splits: Choose whether to preview the split data ("yes", "full", or "no")
Split Again: You'll be prompted to continue or modify the settings
Embedding Method: Choose between "huggingface" and "ollama"
Embedding Model (Hugging Face): Enter the Hugging Face model name (defaults to all-MiniLM-L6-v2)
Embedding Model (Ollama): Enter the Ollama model name (defaults to nomic-embed-text)
Persistence Directory: Specify the directory where the ChromaDB database will be stored
Collection Name: Choose a name for the ChromaDB collection
Example (JSON Input):

python md-embed.py

Follow the prompts, providing the necessary information (input file, output folder, embedding choices, etc.)
Example (Disabling Filters):

python md-embed.py --filters-off

Cleaned Markdown Files (JSON Input): If using JSON input, the script will save cleaned markdown files to the specified output folder
ChromaDB Database: The script will create a ChromaDB database in the specified persistence directory, containing the embeddings and metadata
Logs: The logs directory will contain logs of removed files (if any) and duplicate entries (if using JSON input)
file_to_url.json: Json file that contains the original URL of each document
Error Handling
The script includes error handling for various scenarios, such as:
Invalid input file/folder paths
File I/O errors
Exceptions during data cleaning, splitting, or embedding
Invalid user input for prompts
Errors are logged using the logging module
Notes
The script assumes that the input JSON data has "url" and "markdown" keys for each entry
The script uses uuid4 to generate unique IDs for each document in the vector database
The script processes in batches to deal with a large number of splits