Apache Tika 4 Text Extraction

Apache Tika 4.x Text Extraction & Process Isolation

OpenCrawling leverages Apache Tika 4.1.0 as its enterprise-grade text extraction and document metadata parsing engine. Beginning in OpenCrawling 1.0.0, document text extraction is decoupled and executed using Tika's PipesForkParser, providing hard process isolation, watchdog timeouts, and memory bounds to protect crawler and worker runtimes from unhandled crashes, infinite loops, and Out-Of-Memory (OOM) errors.

---

🛡️ Why Process Isolation?

Traditional document parsing in Java executes parsers (PDFBox, Apache POI, Commons Compress) in-process within the main application heap. In high-throughput enterprise crawling, this exposes worker containers to critical failure modes:

1. Parser Hangs & Infinite Loops: Corrupted or adversarial documents can cause regex catastrophic backtracking or infinite loops inside deep parser stacks.
2. Out-of-Memory (OOM) Errors: Decompression bombs (e.g., zip bombs) or massive rasterized PDFs can exhaust the container's heap, killing the entire worker or ingestion service.
3. JVM Segfaults: Native libraries or JNI bindings (e.g., font renderers, compression codecs) can crash the JVM outright without throwing a catchable Java Exception.
4. Memory Leaks: Long-running workers accumulating uncollected parser cache buffers degrade performance over time.

The Solution: PipesForkParser

OpenCrawling's PipesForkTextExtractor delegates document parsing to a dedicated, watchdog-monitored child JVM:

flowchart TD
    subgraph HostWorker["OpenCrawling Host Worker (oc-runtime / IngestionConsumer)"]
        Stream["Document Stream / Claim-Check"]
        Service["TextExtractionService (PipesForkTextExtractor)"]
        Watchdog["Timeout Watchdog Timer (30s)"]
    end

    subgraph ForkedJVM["Isolated Child Process (PipesServer JVM)"]
        TikaCore["Apache Tika 4.1.0 Engine"]
        Parsers["PDF, Office, HTML, OCR Parsers"]
    end

    Stream --> Service
    Service -->|Pipes IPC / Stdio Discard| ForkedJVM
    Watchdog -.->|Hard Kill on Timeout| ForkedJVM
    ForkedJVM -->|TextExtractionResult & Metadata| Service
    Service --> Chunks["Kafka Chunks Topic (opencrawling-chunks)"]

Loading

• Hard Deadline Enforcement: If parsing exceeds timeout-ms (default 30s), the watchdog forcibly terminates the child process (SIGKILL). The main worker survives and gracefully records a failure or falls back to plain-text recovery.
• Isolated Memory Bounds: The child process is constrained to its own heap via jvm-args (e.g., -Xmx512m), ensuring heap exhaustion never impacts host services.
• Process Recycling: After parsing a configurable number of files (max-files-per-process, default 10,000), the child process is automatically recycled to prevent slow native memory leaks.
• Graceful In-Process Fallback: If process isolation is disabled or fails to initialize on constrained architectures, the system automatically falls back to embedded in-process Tika extraction.

---

🏛️ Architecture & Component Design

The text extraction subsystem is centralized in oc-core and exposed through standard Spring Boot auto-configuration:

1. Central Interfaces & Services

• TextExtractionService: The high-level SPI for text extraction, accepting document byte[] arrays or InputStream with initial metadata (such as mimeType and filename).
• PipesForkTextExtractor: The primary implementation orchestrating PipesForkParser, process watchdog monitoring, and library discovery.
• TextExtractionProperties: Spring Boot @ConfigurationProperties(prefix = "opencrawling.tika") binding all extraction parameters.
• TextExtractionResult: Encapsulates extracted text, merged metadata, truncation flags, and error messages.

2. Output Connectors Integration

All 9 OpenCrawling output connectors consume the centralized TextExtractionService instead of bundling individual Tika dependencies:

• oc-luxir-output-connector (Luxir Search Engine)
• oc-milvus-output-connector (Milvus Vector DB)
• oc-opensearch2-output-connector (OpenSearch 2.x)
• oc-opensearch3-output-connector (OpenSearch 3.x)
• oc-qdrant-output-connector (Qdrant Vector DB)
• oc-seatunnel-output-connector (Apache SeaTunnel)
• oc-solr-output-connector (Apache Solr)
• oc-vector-output-connector (PostgreSQL pgvector)
• oc-vespa-output-connector (Vespa Engine)

---

📦 Container Packaging & Classpath Auto-Discovery

Because PipesForkParser spawns a standalone child JVM process (java -cp ... org.apache.tika.pipes.PipesServer), all Tika parsers and third-party dependencies must be accessible on the child classpath.

Container Thin Jar Layout

OpenCrawling Docker images (Dockerfile.crawler, Dockerfile.ingestion-consumer, Dockerfile.writer-consumer, and oc-runtime/Dockerfile) leverage Spring Boot jar extraction:

# Extract the fat jar into thin application jar and isolated libraries
RUN java -Djarmode=tools -jar app.jar extract --destination /app/extracted

# Copy dependencies to /app/lib
COPY --from=builder /app/extracted/lib ./lib
COPY --from=builder /app/extracted/oc-runtime-1.0.0-SNAPSHOT.jar app.jar

ENV TIKA_EXTRAS_DIR=/app/lib
ENTRYPOINT ["java", "-Dtika.extras.dir=/app/lib", "-jar", "app.jar"]

Auto-Discovery Resolution Hierarchy

When launching the child process, PipesForkTextExtractor resolves tika.extras.dir using the following order of precedence:

1. System Property: -Dtika.extras.dir=/path/to/lib
2. Environment Variable: TIKA_EXTRAS_DIR=/path/to/lib
3. Standard Layout Paths: Checks ./lib and /app/lib
4. Sibling Lib Directory: Discovers ../lib relative to the executing JAR
5. Fat-Jar Extraction Fallback: If running from an unextracted Spring Boot uber JAR in development, it extracts BOOT-INF/lib/*.jar into a temporary directory (/tmp/opencrawling-tika-fork-libs) with built-in Zip Slip protection (CWE-022).

---

🔒 Security & Safe Extraction (Zip Slip Defense)

When unpacking JARs or archives, PipesForkTextExtractor enforces strict Zip Slip path traversal defenses:

• Rejects any archive entry whose name contains .. path traversal sequences.
• Normalizes destination paths using toAbsolutePath().normalize().
• Verifies that the resolved target path begins with the destination directory boundary (outFile.startsWith(destinationDir)), throwing a SecurityException upon any escape attempt.
• Automatically strips ASCII null bytes (\u0000) from extracted text to prevent UTF-8 database insertion errors in PostgreSQL (pgvector).

---

⚙️ Configuration Reference

1. Spring Boot Properties (application.yml)

opencrawling:
  tika:
    enabled: true                     # Enable or disable text extraction
    fork-enabled: true                # Enable PipesForkParser process isolation
    timeout-ms: 30000                 # Hard deadline per document (30s)
    write-limit: 20000000             # Max characters extracted (20M)
    num-clients: 1                    # Parallel forked client processes
    max-files-per-process: 10000      # Recycle child process after N documents
    extract-embedded: false           # Extract embedded documents/attachments
    jvm-args:
      - "-Xmx512m"                    # Child JVM heap limit

2. Environment Variables

Environment Variable	Spring Property	Default	Description
OPENCRAWLING_TIKA_ENABLED	opencrawling.tika.enabled	true	Master switch for text extraction
OPENCRAWLING_TIKA_FORK_ENABLED	opencrawling.tika.fork-enabled	true	Enable process-isolated fork parsing
OPENCRAWLING_TIKA_TIMEOUT_MS	opencrawling.tika.timeout-ms	30000	Hard watchdog timeout in milliseconds
OPENCRAWLING_TIKA_WRITE_LIMIT	opencrawling.tika.write-limit	20000000	Max character extraction limit (-1 for unlimited)
OPENCRAWLING_TIKA_NUM_CLIENTS	opencrawling.tika.num-clients	1	Number of concurrent forked client workers
OPENCRAWLING_TIKA_MAX_FILES_PER_PROCESS	opencrawling.tika.max-files-per-process	10000	Process recycling threshold
OPENCRAWLING_TIKA_EXTRACT_EMBEDDED	opencrawling.tika.extract-embedded	false	Recursively extract embedded attachments
OPENCRAWLING_TIKA_JVM_ARGS	opencrawling.tika.jvm-args	-Xmx512m	Memory and JVM parameters for child processes
TIKA_EXTRAS_DIR	tika.extras.dir	/app/lib	Directory path containing extra parser JARs

---

🖥️ Admin UI Controls

Administrators can inspect and manage Tika process isolation directly from oc-admin-ui:

1. Dashboard Indicator: Displays the active text extraction engine:
   • PipesForkParser (Process-Isolated): Child JVM watchdog monitoring is active.
   • In-Process Embedded Tika: Direct in-memory parsing is active.
2. Settings Panel -> Text Extraction (Tika):
   • Toggle between PipesForkParser and Embedded In-Process Tika.
   • Adjust Watchdog Timeout slider (5s – 120s).
   • Configure Forked JVM Heap Limit (-Xmx256m, -Xmx512m, -Xmx1024m, -Xmx2048m, or custom).
   • Set Process Recycling Threshold and Character Write Limits.
   • Toggle Embedded Attachment Extraction.

---

🧪 Testing & Verification

Run the core text extraction unit test suite to verify fork isolation, fallback, and archive security:

mvn test -pl oc-core -Dtest=PipesForkTextExtractorTest

Expected output:

[INFO] Running org.opencrawling.core.text.PipesForkTextExtractorTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS