Document Lifecycle Actions and Tombstones

OIS Document Lifecycle Actions & Tombstone Deletes 🪦

OpenCrawling supports the Open Ingestion Standard (OIS) specification for document lifecycle management and tombstone deletions. This allows crawlers, event streams, and external applications to publish explicit document removal signals (action: "DELETE") that purge document records and vector embeddings across target search engines and vector databases.

---

📑 Specification & JSON Schema

The document payload contract defines an optional action property defaulting to "UPSERT".

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "OpenIngestionDocument",
  "type": "object",
  "required": ["id", "source"],
  "properties": {
    "id": {
      "type": "string",
      "description": "Unique document identifier in target index"
    },
    "action": {
      "type": "string",
      "enum": ["UPSERT", "DELETE"],
      "default": "UPSERT",
      "description": "Document lifecycle action. UPSERT creates or updates the document; DELETE purges the document and its chunks."
    },
    "source": {
      "type": "object",
      "description": "Metadata describing the origin repository"
    },
    "content": {
      "type": "object",
      "description": "Binary or text content (optional for DELETE tombstones)"
    },
    "metadata": {
      "type": "object",
      "description": "Metadata key-value pairs (optional for DELETE tombstones)"
    },
    "security": {
      "type": "object",
      "description": "Access control list (ACL) configuration (optional for DELETE tombstones)"
    }
  }
}

1. Standard Ingestion Payload (UPSERT)

{
  "id": "doc-101",
  "action": "UPSERT",
  "source": {
    "type": "filesystem",
    "instance": "local"
  },
  "content": {
    "text": "OpenCrawling is an enterprise open-source document crawling pipeline."
  },
  "metadata": {
    "name": "architecture.pdf",
    "mimeType": "application/pdf"
  }
}

2. Tombstone Deletion Payload (DELETE)

For tombstone deletions, content, metadata, and security payloads are optional. Only id, action: "DELETE", and source are required.

{
  "id": "doc-101",
  "action": "DELETE",
  "source": {
    "type": "filesystem",
    "instance": "local"
  }
}

---

🛞 End-to-End Pipeline Event Flow

When a tombstone DELETE action is published to OpenCrawling:

[Crawler / API] ──(Tombstone: action=DELETE)──> [JobOrchestrator]
                                                     │
                                           (Bypass Claim-Check)
                                                     │
                                                     ▼
                                          [Kafka: opencrawling-ingestion]
                                                     │
                                                     ▼
                                          [IngestionConsumer]
                                                     │
                                         (Short-Circuit Tika/Chunking)
                                                     │
                                                     ▼
                                           [Kafka: opencrawling-chunks]
                                                     │
                                                     ▼
                                          [EmbeddingConsumer]
                                                     │
                                         (Short-Circuit AI Embeddings)
                                                     │
                                                     ▼
                                          [Kafka: opencrawling-embedded]
                                                     │
                                                     ▼
                                          [Output Store Writer]
                                                     │
                                         (Purge Document / Chunks)
                                                     │
                                                     ▼
                                [Target Vector Database / Search Engine]

1. JobOrchestrator: Detects DocumentAction.DELETE and bypasses binary stream persistence to the Claim Check Store (Ozone / S3 / Local Storage).
2. IngestionConsumer: Short-circuits Apache Tika parsing and chunking, forwarding a tombstone DocumentChunkMessage directly to opencrawling-chunks.
3. EmbeddingConsumer: Short-circuits AI model inference (Ollama / OpenAI), forwarding a tombstone DocumentEmbeddedMessage directly to opencrawling-embedded.
4. Output Connectors: Vector store writers process DocumentAction.DELETE and execute native deletion queries to purge documents and vector points.

---

🔌 Target Store Deletion Behaviors

Output Connector	Target Store	Deletion Execution Method
oc-vector-output-connector	PostgreSQL (pgvector)	vectorStore.delete(List.of(documentId)) across all dimension tables (vector_store, vector_store_384, vector_store_768, vector_store_1024)
oc-solr-output-connector	Apache Solr	solrClient.deleteByQuery(collection, "id:\"docId\" OR chunk_id:\"chunkId\"")
oc-qdrant-output-connector	Qdrant	client.deleteAsync(collection, List.of(id(pointUuid)))
oc-milvus-output-connector	Milvus	client.delete(DeleteReq.builder().collectionName(col).filter("id == 'chunkId'").build())
oc-vespa-output-connector	Vespa	client.remove(DocumentId.of(namespace, documentType, chunkId))
oc-opensearch2-output-connector	OpenSearch 2.x	client.delete(d -> d.index(indexName).id(chunkId))
oc-opensearch3-output-connector	OpenSearch 3.x	client.delete(d -> d.index(indexName).id(chunkId))

---

☕ Java Client SDK Support (oc-java-client-sdk)

The Java SDK provides strongly-typed models for OIS document payloads and tombstone creation:

Creating a Tombstone Payload in Java

import org.opencrawling.sdk.models.DocumentAction;
import org.opencrawling.sdk.models.DocumentPayload;
import java.util.Map;

// Construct an OIS Tombstone Deletion Payload
DocumentPayload tombstone = DocumentPayload.createDeleteTombstone(
    "doc-101",
    Map.of("type", "filesystem", "instance", "prod-cluster")
);

System.out.println("Payload Action: " + tombstone.action()); // DELETE

---

🛠️ CLI Schema Validation (oc-cli)

Validate tombstone payloads offline using the OpenCrawling CLI:

oc schema validate --file tombstone-payload.json

Output:

✔ OIS Schema Validation Passed: Valid OIS document payload (Action: DELETE)

---

🌐 Ingesting OIS Payloads & Tombstones via HTTP REST

External producers (such as Apache StormCrawler, custom web scrapers, or CDC pipelines) can submit OIS documents and deletion tombstones directly to the runtime HTTP gateway (POST /api/v1/ingest/ois/{jobId}):

1. Ingest an UPSERT Document

curl -X POST http://localhost:8080/api/v1/ingest/ois/1 \
  -H "Authorization: Bearer ${OPENCRAWLING_INGEST_OIS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "https://example.com/page.html",
    "action": "UPSERT",
    "content": {
      "mimeType": "text/html",
      "text": "Extracted text content of the page."
    },
    "metadata": {
      "title": "Page Title"
    },
    "security": {
      "permissions": [
        {"identity": "ROLE_USER", "identityType": "role", "access": "read"}
      ]
    }
  }'

2. Issue a DELETE Tombstone

curl -X POST http://localhost:8080/api/v1/ingest/ois/1 \
  -H "Authorization: Bearer ${OPENCRAWLING_INGEST_OIS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "https://example.com/page.html",
    "action": "DELETE"
  }'

HTTP Response Status Codes

• 202 Accepted: Document stored in claim check and message acknowledged on Kafka.
• 400 Bad Request: Invalid OIS JSON structure or missing required fields (id, content.text or security for UPSERT).
• 401 Unauthorized: Missing or invalid Bearer token.
• 404 Not Found: Endpoint disabled (empty token) or target job ID does not exist.
• 409 Conflict: Target job is in MIGRATION mode (OIS HTTP ingestion supports RAG jobs only).
• 503 Service Unavailable: Storage or Kafka publisher failure.