Skip to content

Google Drive Repository Connector

Piergiorgio Lucidi edited this page Nov 21, 2026 · 1 revision

Google Drive Repository Connector

Google Drive Java Spring Boot OIS Compliance Claim Check

Module: oc-googledrive-repository-connector
Connector Class: org.opencrawling.googledrive.GoogleDriveRepositoryConnector
Client Class: org.opencrawling.googledrive.GoogleDriveClient
Document Builder: org.opencrawling.googledrive.GoogleDriveDocumentBuilder
Security Mapper: org.opencrawling.googledrive.GoogleDriveSecurityMapper
Properties Class: org.opencrawling.googledrive.GoogleDriveProperties
Crawl State: org.opencrawling.googledrive.GoogleDriveCrawlState & org.opencrawling.runtime.connector.ControlPlaneGoogleDriveStateStore
Discovery & Single-Doc SPI: Implements DocumentFetchingRepositoryConnector (DiscoverableRepositoryConnector: explore, dryRun, crawlStart, fetchDocument)
Spring Bean: @Component (activated when spring.opencrawling.connector.type=googledrive or SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=googledrive)
Introduced in: Issue #339, PR #436


📌 Overview

The Google Drive Repository Connector (oc-googledrive-repository-connector) crawls Google Drive (My Drive) and Google Workspace Shared Drives directly through the official Google Drive REST API v3 (https://www.googleapis.com/drive/v3) using native Java 25 java.net.http.HttpClient and pure java.security PKCS#8 RSA RS256 JWT signing—with zero external Google SDK dependencies.

It streams binary files and exported native Google Workspace documents into OpenCrawling as Open Ingestion Standard (OIS) documents with user-facing webViewLink URLs, original Drive titles, folder hierarchy breadcrumbs, composite version keys, custom properties, incremental Changes API state stored in the shared Control Plane database (oc_setting), periodic full reconciliation with an empty-listing guard, explicit TOO_LARGE ledger reporting, pre-crawl discovery, and fail-closed Zero-Trust Google Workspace ACLs.

Important

Least-Privilege Access (No Domain-Wide Delegation Required): The connector uses the strictly read-only https://www.googleapis.com/auth/drive.readonly scope by default. With a Google Cloud Service Account, simply share the target folder(s) or Shared Drive directly with the Service Account email (Viewer role). Domain-Wide Delegation (delegated-user / JWT sub claim) is completely optional and never required.


🏗️ Architecture & Flow

graph TD
    subgraph GWorkspace["Google Cloud & Google Workspace"]
        OAuth["Google OAuth 2.0 (Service Account RS256 JWT / Refresh Token)"]
        DriveApi["Google Drive REST API v3 (/about, /drives, /files, /changes, /permissions)"]
        NativeDocs["Native Workspace Files (/export: Docs -> HTML, Slides -> text)"]
        Binaries["Binary Files (?alt=media: PDF, DOCX, TXT, ...)"]

        OAuth -->|Bearer Token| DriveApi
        NativeDocs --> DriveApi
        Binaries --> DriveApi
    end

    subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
        Client["GoogleDriveClient (Pure Java RS256 Signer, Token Cache, 429/5xx Backoff)"]
        Connector["GoogleDriveRepositoryConnector (Changes API, Tree & Discovery SPI)"]
        StateStore["ControlPlaneGoogleDriveStateStore (Shared ControlPlaneStore / oc_setting)"]
        VThreads["Virtual Threads (Bounded Parallelism)"]
        DocBld["GoogleDriveDocumentBuilder (webViewLink, Breadcrumbs, versionKey, TOO_LARGE)"]
        SecMap["GoogleDriveSecurityMapper (Fail-Closed User, Group, Domain, Opt-In Anyone)"]
        TempStream["TempFileInputStream (Constant-Heap Spool & Auto-Delete)"]
        Orchestrator["JobOrchestrator (TOO_LARGE -> SKIPPED Ledger + Trace WARN)"]
        ClaimCheck["Claim-Check Store (Local / Apache Ozone / S3)"]
        Tika["Apache Tika 4 Text Extraction"]

        DriveApi <-->|REST JSON + Stream| Client
        Client --> Connector
        Connector <--> StateStore
        Connector --> VThreads
        VThreads --> DocBld
        VThreads --> SecMap
        VThreads --> TempStream
        DocBld --> Orchestrator
        SecMap --> Orchestrator
        TempStream --> ClaimCheck
        ClaimCheck --> Tika

        OIS["OIS Document (UPSERT / DELETE Tombstone)"]
        Orchestrator --> OIS
        Tika --> OIS

        OIS --> Kafka["Kafka Pipeline (opencrawling-ingestion -> chunks -> embedded)"]
        Kafka --> Sinks[("Vector & Lakehouse Stores (PGVector, Vespa, Qdrant, Milvus, Solr, OpenSearch, Elasticsearch, Iceberg, ...)")]
    end

    subgraph AI["AI Applications & Zero-Trust MCP"]
        Sinks --> SecureMcp["OpenCrawling Secure MCP Server (McpAclEvaluator & DiscoveryMcpTools)"]
        SecureMcp -->|Zero-Trust User, Group & Domain ACL Enforcement| Agent["Enterprise RAG Assistant / Agent"]
    end
Loading

✅ Prerequisites

  1. Google Cloud Project with Google Drive API enabled:
    • Service Account (recommended): Create a Service Account in Google Cloud IAM, generate a JSON key, and either paste the inline JSON into service-account-json (automatically masked as ******** by the Platform API and Admin UI) or mount a key file path.
    • Share target scope with the Service Account: Grant the Service Account email Viewer access to the target Shared Drive (drive-id) or explicit folder(s) (root-folder-ids). No Google Workspace admin Domain-Wide Delegation is required.
    • Optional Domain-Wide Delegation: Only if you want to impersonate a Google Workspace user, authorize the Service Account client ID for https://www.googleapis.com/auth/drive.readonly in Google Workspace Admin Console and set delegated-user.
    • Alternative OAuth 2.0 Refresh Token: Set auth-type: oauth2 with client-id, client-secret, and refresh-token (or auth-type: bearer with a short-lived access-token for testing).
  2. Network Access: Outbound HTTPS to https://oauth2.googleapis.com and https://www.googleapis.com (both overridable via token-url and api-base-url for local testing against the included Drive v3 mock).
  3. Connection Verification: Click Test Connection in the Admin UI or run oc connector check --name <name> --type repository. The checker probes GET /drive/v3/about, verifies GET /drive/v3/drives/{driveId} when configured, and checks reachability of every configured root-folder-ids entry.

🚀 Key Features

  • Flexible Scope & Shared Drives:
    • Crawl an entire My Drive (root automatically resolved via GET /drive/v3/files/root to its canonical folder ID), a Shared Drive (drive-id), or one or more explicit folder IDs (root-folder-id / root-folder-ids).
    • Control subfolder traversal via include-subfolders (true by default), Shared Drive visibility via include-shared-drives (true by default, sending supportsAllDrives=true & includeItemsFromAllDrives=true), and shortcut resolution via follow-shortcuts (false by default).
  • Native Google Workspace Exports & Default MIME Exclusions:
    • Exports Google Docs as HTML (text/html) by default so document headings survive cleanly for structure-aware chunking (#385).
    • Exports Google Slides as plain text (text/plain) by default.
    • Configurable export targets for Google Sheets (xlsx) and Google Drawings (pdf).
    • Excludes Google Sheets (application/vnd.google-apps.spreadsheet) and Google Forms (application/vnd.google-apps.form) by default via excluded-mime-types so tabular grids and form definitions do not pollute RAG indices (set to "" or "none" to clear exclusions).
  • Constant-Heap Streaming & Explicit TOO_LARGE Reporting (#397):
    • Streams binary downloads (?alt=media) and workspace exports (/export) directly to disk via TempFileInputStream (constant heap regardless of file size, auto-deleted on close).
    • Files exceeding max-file-size-bytes (default 52428800 = 50 MB) or Google Drive's 10 MB /export cap (exportSizeLimitExceeded) are never silently dropped—the connector emits a metadata-only sentinel (opencrawling.document.status=TOO_LARGE), and JobOrchestrator records a SKIPPED entry (errorCode="TOO_LARGE") in the execution ledger (DocumentLedgerEntry.skipped(...)) and a WARN event in the trace store.
  • User-Facing Metadata & Composite Version Keys:
    • Emits canonical internal URIs (googledrive://drives/<driveId>/files/<fileId>) while surfacing webViewLink as url / webViewLink / googledrive.webViewLink, the original Drive file name as title, the parent folder hierarchy as breadcrumbs and path, and Drive properties/appProperties as googledrive_prop_<key>.
    • Computes a composite googledrive.versionKey from Drive's version, headRevisionId, and md5Checksum (falling back to modifiedTime) so unchanged files are skipped without downloading bytes.
  • Incremental Changes API, Multi-Replica Control Plane State (#395) & Periodic Full Reconciliation (#396):
    • changes mode (default): Persists the Changes API startPageToken and known item/folder snapshots in OpenCrawling's shared ControlPlaneStore (ControlPlaneGoogleDriveStateStore, key connector-state/googledrive-<sha256>.json stored in oc_setting under JDBC mode), allowing scheduled runs to execute on any runtime replica without losing or diverging state.
    • Handles creations, edits, renames, deletions, trashed files, moves out of scope, cascading folder tombstones (trashing, deleting, or moving a folder out of scope tombstones all known descendant files), and automatic fallback to full reconciliation on expired pageToken (400/410).
    • Supports scheduled periodic full reconciliation (full-scan-interval, full-scan-window, full-scan-timezone) guarded by an empty-listing guard: if a full enumeration returns 0 files while files are already indexed, the run logs a warning and emits zero tombstones to prevent accidental index wipes during transient outages or permission revocations.
  • Pre-Crawl Discovery & Single-Document Fetching:
    • Implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument) with page-by-page VisibilityFilter enforcement against live Drive permissions, enabling AI agents and Dynamic Jobs to inspect folders and fetch individual Drive files on demand without leaking hidden folders.

⚙️ Configuration Properties

Configure via Spring Boot properties (spring.opencrawling.connector.type=googledrive and spring.opencrawling.connector.googledrive.*) or via the Admin UI / Platform API / connectors.json (org.opencrawling.googledrive.GoogleDriveRepositoryConnector, camelCase keys).

Spring Property (spring.opencrawling.connector.googledrive.*) connectors.json Key Default Description
api-base-url apiBaseUrl https://www.googleapis.com/drive/v3 Google Drive API v3 base URL (overridable for local mock testing)
token-url tokenUrl https://oauth2.googleapis.com/token OAuth 2.0 token endpoint URL
auth-type authType service_account service_account, oauth2, or bearer
service-account-json serviceAccountJson — Inline Service Account JSON or path to key file (masked in API/UI)
service-account-key-file serviceAccountKeyFile — Alternative filesystem path to Service Account JSON key file
delegated-user delegatedUser — Optional Google Workspace user email for Domain-Wide Delegation (sub claim)
scope scope https://www.googleapis.com/auth/drive.readonly OAuth 2.0 scope requested for Service Account JWTs
client-id, client-secret, refresh-token clientId, clientSecret, refreshToken — OAuth 2.0 credentials when auth-type: oauth2
access-token accessToken — Static Bearer access token when auth-type: bearer (testing)
drive-id driveId — Shared Drive ID (empty for My Drive)
root-folder-id rootFolderId root Single starting folder ID (root resolves to My Drive root ID)
root-folder-ids rootFolderIds — Comma-separated folder IDs scoping the crawl
include-subfolders includeSubfolders true Descend recursively into subfolders
include-shared-drives includeSharedDrives true Send supportsAllDrives=true and includeItemsFromAllDrives=true
follow-shortcuts followShortcuts false Resolve application/vnd.google-apps.shortcut targets
crawl-mode crawlMode changes changes (Changes API incremental) or tree (full folder traversal)
state-file stateFile — Optional local JSON state path override (leave blank to use shared ControlPlaneStore)
full-scan-interval fullScanInterval (empty; P7D in UI default) ISO-8601 duration for periodic full reconciliation inside a changes job
full-scan-window fullScanWindow — Optional 24h maintenance window (01:00-05:00, may wrap midnight)
full-scan-timezone fullScanTimezone UTC IANA timezone for full-scan-window
export-google-docs exportGoogleDocs true Export native Google Workspace files via /export
export-doc-mime-type exportDocMimeType text/html Export MIME type for Google Docs
export-slides-mime-type exportSlidesMimeType text/plain Export MIME type for Google Slides
export-sheet-mime-type exportSheetMimeType application/vnd.openxmlformats-officedocument.spreadsheetml.sheet Export MIME type for Google Sheets (when not excluded)
export-drawing-mime-type exportDrawingMimeType application/pdf Export MIME type for Google Drawings
excluded-mime-types excludedMimeTypes application/vnd.google-apps.spreadsheet,application/vnd.google-apps.form Comma-separated MIME types to skip ("" or "none" to exclude nothing)
included-mime-types includedMimeTypes — Optional allow-list of MIME types
max-file-size-bytes maxFileSizeBytes 52428800 Maximum content size in bytes (50 MB); larger files are reported as TOO_LARGE
include-custom-properties includeCustomProperties true Emit Drive properties / appProperties as googledrive_prop_<key>
include-acls includeAcls true Fetch /permissions and enforce Zero-Trust OIS ACLs
allow-anyone-public allowAnyonePublic false Whether type=anyone permissions map to public read access
domain-identity-prefix domainIdentityPrefix domain: Prefix for type=domain group principals (e.g., domain:example.com)
page-size pageSize 100 Drive API page size (1–1000)
parallelism parallelism 4 Concurrent files processed via Java 25 Virtual Threads
max-retries maxRetries 5 Retries on HTTP 429, Drive 403 rate-limit errors, 5xx, and transient I/O errors
initial-backoff-ms initialBackoffMs 1000 Base exponential backoff delay in milliseconds
timeout-seconds timeoutSeconds 30 HTTP connect and request timeout in seconds

📄 Example application.yml

spring:
  opencrawling:
    repository-connector-type: googledrive
    connector:
      type: googledrive
      googledrive:
        auth-type: "service_account"
        service-account-json: "${GOOGLEDRIVE_SERVICE_ACCOUNT_JSON}"
        delegated-user: "" # optional; leave empty when sharing folders/Shared Drives directly with the SA
        drive-id: "0AO_SUPPORT"
        root-folder-ids: "folder-procedures,folder-runbooks"
        include-subfolders: true
        include-shared-drives: true
        crawl-mode: "changes"
        full-scan-interval: "P7D"
        full-scan-window: "01:00-05:00"
        full-scan-timezone: "UTC"
        export-google-docs: true
        export-doc-mime-type: "text/html"
        export-slides-mime-type: "text/plain"
        excluded-mime-types: "application/vnd.google-apps.spreadsheet,application/vnd.google-apps.form"
        max-file-size-bytes: 52428800
        include-custom-properties: true
        include-acls: true
        allow-anyone-public: false
        page-size: 100
        parallelism: 4

📦 OIS Document Mapping & Payloads

1. File Document (action: "UPSERT")

Each file uses the canonical URI googledrive://drives/<driveId>/files/<fileId> (or googledrive://drives/my-drive/files/<fileId> for My Drive):

{
  "id": "googledrive://drives/0AO_SUPPORT/files/doc-reset",
  "action": "UPSERT",
  "acl": "alice@example.com,support@example.com",
  "metadata": {
    "source_type": ["googledrive"],
    "title": ["Password Reset Procedure"],
    "name": ["Password Reset Procedure.html"],
    "filename": ["Password Reset Procedure.html"],
    "mimeType": ["text/html"],
    "url": ["https://docs.google.com/document/d/doc-reset/edit"],
    "webViewLink": ["https://docs.google.com/document/d/doc-reset/edit"],
    "googledrive.webViewLink": ["https://docs.google.com/document/d/doc-reset/edit"],
    "path": ["/Support Knowledge Base/Procedures/Password Reset Procedure.html"],
    "googledrive.path": ["/Support Knowledge Base/Procedures/Password Reset Procedure.html"],
    "googledrive.folderPath": ["/Support Knowledge Base/Procedures"],
    "breadcrumbs": ["Support Knowledge Base", "Procedures"],
    "googledrive.breadcrumbs": ["Support Knowledge Base > Procedures"],
    "googledrive.fileId": ["doc-reset"],
    "googledrive.originalName": ["Password Reset Procedure"],
    "googledrive.originalMimeType": ["application/vnd.google-apps.document"],
    "googledrive.exported": ["true"],
    "googledrive.driveId": ["0AO_SUPPORT"],
    "googledrive.driveName": ["Support Knowledge Base"],
    "googledrive.version": ["14"],
    "googledrive.versionKey": ["v:14"],
    "googledrive.owners": ["alice@example.com"],
    "googledrive.lastModifyingUser": ["bob@example.com"],
    "googledrive_prop_department": ["Support"],
    "googledrive_prop_classification": ["Internal"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      { "identity": "alice@example.com", "identityType": "user", "displayName": "Alice", "access": "write" },
      { "identity": "support@example.com", "identityType": "group", "displayName": "Support", "access": "read" }
    ]
  }
}

2. Deletion Tombstone (action: "DELETE")

Emitted when a file is deleted, trashed, moved outside the configured root scope, or when a parent folder is trashed/deleted/moved out of scope (cascading tombstones to every known descendant):

{
  "id": "googledrive://drives/0AO_SUPPORT/files/doc-archived-note",
  "action": "DELETE",
  "metadata": {
    "source_type": ["googledrive"],
    "googledrive.fileId": ["doc-archived-note"],
    "googledrive.driveId": ["0AO_SUPPORT"],
    "googledrive.status": ["DELETED"]
  }
}

3. Oversized File Ledger Entry (TOO_LARGE)

When a binary exceeds max-file-size-bytes or Google Drive rejects /export with exportSizeLimitExceeded (> 10 MB), GoogleDriveDocumentBuilder.buildTooLarge(...) sets opencrawling.document.status=TOO_LARGE and opencrawling.document.reason=<detail>. JobOrchestrator intercepts this sentinel before Claim-Check upload and records DocumentLedgerEntry.skipped(..., "TOO_LARGE", reason) in the job audit manifest plus a WARN trace entry so operators have full visibility into skipped files.


🔒 Zero-Trust Security & ACL Mapping

For each file (include-acls: true), GoogleDriveSecurityMapper reads GET /drive/v3/files/{fileId}/permissions and maps Drive permissions into OIS PermissionRules on a strict fail-closed basis. Deleted permissions (deleted: true) and expired permissions (expirationTime <= now) are ignored. Roles owner, organizer, fileOrganizer, and writer map to write; commenter and reader map to read.

Drive Permission Type Principal Emitted (identity) identityType Notes
user Lower-case emailAddress (e.g., alice@example.com) user Higher access wins (write overrides read on duplicate principals)
group Lower-case group emailAddress (e.g., support@example.com) group Matches Google Groups / IdP group claims in user tokens
domain <domainIdentityPrefix><domain> (default domain:example.com) group Configurable via domain-identity-prefix
anyone public (only when allow-anyone-public: true) public Ignored by default (allow-anyone-public: false, fail-closed)
Empty / Unreadable Permissions (None — SecurityConfig.none()) — Fail-closed (permissions: [], acl: ""); never defaults to public

Querying with the Secure MCP Server (oc-mcp-server)

{
  "name": "secureVectorSearch",
  "arguments": {
    "query": "password reset MFA hardware recovery",
    "userPrincipal": "alice@example.com",
    "userRoles": "support@example.com,domain:example.com",
    "maxResults": 10
  }
}

🔄 Incremental Sync, Multi-Replica State & Reconciliation

  1. Shared Control Plane State (ControlPlaneGoogleDriveStateStore): When state-file is left empty (the default), the connector persists GoogleDriveCrawlState.State (startPageToken, lastFullScanAt, known file versionKey & SHA-256 permissionsFingerprint, and folder parent maps) inside OpenCrawling's ControlPlaneStore (oc_setting under opencrawling.control-plane.store=jdbc). Any runtime replica picking up the next scheduled job resumes from the exact Changes API token.
  2. Permission-Only Reconciliation (full-scan-interval or crawl-mode: tree): Because Google Drive Changes API events may not fire for every inherited or silent permission update, configuring full-scan-interval (e.g., P7D, optionally constrained by full-scan-window and full-scan-timezone) or running crawl-mode: tree re-enumerates the scope, compares each file's versionKey and SHA-256 permissionsFingerprint, skips unchanged files without downloading content, and re-emits only files whose content or permissions changed.
  3. Empty-Listing Guard (#396): If a full enumeration returns 0 files while files are already present in saved state, the connector logs a warning and preserves existing state instead of emitting mass DELETE tombstones.

🌐 Google Drive REST API v3 Endpoints Used

Operation Endpoint
Acquire Access Token POST https://oauth2.googleapis.com/token (urn:ietf:params:oauth:grant-type:jwt-bearer or refresh_token)
Validate Identity / Connection GET /drive/v3/about?fields=user,storageQuota
List / Verify Shared Drives GET /drive/v3/drives, GET /drive/v3/drives/{driveId}
Resolve Root / Folder / File Metadata GET /drive/v3/files/{fileId}?supportsAllDrives=true&fields=...
List Folder Children GET /drive/v3/files?q='folderId'+in+parents+and+trashed=false&supportsAllDrives=true&includeItemsFromAllDrives=true
Get Changes Start Token GET /drive/v3/changes/startPageToken?supportsAllDrives=true[&driveId=...]
List Incremental Changes GET /drive/v3/changes?pageToken={token}&supportsAllDrives=true&includeItemsFromAllDrives=true[&driveId=...]
List Item Permissions GET /drive/v3/files/{fileId}/permissions?supportsAllDrives=true
Export Google Workspace File GET /drive/v3/files/{fileId}/export?mimeType={targetMime}
Download Binary File GET /drive/v3/files/{fileId}?alt=media&supportsAllDrives=true

🖥️ Admin UI & CLI Integration

Admin UI (oc-admin-ui)

  1. Open Connectors → New Connector and select Google Drive (Google Drive API v3) (org.opencrawling.googledrive.GoogleDriveRepositoryConnector).
  2. Choose an authentication mode (Service Account JSON, OAuth 2.0 Refresh Token, or Bearer Token), paste the Service Account JSON (automatically masked on save/read), and optionally specify a Drive ID (Shared Drive) and/or Root Folder IDs.
  3. Configure export MIME types, excluded MIME types (Google Sheets & Forms excluded by default), maxFileSizeBytes, periodic full scan schedule, and Zero-Trust ACL options.
  4. Click Test Connection to validate credentials, Shared Drive access, and configured root folders.

OpenCrawling CLI (oc-cli)

# Validate Google Drive credentials, Shared Drive, and root folder accessibility
oc connector check --name GoogleDrive_Repository --type repository

🧪 Testing & Verification

1. Connector Unit & Embedded HTTP Mock Tests

mvn test -pl oc-googledrive-repository-connector -am

Runs 21 unit and embedded HTTP mock tests (GoogleDrivePropertiesTest, GoogleDriveSecurityMapperTest, GoogleDriveClientTest, GoogleDriveRepositoryConnectorTest) covering RS256 Service Account JWT generation, OAuth2 refresh, 429/5xx backoff, HTML/plain-text exports, default Sheets/Forms exclusion, TOO_LARGE reporting (size cap + Drive 10 MB exportSizeLimitExceeded), incremental Changes API sync, cascading folder tombstones, expired pageToken fallback, empty-listing guard, fail-closed ACLs, and Discovery/DocumentFetching SPIs.

2. Standalone Emulator, Contract & Runtime Wiring Suite

./scripts/test-googledrive-connector.sh

Starts the stdlib Python 3 Google Drive REST API v3 & OAuth2 emulator (oc-googledrive-repository-connector/googledrive-mock/googledrive_mock.py), validates HTTP 401 rejection, OAuth2 /token, /drive/v3/about, /drive/v3/drives, /drive/v3/changes/startPageToken, HTML export, and PDF binary streaming, then executes all 21 connector tests and 128 oc-runtime wiring tests (ConnectorKindTest, ConnectorFactoryTest, ConnectorCheckerServiceTest, ControlPlaneGoogleDriveStateStoreTest with two independent JDBC replicas sharing state, PlatformApiV1UnitTest secret masking for serviceAccountJson, and PlatformApiV1UiParityTest). Also wired into .github/workflows/integration-tests.yml as "Google Drive Connector Test Suite".

3. Full Decoupled Pipeline Integration Test (Kafka + Tika + Ollama + pgvector + MCP)

./scripts/test-googledrive-decoupled.sh          # Production Dockerfiles
./scripts/test-googledrive-decoupled.sh --local  # Fast local BuildKit ~/.m2 cache overlay
./scripts/test-googledrive-decoupled.sh -s -k    # Skip build and keep containers for inspection

Boots oc-googledrive-repository-connector/docker/docker-compose-decoupled-with-googledrive.yml and verifies 9 end-to-end steps:

  1. Initial oc-crawler (changes) crawl into pgvector (vector_store_1024): exact live count (6 docs covering Google Doc HTML export, Google Slides plain text, valid Tika-extracted PDF, sub-folder text, restricted text, and anyone-link doc), Sheets & Forms excluded (0 rows), 100 MB oversized PDF reported as 1 too_large (0 rows, 0 failed).
  2. Rich metadata (title, url / googledrive.webViewLink, breadcrumbs, versionKey, googledrive_prop_*), fail-closed Zero-Trust ACLs (user, group, domain:example.com, closed anyone), and shared JDBC Control Plane startPageToken persistence in PostgreSQL oc_setting.
  3. Secure MCP Server (secureVectorSearch, getDocumentContent, listAccessibleSources) enforcement across Alice, Support group, Domain member, Bob, and Outsider.
  4. Live Drive mutations + incremental Changes API sync (edit Google Doc, create new Google Doc, delete Slides deck, trash parent folder triggering cascading descendant tombstone, and startPageToken advancement in oc_setting).
  5. Permission-only update (without Change event) left untouched by changes and reconciled cleanly by a tree full scan reusing shared JDBC state.

⚠️ Known Limitations

  • Google Drive 10 MB Native Export Limit: Google Drive's /files/{fileId}/export endpoint caps exported native Workspace files at 10 MB (exportSizeLimitExceeded). Such files are explicitly reported as TOO_LARGE (SKIPPED) in the execution ledger rather than failing the job.
  • Google Groups Membership Expansion: Drive /permissions returns group email addresses (e.g., support@example.com) and domain names (e.g., domain:example.com), not individual group members. Callers querying via MCP pass their Google Group email addresses and domain:<domain> claims in userRoles.
  • Comments & Revision History: File comments and historical revisions are not indexed as separate documents; only the current head revision is exported/downloaded.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally