Repository navigation
Google Drive Repository Connector
Module: oc-googledrive-repository-connector
Connector Class: org.opencrawling.googledrive.GoogleDriveRepositoryConnector
Client Class: org.opencrawling.googledrive.GoogleDriveClient
Document Builder: org.opencrawling.googledrive.GoogleDriveDocumentBuilder
Security Mapper: org.opencrawling.googledrive.GoogleDriveSecurityMapper
Properties Class: org.opencrawling.googledrive.GoogleDriveProperties
Crawl State: org.opencrawling.googledrive.GoogleDriveCrawlState & org.opencrawling.runtime.connector.ControlPlaneGoogleDriveStateStore
Discovery & Single-Doc SPI: Implements DocumentFetchingRepositoryConnector (DiscoverableRepositoryConnector: explore, dryRun, crawlStart, fetchDocument)
Spring Bean: @Component (activated when spring.opencrawling.connector.type=googledrive or SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=googledrive)
Introduced in: Issue #339, PR #436
The Google Drive Repository Connector (oc-googledrive-repository-connector) crawls Google Drive (My Drive) and Google Workspace Shared Drives directly through the official Google Drive REST API v3 (https://www.googleapis.com/drive/v3) using native Java 25 java.net.http.HttpClient and pure java.security PKCS#8 RSA RS256 JWT signing—with zero external Google SDK dependencies.
It streams binary files and exported native Google Workspace documents into OpenCrawling as Open Ingestion Standard (OIS) documents with user-facing webViewLink URLs, original Drive titles, folder hierarchy breadcrumbs, composite version keys, custom properties, incremental Changes API state stored in the shared Control Plane database (oc_setting), periodic full reconciliation with an empty-listing guard, explicit TOO_LARGE ledger reporting, pre-crawl discovery, and fail-closed Zero-Trust Google Workspace ACLs.
Important
Least-Privilege Access (No Domain-Wide Delegation Required): The connector uses the strictly read-only https://www.googleapis.com/auth/drive.readonly scope by default. With a Google Cloud Service Account, simply share the target folder(s) or Shared Drive directly with the Service Account email (Viewer role). Domain-Wide Delegation (delegated-user / JWT sub claim) is completely optional and never required.
graph TD
subgraph GWorkspace["Google Cloud & Google Workspace"]
OAuth["Google OAuth 2.0 (Service Account RS256 JWT / Refresh Token)"]
DriveApi["Google Drive REST API v3 (/about, /drives, /files, /changes, /permissions)"]
NativeDocs["Native Workspace Files (/export: Docs -> HTML, Slides -> text)"]
Binaries["Binary Files (?alt=media: PDF, DOCX, TXT, ...)"]
OAuth -->|Bearer Token| DriveApi
NativeDocs --> DriveApi
Binaries --> DriveApi
end
subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
Client["GoogleDriveClient (Pure Java RS256 Signer, Token Cache, 429/5xx Backoff)"]
Connector["GoogleDriveRepositoryConnector (Changes API, Tree & Discovery SPI)"]
StateStore["ControlPlaneGoogleDriveStateStore (Shared ControlPlaneStore / oc_setting)"]
VThreads["Virtual Threads (Bounded Parallelism)"]
DocBld["GoogleDriveDocumentBuilder (webViewLink, Breadcrumbs, versionKey, TOO_LARGE)"]
SecMap["GoogleDriveSecurityMapper (Fail-Closed User, Group, Domain, Opt-In Anyone)"]
TempStream["TempFileInputStream (Constant-Heap Spool & Auto-Delete)"]
Orchestrator["JobOrchestrator (TOO_LARGE -> SKIPPED Ledger + Trace WARN)"]
ClaimCheck["Claim-Check Store (Local / Apache Ozone / S3)"]
Tika["Apache Tika 4 Text Extraction"]
DriveApi <-->|REST JSON + Stream| Client
Client --> Connector
Connector <--> StateStore
Connector --> VThreads
VThreads --> DocBld
VThreads --> SecMap
VThreads --> TempStream
DocBld --> Orchestrator
SecMap --> Orchestrator
TempStream --> ClaimCheck
ClaimCheck --> Tika
OIS["OIS Document (UPSERT / DELETE Tombstone)"]
Orchestrator --> OIS
Tika --> OIS
OIS --> Kafka["Kafka Pipeline (opencrawling-ingestion -> chunks -> embedded)"]
Kafka --> Sinks[("Vector & Lakehouse Stores (PGVector, Vespa, Qdrant, Milvus, Solr, OpenSearch, Elasticsearch, Iceberg, ...)")]
end
subgraph AI["AI Applications & Zero-Trust MCP"]
Sinks --> SecureMcp["OpenCrawling Secure MCP Server (McpAclEvaluator & DiscoveryMcpTools)"]
SecureMcp -->|Zero-Trust User, Group & Domain ACL Enforcement| Agent["Enterprise RAG Assistant / Agent"]
end
-
Google Cloud Project with Google Drive API enabled:
-
Service Account (recommended): Create a Service Account in Google Cloud IAM, generate a JSON key, and either paste the inline JSON into
service-account-json(automatically masked as********by the Platform API and Admin UI) or mount a key file path. -
Share target scope with the Service Account: Grant the Service Account email Viewer access to the target Shared Drive (
drive-id) or explicit folder(s) (root-folder-ids). No Google Workspace admin Domain-Wide Delegation is required. -
Optional Domain-Wide Delegation: Only if you want to impersonate a Google Workspace user, authorize the Service Account client ID for
https://www.googleapis.com/auth/drive.readonlyin Google Workspace Admin Console and setdelegated-user. -
Alternative OAuth 2.0 Refresh Token: Set
auth-type: oauth2withclient-id,client-secret, andrefresh-token(orauth-type: bearerwith a short-livedaccess-tokenfor testing).
-
Service Account (recommended): Create a Service Account in Google Cloud IAM, generate a JSON key, and either paste the inline JSON into
-
Network Access: Outbound HTTPS to
https://oauth2.googleapis.comandhttps://www.googleapis.com(both overridable viatoken-urlandapi-base-urlfor local testing against the included Drive v3 mock). -
Connection Verification: Click Test Connection in the Admin UI or run
oc connector check --name <name> --type repository. The checker probesGET /drive/v3/about, verifiesGET /drive/v3/drives/{driveId}when configured, and checks reachability of every configuredroot-folder-idsentry.
-
Flexible Scope & Shared Drives:
- Crawl an entire My Drive (
rootautomatically resolved viaGET /drive/v3/files/rootto its canonical folder ID), a Shared Drive (drive-id), or one or more explicit folder IDs (root-folder-id/root-folder-ids). - Control subfolder traversal via
include-subfolders(trueby default), Shared Drive visibility viainclude-shared-drives(trueby default, sendingsupportsAllDrives=true&includeItemsFromAllDrives=true), and shortcut resolution viafollow-shortcuts(falseby default).
- Crawl an entire My Drive (
-
Native Google Workspace Exports & Default MIME Exclusions:
- Exports Google Docs as HTML (
text/html) by default so document headings survive cleanly for structure-aware chunking (#385). - Exports Google Slides as plain text (
text/plain) by default. - Configurable export targets for Google Sheets (
xlsx) and Google Drawings (pdf). - Excludes Google Sheets (
application/vnd.google-apps.spreadsheet) and Google Forms (application/vnd.google-apps.form) by default viaexcluded-mime-typesso tabular grids and form definitions do not pollute RAG indices (set to""or"none"to clear exclusions).
- Exports Google Docs as HTML (
-
Constant-Heap Streaming & Explicit
TOO_LARGEReporting (#397):- Streams binary downloads (
?alt=media) and workspace exports (/export) directly to disk viaTempFileInputStream(constant heap regardless of file size, auto-deleted on close). - Files exceeding
max-file-size-bytes(default52428800= 50 MB) or Google Drive's 10 MB/exportcap (exportSizeLimitExceeded) are never silently dropped—the connector emits a metadata-only sentinel (opencrawling.document.status=TOO_LARGE), andJobOrchestratorrecords aSKIPPEDentry (errorCode="TOO_LARGE") in the execution ledger (DocumentLedgerEntry.skipped(...)) and aWARNevent in the trace store.
- Streams binary downloads (
-
User-Facing Metadata & Composite Version Keys:
- Emits canonical internal URIs (
googledrive://drives/<driveId>/files/<fileId>) while surfacingwebViewLinkasurl/webViewLink/googledrive.webViewLink, the original Drive file name astitle, the parent folder hierarchy asbreadcrumbsandpath, and Driveproperties/appPropertiesasgoogledrive_prop_<key>. - Computes a composite
googledrive.versionKeyfrom Drive'sversion,headRevisionId, andmd5Checksum(falling back tomodifiedTime) so unchanged files are skipped without downloading bytes.
- Emits canonical internal URIs (
-
Incremental Changes API, Multi-Replica Control Plane State (
#395) & Periodic Full Reconciliation (#396):-
changesmode (default): Persists the Changes APIstartPageTokenand known item/folder snapshots in OpenCrawling's sharedControlPlaneStore(ControlPlaneGoogleDriveStateStore, keyconnector-state/googledrive-<sha256>.jsonstored inoc_settingunder JDBC mode), allowing scheduled runs to execute on any runtime replica without losing or diverging state. - Handles creations, edits, renames, deletions, trashed files, moves out of scope, cascading folder tombstones (trashing, deleting, or moving a folder out of scope tombstones all known descendant files), and automatic fallback to full reconciliation on expired
pageToken(400/410). - Supports scheduled periodic full reconciliation (
full-scan-interval,full-scan-window,full-scan-timezone) guarded by an empty-listing guard: if a full enumeration returns0files while files are already indexed, the run logs a warning and emits zero tombstones to prevent accidental index wipes during transient outages or permission revocations.
-
-
Pre-Crawl Discovery & Single-Document Fetching:
- Implements
DiscoverableRepositoryConnector(explore,dryRun,crawlStart) andDocumentFetchingRepositoryConnector(fetchDocument) with page-by-pageVisibilityFilterenforcement against live Drive permissions, enabling AI agents and Dynamic Jobs to inspect folders and fetch individual Drive files on demand without leaking hidden folders.
- Implements
Configure via Spring Boot properties (spring.opencrawling.connector.type=googledrive and spring.opencrawling.connector.googledrive.*) or via the Admin UI / Platform API / connectors.json (org.opencrawling.googledrive.GoogleDriveRepositoryConnector, camelCase keys).
Spring Property (spring.opencrawling.connector.googledrive.*) |
connectors.json Key |
Default | Description |
|---|---|---|---|
api-base-url |
apiBaseUrl |
https://www.googleapis.com/drive/v3 |
Google Drive API v3 base URL (overridable for local mock testing) |
token-url |
tokenUrl |
https://oauth2.googleapis.com/token |
OAuth 2.0 token endpoint URL |
auth-type |
authType |
service_account |
service_account, oauth2, or bearer
|
service-account-json |
serviceAccountJson |
— | Inline Service Account JSON or path to key file (masked in API/UI) |
service-account-key-file |
serviceAccountKeyFile |
— | Alternative filesystem path to Service Account JSON key file |
delegated-user |
delegatedUser |
— | Optional Google Workspace user email for Domain-Wide Delegation (sub claim) |
scope |
scope |
https://www.googleapis.com/auth/drive.readonly |
OAuth 2.0 scope requested for Service Account JWTs |
client-id, client-secret, refresh-token
|
clientId, clientSecret, refreshToken
|
— | OAuth 2.0 credentials when auth-type: oauth2
|
access-token |
accessToken |
— | Static Bearer access token when auth-type: bearer (testing) |
drive-id |
driveId |
— | Shared Drive ID (empty for My Drive) |
root-folder-id |
rootFolderId |
root |
Single starting folder ID (root resolves to My Drive root ID) |
root-folder-ids |
rootFolderIds |
— | Comma-separated folder IDs scoping the crawl |
include-subfolders |
includeSubfolders |
true |
Descend recursively into subfolders |
include-shared-drives |
includeSharedDrives |
true |
Send supportsAllDrives=true and includeItemsFromAllDrives=true
|
follow-shortcuts |
followShortcuts |
false |
Resolve application/vnd.google-apps.shortcut targets |
crawl-mode |
crawlMode |
changes |
changes (Changes API incremental) or tree (full folder traversal) |
state-file |
stateFile |
— | Optional local JSON state path override (leave blank to use shared ControlPlaneStore) |
full-scan-interval |
fullScanInterval |
(empty; P7D in UI default) |
ISO-8601 duration for periodic full reconciliation inside a changes job |
full-scan-window |
fullScanWindow |
— | Optional 24h maintenance window (01:00-05:00, may wrap midnight) |
full-scan-timezone |
fullScanTimezone |
UTC |
IANA timezone for full-scan-window
|
export-google-docs |
exportGoogleDocs |
true |
Export native Google Workspace files via /export
|
export-doc-mime-type |
exportDocMimeType |
text/html |
Export MIME type for Google Docs |
export-slides-mime-type |
exportSlidesMimeType |
text/plain |
Export MIME type for Google Slides |
export-sheet-mime-type |
exportSheetMimeType |
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet |
Export MIME type for Google Sheets (when not excluded) |
export-drawing-mime-type |
exportDrawingMimeType |
application/pdf |
Export MIME type for Google Drawings |
excluded-mime-types |
excludedMimeTypes |
application/vnd.google-apps.spreadsheet,application/vnd.google-apps.form |
Comma-separated MIME types to skip ("" or "none" to exclude nothing) |
included-mime-types |
includedMimeTypes |
— | Optional allow-list of MIME types |
max-file-size-bytes |
maxFileSizeBytes |
52428800 |
Maximum content size in bytes (50 MB); larger files are reported as TOO_LARGE
|
include-custom-properties |
includeCustomProperties |
true |
Emit Drive properties / appProperties as googledrive_prop_<key>
|
include-acls |
includeAcls |
true |
Fetch /permissions and enforce Zero-Trust OIS ACLs |
allow-anyone-public |
allowAnyonePublic |
false |
Whether type=anyone permissions map to public read access |
domain-identity-prefix |
domainIdentityPrefix |
domain: |
Prefix for type=domain group principals (e.g., domain:example.com) |
page-size |
pageSize |
100 |
Drive API page size (1–1000) |
parallelism |
parallelism |
4 |
Concurrent files processed via Java 25 Virtual Threads |
max-retries |
maxRetries |
5 |
Retries on HTTP 429, Drive 403 rate-limit errors, 5xx, and transient I/O errors |
initial-backoff-ms |
initialBackoffMs |
1000 |
Base exponential backoff delay in milliseconds |
timeout-seconds |
timeoutSeconds |
30 |
HTTP connect and request timeout in seconds |
spring:
opencrawling:
repository-connector-type: googledrive
connector:
type: googledrive
googledrive:
auth-type: "service_account"
service-account-json: "${GOOGLEDRIVE_SERVICE_ACCOUNT_JSON}"
delegated-user: "" # optional; leave empty when sharing folders/Shared Drives directly with the SA
drive-id: "0AO_SUPPORT"
root-folder-ids: "folder-procedures,folder-runbooks"
include-subfolders: true
include-shared-drives: true
crawl-mode: "changes"
full-scan-interval: "P7D"
full-scan-window: "01:00-05:00"
full-scan-timezone: "UTC"
export-google-docs: true
export-doc-mime-type: "text/html"
export-slides-mime-type: "text/plain"
excluded-mime-types: "application/vnd.google-apps.spreadsheet,application/vnd.google-apps.form"
max-file-size-bytes: 52428800
include-custom-properties: true
include-acls: true
allow-anyone-public: false
page-size: 100
parallelism: 4Each file uses the canonical URI googledrive://drives/<driveId>/files/<fileId> (or googledrive://drives/my-drive/files/<fileId> for My Drive):
{
"id": "googledrive://drives/0AO_SUPPORT/files/doc-reset",
"action": "UPSERT",
"acl": "alice@example.com,support@example.com",
"metadata": {
"source_type": ["googledrive"],
"title": ["Password Reset Procedure"],
"name": ["Password Reset Procedure.html"],
"filename": ["Password Reset Procedure.html"],
"mimeType": ["text/html"],
"url": ["https://docs.google.com/document/d/doc-reset/edit"],
"webViewLink": ["https://docs.google.com/document/d/doc-reset/edit"],
"googledrive.webViewLink": ["https://docs.google.com/document/d/doc-reset/edit"],
"path": ["/Support Knowledge Base/Procedures/Password Reset Procedure.html"],
"googledrive.path": ["/Support Knowledge Base/Procedures/Password Reset Procedure.html"],
"googledrive.folderPath": ["/Support Knowledge Base/Procedures"],
"breadcrumbs": ["Support Knowledge Base", "Procedures"],
"googledrive.breadcrumbs": ["Support Knowledge Base > Procedures"],
"googledrive.fileId": ["doc-reset"],
"googledrive.originalName": ["Password Reset Procedure"],
"googledrive.originalMimeType": ["application/vnd.google-apps.document"],
"googledrive.exported": ["true"],
"googledrive.driveId": ["0AO_SUPPORT"],
"googledrive.driveName": ["Support Knowledge Base"],
"googledrive.version": ["14"],
"googledrive.versionKey": ["v:14"],
"googledrive.owners": ["alice@example.com"],
"googledrive.lastModifyingUser": ["bob@example.com"],
"googledrive_prop_department": ["Support"],
"googledrive_prop_classification": ["Internal"]
},
"security": {
"inheritanceEnabled": false,
"permissions": [
{ "identity": "alice@example.com", "identityType": "user", "displayName": "Alice", "access": "write" },
{ "identity": "support@example.com", "identityType": "group", "displayName": "Support", "access": "read" }
]
}
}Emitted when a file is deleted, trashed, moved outside the configured root scope, or when a parent folder is trashed/deleted/moved out of scope (cascading tombstones to every known descendant):
{
"id": "googledrive://drives/0AO_SUPPORT/files/doc-archived-note",
"action": "DELETE",
"metadata": {
"source_type": ["googledrive"],
"googledrive.fileId": ["doc-archived-note"],
"googledrive.driveId": ["0AO_SUPPORT"],
"googledrive.status": ["DELETED"]
}
}When a binary exceeds max-file-size-bytes or Google Drive rejects /export with exportSizeLimitExceeded (> 10 MB), GoogleDriveDocumentBuilder.buildTooLarge(...) sets opencrawling.document.status=TOO_LARGE and opencrawling.document.reason=<detail>. JobOrchestrator intercepts this sentinel before Claim-Check upload and records DocumentLedgerEntry.skipped(..., "TOO_LARGE", reason) in the job audit manifest plus a WARN trace entry so operators have full visibility into skipped files.
For each file (include-acls: true), GoogleDriveSecurityMapper reads GET /drive/v3/files/{fileId}/permissions and maps Drive permissions into OIS PermissionRules on a strict fail-closed basis. Deleted permissions (deleted: true) and expired permissions (expirationTime <= now) are ignored. Roles owner, organizer, fileOrganizer, and writer map to write; commenter and reader map to read.
| Drive Permission Type | Principal Emitted (identity) |
identityType |
Notes |
|---|---|---|---|
user |
Lower-case emailAddress (e.g., alice@example.com) |
user |
Higher access wins (write overrides read on duplicate principals) |
group |
Lower-case group emailAddress (e.g., support@example.com) |
group |
Matches Google Groups / IdP group claims in user tokens |
domain |
<domainIdentityPrefix><domain> (default domain:example.com) |
group |
Configurable via domain-identity-prefix
|
anyone |
public (only when allow-anyone-public: true) |
public |
Ignored by default (allow-anyone-public: false, fail-closed) |
| Empty / Unreadable Permissions | (None — SecurityConfig.none()) |
— |
Fail-closed (permissions: [], acl: ""); never defaults to public
|
{
"name": "secureVectorSearch",
"arguments": {
"query": "password reset MFA hardware recovery",
"userPrincipal": "alice@example.com",
"userRoles": "support@example.com,domain:example.com",
"maxResults": 10
}
}-
Shared Control Plane State (
ControlPlaneGoogleDriveStateStore): Whenstate-fileis left empty (the default), the connector persistsGoogleDriveCrawlState.State(startPageToken,lastFullScanAt, known fileversionKey& SHA-256permissionsFingerprint, and folder parent maps) inside OpenCrawling'sControlPlaneStore(oc_settingunderopencrawling.control-plane.store=jdbc). Any runtime replica picking up the next scheduled job resumes from the exact Changes API token. -
Permission-Only Reconciliation (
full-scan-intervalorcrawl-mode: tree): Because Google Drive Changes API events may not fire for every inherited or silent permission update, configuringfull-scan-interval(e.g.,P7D, optionally constrained byfull-scan-windowandfull-scan-timezone) or runningcrawl-mode: treere-enumerates the scope, compares each file'sversionKeyand SHA-256permissionsFingerprint, skips unchanged files without downloading content, and re-emits only files whose content or permissions changed. -
Empty-Listing Guard (
#396): If a full enumeration returns0files while files are already present in saved state, the connector logs a warning and preserves existing state instead of emitting massDELETEtombstones.
| Operation | Endpoint |
|---|---|
| Acquire Access Token |
POST https://oauth2.googleapis.com/token (urn:ietf:params:oauth:grant-type:jwt-bearer or refresh_token) |
| Validate Identity / Connection | GET /drive/v3/about?fields=user,storageQuota |
| List / Verify Shared Drives |
GET /drive/v3/drives, GET /drive/v3/drives/{driveId}
|
| Resolve Root / Folder / File Metadata | GET /drive/v3/files/{fileId}?supportsAllDrives=true&fields=... |
| List Folder Children | GET /drive/v3/files?q='folderId'+in+parents+and+trashed=false&supportsAllDrives=true&includeItemsFromAllDrives=true |
| Get Changes Start Token | GET /drive/v3/changes/startPageToken?supportsAllDrives=true[&driveId=...] |
| List Incremental Changes | GET /drive/v3/changes?pageToken={token}&supportsAllDrives=true&includeItemsFromAllDrives=true[&driveId=...] |
| List Item Permissions | GET /drive/v3/files/{fileId}/permissions?supportsAllDrives=true |
| Export Google Workspace File | GET /drive/v3/files/{fileId}/export?mimeType={targetMime} |
| Download Binary File | GET /drive/v3/files/{fileId}?alt=media&supportsAllDrives=true |
- Open Connectors → New Connector and select Google Drive (Google Drive API v3) (
org.opencrawling.googledrive.GoogleDriveRepositoryConnector). - Choose an authentication mode (Service Account JSON, OAuth 2.0 Refresh Token, or Bearer Token), paste the Service Account JSON (automatically masked on save/read), and optionally specify a
Drive ID(Shared Drive) and/orRoot Folder IDs. - Configure export MIME types, excluded MIME types (Google Sheets & Forms excluded by default),
maxFileSizeBytes, periodic full scan schedule, and Zero-Trust ACL options. - Click Test Connection to validate credentials, Shared Drive access, and configured root folders.
# Validate Google Drive credentials, Shared Drive, and root folder accessibility
oc connector check --name GoogleDrive_Repository --type repositorymvn test -pl oc-googledrive-repository-connector -amRuns 21 unit and embedded HTTP mock tests (GoogleDrivePropertiesTest, GoogleDriveSecurityMapperTest, GoogleDriveClientTest, GoogleDriveRepositoryConnectorTest) covering RS256 Service Account JWT generation, OAuth2 refresh, 429/5xx backoff, HTML/plain-text exports, default Sheets/Forms exclusion, TOO_LARGE reporting (size cap + Drive 10 MB exportSizeLimitExceeded), incremental Changes API sync, cascading folder tombstones, expired pageToken fallback, empty-listing guard, fail-closed ACLs, and Discovery/DocumentFetching SPIs.
./scripts/test-googledrive-connector.shStarts the stdlib Python 3 Google Drive REST API v3 & OAuth2 emulator (oc-googledrive-repository-connector/googledrive-mock/googledrive_mock.py), validates HTTP 401 rejection, OAuth2 /token, /drive/v3/about, /drive/v3/drives, /drive/v3/changes/startPageToken, HTML export, and PDF binary streaming, then executes all 21 connector tests and 128 oc-runtime wiring tests (ConnectorKindTest, ConnectorFactoryTest, ConnectorCheckerServiceTest, ControlPlaneGoogleDriveStateStoreTest with two independent JDBC replicas sharing state, PlatformApiV1UnitTest secret masking for serviceAccountJson, and PlatformApiV1UiParityTest). Also wired into .github/workflows/integration-tests.yml as "Google Drive Connector Test Suite".
./scripts/test-googledrive-decoupled.sh # Production Dockerfiles
./scripts/test-googledrive-decoupled.sh --local # Fast local BuildKit ~/.m2 cache overlay
./scripts/test-googledrive-decoupled.sh -s -k # Skip build and keep containers for inspectionBoots oc-googledrive-repository-connector/docker/docker-compose-decoupled-with-googledrive.yml and verifies 9 end-to-end steps:
- Initial
oc-crawler(changes) crawl intopgvector(vector_store_1024): exact live count (6docs covering Google Doc HTML export, Google Slides plain text, valid Tika-extracted PDF, sub-folder text, restricted text, andanyone-link doc), Sheets & Forms excluded (0rows), 100 MB oversized PDF reported as1 too_large(0rows,0 failed). - Rich metadata (
title,url/googledrive.webViewLink,breadcrumbs,versionKey,googledrive_prop_*), fail-closed Zero-Trust ACLs (user,group,domain:example.com, closedanyone), and shared JDBC Control PlanestartPageTokenpersistence in PostgreSQLoc_setting. - Secure MCP Server (
secureVectorSearch,getDocumentContent,listAccessibleSources) enforcement across Alice, Support group, Domain member, Bob, and Outsider. - Live Drive mutations + incremental Changes API sync (edit Google Doc, create new Google Doc, delete Slides deck, trash parent folder triggering cascading descendant tombstone, and
startPageTokenadvancement inoc_setting). - Permission-only update (without Change event) left untouched by
changesand reconciled cleanly by atreefull scan reusing shared JDBC state.
-
Google Drive 10 MB Native Export Limit: Google Drive's
/files/{fileId}/exportendpoint caps exported native Workspace files at 10 MB (exportSizeLimitExceeded). Such files are explicitly reported asTOO_LARGE(SKIPPED) in the execution ledger rather than failing the job. -
Google Groups Membership Expansion: Drive
/permissionsreturns group email addresses (e.g.,support@example.com) and domain names (e.g.,domain:example.com), not individual group members. Callers querying via MCP pass their Google Group email addresses anddomain:<domain>claims inuserRoles. - Comments & Revision History: File comments and historical revisions are not indexed as separate documents; only the current head revision is exported/downloaded.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite