Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/docs/ops/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ description: CocoIndex Built-in Sources

The `LocalFile` source imports files from a local file system.

### Spec

The spec takes the following fields:
* `path` (type: `str`, required): full path of the root directory to import files from
* `binary` (type: `bool`, optional): whether reading files as binary (instead of text)
Expand All @@ -24,7 +26,45 @@ The spec takes the following fields:

:::

### Schema

The output is a table with the following sub fields:
* `filename` (key, type: `str`): the filename of the file, including the path, relative to the root directory, e.g. `"dir1/file1.md"`
* `content` (type: `str` if `binary` is `False`, otherwise `bytes`): the content of the file

## GoogleDrive

The `GoogleDrive` source imports files from Google Drive.

### Setup for Google Drive

To access files in Google Drive, the `GoogleDrive` source will need to authenticate by service accounts.

1. Register / login in **Google Cloud**.
2. In [**Google Cloud Console**](https://console.cloud.google.com/), search for *Service Accounts*, to enter the *IAM & Admin / Service Accounts* page.
- **Create a new service account**: Click *+ Create Service Account*. Follow the instructions to finish service account creation.
- **Add a key and download the credential**: Under "Actions" for this new service account, click *Manage keys* → *Add key* → *Create new key* → *JSON*.
Download the key file to a safe place.
3. In **Google Cloud Console**, search for *Google Drive API*. Enable this API.
4. In **Google Drive**, share the folders containing files that need to be imported through your source with the service account's email address.
**Viewer permission** is sufficient.
- The email address can be found under the *IAM & Admin / Service Accounts* page (in Step 2), in the format of `{service-account-id}@{gcp-project-id}.iam.gserviceaccount.com`.
- Copy the folder ID. Folder ID can be found from the last part of the folder's URL, e.g. `https://drive.google.com/drive/u/0/folders/{folder-id}` or `https://drive.google.com/drive/folders/{folder-id}?usp=drive_link`.


### Spec

The spec takes the following fields:

* `service_account_credential_path` (type: `str`, required): full path to the service account credential file in JSON format.
* `root_folder_ids` (type: `list[str]`, required): a list of Google Drive folder IDs to import files from.
* `binary` (type: `bool`, optional): whether reading files as binary (instead of text).

### Schema

The output is a table with the following sub fields:

* `file_id` (key, type: `str`): the ID of the file in Google Drive.
* `filename` (type: `str`): the filename of the file, without the path, e.g. `"file1.md"`
* `mime_type` (type: `str`): the MIME type of the file.
* `content` (type: `str` if `binary` is `False`, otherwise `bytes`): the content of the file.