v1.2.0

What's Changed

Discord source connector

cocoindex.connectors.discord indexes the messages and threads of Discord text, announcement and forum channels, using the token of a bot you create and invite to your server (pip install cocoindex[discord]): DiscordChannel(app=DiscordApp(token=...), channel_id=...).

• DiscordChannel.items() yields one DiscordMessage per message in the channel and one DiscordThread per thread, active or archived, including forum posts. Both have read_text() for embedding
• Each run lists the channel, and only the messages and threads that changed are reprocessed; a thread's messages are fetched only when the thread reruns. message_fields and thread_fields choose which fields are exposed, and so what counts as a change
• Misconfigurations that Discord answers with an empty or text-less success (the Message Content intent off, no Read Message History) and lost access raise an error, so the channel keeps its outputs instead of having them deleted
• Two examples: semantic search over a channel (discord_text_embedding), and support threads turned into an FAQ knowledge base (discord_support_faq)

by @jerryqu-coco in #2695

Google Drive: sign in with a personal Google account (experimental)

Beside a service account, the Google Drive connector can read a Drive as a personal Google account signed in once through OAuth: GoogleDriveSource(root_folder_ids=[...], user_account=google_drive.UserAccount()). Sign in with cocoindex auth sign-in google-drive <client_secret.json> (sign out with cocoindex auth sign-out google-drive), or call UserAccount().sign_in(...) from your own program. The client file comes from a Desktop app OAuth client you create in your own Google Cloud project; the connector docs walk through it. A host with no browser prints the link and takes the pasted redirect address. The sign-in asks for read-only Drive access and is stored only on your computer; the connector docs describe the token file and how to revoke it. The service-account path is unchanged by @georgeh0 in #2687, #2689, #2791 and #2792

Data dependencies for memoized functions and components (experimental)

• coco.DataDep and coco.use_data_dep() let a memoized function or component record data it read that isn't among its arguments, such as files an agent chose to open. Its memo is reused only while every recorded read is still valid. Being experimental, they are outside the compatibility promise: a later release, a minor one included, may change or remove them by @georgeh0 in #2662, #2668 and #2674
• A memoized ancestor of a component that recorded data dependencies is reused when those dependencies all still validate, instead of re-running every time by @georgeh0 in #2671
• New example, CocoIndex Agents: an incremental agent harness built on data dependencies. Agent sessions are memoized on what they read, and it reads local folders, PDFs and a Google Drive folder as the signed-in account by @georgeh0 in #2659, #2686 and #2688

A calmer, clearer progress display

cocoindex update, cocoindex drop and update_blocking(report_to_stdout=True) show one status line (Running…, ● Watching for changes, ✓ Done in 3.0s), with each component in an aligned column showing a progress bar, its rate and its failures. The summary lists the first failures under their component, and the final status counts failures inside stats groups too. Every failure also logs one ERROR line, with its traceback in scrolling output by @badmonster0 in #2717 and #2731, and @georgeh0 in #2744

Other changes

• feat(engine): declaring the same thing twice in one run (a target state key, a component path such as a duplicate mount_each key, a use_state key, a ContextKey, an app name) raises cocoindex.AlreadyDeclaredError. It subclasses ValueError, so existing except ValueError handlers keep working by @georgeh0 in #2672
• perf(core): a run that changes nothing in a component no longer writes to the state store, and the submit and memo paths decode and allocate less. Warm runs of many unchanged components are faster, most of all on disks with slow fsync by @badmonster0 in #2784 and #2785
• fix(core): in a live component, a delete issued while the same item's update was still running left that update's target states behind; the delete now wins. A process_live that fails before it is ready now fails its parent with its own error, not "Child component build cancelled" by @badmonster0 in #2779
• fix(core): a component applies its target states parent-first, so an attachment (such as a vector index) is no longer created before the table it belongs to. LanceDB sometimes refused to build an index on a first run because of this by @badmonster0 in #2769
• fix(zvec): a component declaring more than 1024 documents no longer fails every run, and deleting a document that is already missing no longer fails by @badmonster0 in #2771
• fix(splitter): RecursiveSplitter no longer drops text that comes after the end of a grammar's root node (e.g. after a stray byte order mark in YAML) by @badmonster0 in #2786
• fix(batching): a queued batch whose callers have all gone away (a timeout, a cancelled update) is no longer run, saving a wasted model request or GPU batch by @badmonster0 in #2787
• fix(ratelimit): RateLimiter raises ValueError for a NaN or too-small rate and a non-finite burst_window_secs, instead of crashing by @badmonster0 in #2764
• fix(litellm): an import error inside litellm (such as a missing optional dependency) fails at once, instead of being retried silently for ten minutes as a connection error by @georgeh0 in #2685

This release also includes the fixes of v1.1.1.

Full Changelog: v1.1.0...v1.2.0