Skip to content

v1.1.0

Choose a tag to compare

@georgeh0 georgeh0 released this 01 Nov 18:15
952e3df

What's Changed

Connector key fixes: no more collisions or lost writes

Every source item and target state has a key. CocoIndex uses the key to match an object with its previous run, and to decide what to create, update or delete. Prompted by the Google Drive fix in #2485, this release audits how every Python connector builds its keys. It fixes each case where the connector gave two different objects one key, or where a key that changed for the same object made CocoIndex delete that object. Two keys you choose can also reach one object, such as overlapping directories or two ContextKeys for one database. CocoIndex can't detect that, so it stays your responsibility; the docs now point out where it can happen.

Two different objects shared one key, so one overwrote or deleted the other:

  • Google Drive: files with the same name, even in one folder. Items are now keyed by file ID by @Sujit-1509 in #2485
  • S3, Azure Blob and OCI: object keys that differ only in a doubled or leading /, a ./ segment, or \ versus /; and, with prefix="docs", docs/a.md and docsa.md. Paths in different buckets also compared equal by @georgeh0 in #2601
  • Kafka: the same message key on two topics of one topic_as_map feed. A tombstone on one topic deleted the other topic's outputs by @georgeh0 in #2597
  • Neo4j and FalkorDB: a node label and a relationship type with the same name, and derived edge ids of different endpoints (1 vs "1", ids containing _). The fix comes as new node and relationship target APIs (see "When you upgrade") by @georgeh0 in #2612
  • SurrealDB: derived relation IDs of different endpoints, e.g. doc:'chunk_1' and doc_chunk:1 linked to the same tag by @georgeh0 in #2599
  • zvec: 1 and "1" in an Any, object or union key column became one document by @georgeh0 in #2608 and #2613
  • Postgres: two vector indexes on one table whose names PostgreSQL truncates to the same 63 bytes, so only one of them could exist by @georgeh0 in #2603 and #2615
  • Valkey: a payload key or FieldDef named vector, the hash field that holds the document's vector by @georgeh0 in #2606

One object's key changed between runs, and the delete for the old key removed what the new key had just written:

  • localfs: a case-only rename (on macOS and Windows), or ./a.txt changed to a.txt. The file stayed missing until its content changed by @georgeh0 in #2598
  • Qdrant: a UUID point ID written in another text form (upper case, no hyphens, URN) by @georgeh0 in #2602
  • Kafka target: a message key that switched between str and bytes. Its tombstone erased the record on a compacted topic by @georgeh0 in #2597
  • Neo4j and FalkorDB: a renamed vector index by @georgeh0 in #2611

A delete missed its object, leaving stale data behind:

  • LanceDB: every row delete failed for a key column that needs quoting (doc-id), or that holds binary, fixed-size binary or the int64 minimum by @georgeh0 in #2600
  • Doris: keys containing \. Each update appended a duplicate row, removed items stayed, and a trailing \ broke the statement by @georgeh0 in #2605
  • Neo4j and FalkorDB: an edge with an explicit id whose endpoints changed. Neo4j kept the old edge beside the new one, and FalkorDB failed with unique constraint violation by @georgeh0 in #2611
  • Neo4j and FalkorDB: dropping a node table (un-declaring it, or cocoindex drop) left its vector indexes by @georgeh0 in #2618

A change went undetected:

  • S3, Azure Blob, OCI and Google Drive binary files: an overwrite that kept the modification time (S3 times are whole seconds). An equal time is now checked against the ETag or MD5 by @georgeh0 in #2601
  • Doris: a changed vector or inverted index parameter (index_type, metric_type, dimension, parser, ...). It now rebuilds the table by @georgeh0 in #2605

Docs:

  • The docs explain that a target state's identity is its key, and where two keys can reach one object: target states, contexts, mount_each(), and Postgres, SQLite and BigQuery tables. Mounting one component path twice in one run was already an error; its message now says how to fix it by @georgeh0 in #2610 and #2609

When you upgrade:

  • Deprecated, still working (with a DeprecationWarning, and with the same keys as before):
    • Neo4j and FalkorDB: table_target, declare_table_target, mount_table_target, relation_target, declare_relation_target, mount_relation_target and RelationTarget.declare_relation. Use node_target, declare_node_target, mount_node_target (they return a NodeTarget), relationship_target, declare_relationship_target, mount_relationship_target and RelationshipTarget.declare_relationship, with the same parameters. Switching an existing table takes two updates, described in the connector docs under "Switching an existing table"; skipping the first one deletes the table's nodes or edges. On the new APIs, Neo4j labels and relationship types can't contain __ or end with _, and derived edge ids need int or str endpoint ids (#2612)
    • Kafka: topic_as_map(consumer, [...]) with a list of topics. With one topic, pass it as a string; the keys don't change. With several, use one feed and one consumer per topic, each under its own subpath. That changes the item keys, so follow the replay steps in the Kafka docs (#2597)
    • Postgres: TableTarget(provider, table_schema). Pass the table name: TableTarget(provider, table_name, table_schema) (#2603, #2615)
  • Behavior change: for S3, Azure Blob and OCI, prefix names a folder. prefix="docs" lists docs/..., but no longer docsa.md or docs-old/x, and target states from those objects are removed on the next update. To select objects by how their names start, use path_matcher (#2601)
  • New errors, for declarations that could never work correctly: DirTarget names that are absolute or leave the directory (#2598); an Any, object, union or bytes key column in a new zvec collection (#2608); LanceDB key column types that deletes can't filter on, such as dates, timestamps, float16 and lists (#2600); Postgres vector indexes whose truncated names collide (#2603); the Valkey field name vector (#2606). FilePaths of different types no longer compare equal, and ordering them raises TypeError (#2601)
  • One-time effects on the first update: Google Drive components get new keys and are reprocessed (#2485); objects that shared a key become separate items (#2601); SurrealDB relations declared without an explicit id are re-created under a new record ID (#2599); DirTarget files declared under a name that now normalizes (such as ./a.txt) are written once under the new name (#2598). To remove duplicate graph edges left by earlier versions, run one update with --full-reprocess (#2611)

Other changes

  • feat(fn): coco.LogicDep and coco.use_logic_dep() let a call declare a logic dependency for itself only, such as the version of the per-language extractor it used. Bumping that version re-runs only the calls that used it, and their memoized callers. Also fixed: a function-memo hit lost the logic deps its entry recorded, so a caller re-stored on that hit missed later changes to deeper callees by @georgeh0 in #2590
  • feat(batching): @coco.fn.as_async(batching=True, ...) takes max_in_flight, which bounds how many batches run at once, and rate_limiter, which charges one token per batch instead of per call. For a limiter kept on the instance, pass a callable that receives self by @georgeh0 in #2588 and #2589
  • feat(engine): live components share the max_inflight_components limit fairly, so one live component's large update (e.g. a full re-index) no longer holds back another's small refresh. An update that runs alone still gets the whole pool by @georgeh0 in #2624
  • perf(engine): moving many target states from one component to another in one commit is no longer quadratic (5,000 targets: 243 s before, 1 to 2 s now) by @georgeh0 in #2621
  • fix(memo): memo keys and fingerprints depend only on values. A structure that references one object twice fingerprints the same as one holding equal copies, and a temporary object freed during fingerprinting can no longer cause a wrong memo hit when Python reuses its id. One-time effect: for values with shared sub-objects, fingerprints change. Memoized work may run once more, generate_id / generate_uuid / IdGenerator / UuidGenerator may return new IDs (which changes row keys built from them), and rows tracked by fingerprint_object are written again by @hatemadi15 in #2411
  • fix(gpu_pool): a GPU released to a cancelled acquire goes back to the pool. Before, every later acquire of that GPU hung by @georgeh0 in #2595
  • fix(runtime): coco.stop() / coco.stop_blocking() release the lifespan-provided resources that registered apps held, instead of keeping them until each app's next update by @hatemadi15 in #2425
  • fix(environment): opening a db_path that is still open in the process raises a RuntimeError that names the path and the way out (Environment.close()), instead of a misleading message about "different options" by @Siddesh-bype in #2561
  • fix(connectorkits): cancelling a pending read of a synchronous source listing (localfs, Google Drive) no longer hangs asyncio.run() at shutdown by @Sujit-1509 in #2619
  • fix(serde): VectorSchema(dtype=np.float32) (the class, not a dtype) now reloads on the second run. Other classes the state reader can't load now fail when first written, with a hint to use coco.unpickle_safe. Before, the error came on the next run by @rtmalikian in #2536
  • fix(connectors): the repr of Snowflake and Doris connection configs leaves out the password by @jerryqu-coco in #2620
  • fix(oci_object_storage): zero-length reads, and partial reads of empty objects, return empty bytes instead of failing with HTTP 416 by @pranav440 in #2466
  • fix(code_ast): C# file-scoped namespaces (namespace X;) now show in the render_match context frames of the declarations after them by @georgeh0 in #2629
  • Docs fixes: examples that failed when run as written
    • docs(ops): the SeparatorSplitter usage example now runs as written by @March-7 in #2476
    • docs(localfs): the lifespan example uses coco.EnvironmentBuilder; coco.EnvBuilder doesn't exist by @Siddesh-bype in #2397
    • docs: the get_context example no longer calls the async coco.default_env() as if it were synchronous, and the App guide's component path tree matches the subpaths mount() derives by @hardness1020 in #2416

New Contributors

Full Changelog: v1.0.26...v1.1.0