LanceDB Async API epic

[changhiskhan]

March 25th
finish up what's already planned

March 18th

Goal: have all new SDKs available by EOW

• switch for Rust - target Mon

• switch for Typescript - target Wed - maybe

• pull the switch for Python APIs (stop documentation on sync APIs), target Fri - maybe

• Publish lancedb rust crate (keep vectordb rust crate for a little while)

• Publish lancedb npm package with new nodejs api (keep vectordb npm package for a little while)

• Move LanceDB python onto lancedb rust crate via pyo3 Feature: make Python SDK based on Rust SDK #849

• Add async API for both local and remote in python SDK Feature: Implement Async Embedding Creation for Enhanced Performance #830

• Feature: Python vectordb-based lance POC #997

• bug(node): Several bugs when trying to setup a simple use case #882 (related QoL issue for TypeScript changing arrow to peer dependency)
  ETA: PR to put in review by the end of next Tu, Mar 5

Cleanup tasks (mostly stuff deferred from larger PRs to avoid losing momentum due to smaller things)

• Feature: fully add support for table_names paging #1030
• Feature: Allow add/create table to accept data from an asynchronous stream #1026
• Feature: hide native table under feature in lancedb #1027
• Feature: allow nodejs API to specify read/write params #1040
• Feature: allow specifying cache size when opening a table in python/node SDK #1041
• Flesh out remote impl
• Feature: update docs to use async APIs #1044

Nice to haves:

• Feature: Use C data interface instead of IPC for marshalling arrow data from JS to Rust #1060

[skyfallsin]

excited to see this in the queue, asyncio support in Python would go far!

[changhiskhan]

Release plan:

TENTATIVE:

1. Monday 3/18 - lancedb rust crate should be released.
2. Friday 3/22 TypeScript - lancedb npm package to be released
3. Friday 3/22 Python - async API to be released