EPIC: Drop support for old vectordb SDKs

[wjones127]

We have old vectordb SDKs in Rust and Javascript. We would like to drop support in favor of the newer lancedb SDKs, so we don't have to maintain two different SDKs.

Part 1: Feature parity

What features do we need in newer SDKs to have parity with old ones?

• Feature: port python's embedding funtions to Rust #994
• Feature: expose versioning in LanceDB Node / Rust #1007
• Add support for "Remote" connections #1362
• Migrate LanceDB Cloud integration tests to use lancedb
• Add dynamodb integration test
  • See: https://github.com/lancedb/lancedb/blob/main/node/src/integration_test/test.ts
• Middleware?
  • See: https://github.com/lancedb/lancedb/blob/main/node/src/middleware.ts
  • And: here
• Table.add() should return number of rows written #1340
• feat(nodejs): mergeInsert #1349
  • Was added to vectordb in https://github.com/lancedb/lancedb/pull/915/files
• nodejs: Table.indexStats() in @lancedb #1359
• Validate lancedb works in Electron app
• Feature(nodejs): Table.search function for lancedb sdk #1256

Low priority:

• Feature: allow nodejs API to specify read/write params #1040
• Support restore(version) in Node

Part 1.5 Fix important bugs

• bug(node): list of number input causes panic #758
• bug(node): Several bugs when trying to setup a simple use case #882
• bug(node): unable to 'add' to a table created from python #1289
• Feature(nodejs): Table.toArrow function #1254

Part 2: Docs migration

Transition period needs to have both vectordb and lancedb are active. To make transition easier, can have two tabs for all examples: Typescript lancedb and Typescript vectordb (legacy).

• Blog post: why we are doing this transition, how to transition, benefits to expect
  • Need this early so we can have earlier conversations with partners and integrations
  • Send that / create an issue in downstream repos
• Add Typescript lancedb examples to all docs sections
• Update https://lancedb.github.io/lancedb/migration/
  • Update Features not yet supported
  • Make clearer which packages we are talking about (lancedb vs vectordb in Node and Rust, async vs sync in Python)
• Update installation instructions for node to point to lancedb
  • Timeline for support: Support package for 2 months, keep docs for 4 months
  • Need to confirm timeline works for all parties involved (customers, integration partners)
• (Fill in missing features here)

Part 3: Cleanup

What do we need to do to wrap up the old SDKs

• (After 2 months) Delete node directory from codebase. This will end the release process for that package.
• Migrate downstream integrations?
  • TODO: get a list of integrations

Part 4: Cleanup docs

Remove the documentation for the vectordb package.

• (After 4 months) Remove mentions of vectordb package from docs. Feature: update docs to use async APIs #1044

[westonpace]

I had started a somewhat similarly-intentioned list here: #996 However, I think they are slightly different. I added #994 as I think that is the main feature gap between vectordb and lancedb today. Almost all of the examples use embedding functions in some way and so I struggled to switch over the docs and eventually abandoned it for now.