docs: Updates and refactor

[prrao87]

This PR makes incremental changes to the documentation.

• Closes Docs: UI updates and fixes #697
• Closes Docs: Homepage and intro #698

Chores

• Add dark mode
• Fix headers in navbar
• Add extra.css to customize navbar styles
• Customize fonts for prose/code blocks, navbar and admonitions
• Inspect all admonition boxes (remove redundant dropdowns) and improve clarity and readability
• Ensure that all images in the docs have white background (not transparent) to be viewable in dark mode
• Improve code formatting in code blocks to make them consistent with autoformatters (eslint/ruff)
• Add bolder weight to h1 headers
• Add diagram showing the difference between embedded (OSS) and serverless (Cloud)
• Fix Creating an empty table section: right now, the subheaders are not clickable.
• In critical data ingestion methods like table.add (among others), the type signature often does not match the actual code
• Proof-read each documentation section and rewrite as necessary to provide more context, use cases, and explanations so it reads less like reference documentation. This is especially important for CRUD and search sections since those are so central to the user experience.

Restructure/new content

• The section for Adding data only shows examples for pandas and iterables. We should include pydantic models, arrow tables, etc.
• Add conceptual tutorial for IVF-PQ index
• Clearly separate vector search, FTS and filtering sections so that these are easier to find
• Add docs on refine factor to explain its importance for recall. Closes Add docs on refine_factor #716
• Add an FAQ page showing answers to commonly asked questions about LanceDB. Closes docs: Add FAQ section #746
• Add simple polars example to the integrations section. Closes Docs: Add polars integration example #756 and closes Add Polars integration examples #153
• Add basic docs for the Rust API (more detailed API docs can come later). Closes docs: Add basic docs for the Rust API #781
• Add a section on the various storage options on local vs. cloud (S3, EBS, EFS, local disk, etc.) and the tradeoffs involved. Closes docs: Add a section on the various storage options on local vs. cloud #782
• Revamp filtering docs: add pre-filtering examples and redo headers and update content for SQL filters. Closes docs: Add pre-filtering examples #783 and closes docs: Add SQL filtering section #784.
• Add docs for data management: compaction, cleaning up old versions and incremental indexing. Closes docs: Data management: compaction, cleaning up old versions and incremental indexing #785
• Add a benchmark section that also discusses some best practices. Closes docs: Add a section on best practices/benchmark results #787

[prrao87]

The docs from this branch are ready for review. I've left out the benchmarks and the Rust docs in the tasks, but I think most of the other things we discussed are in there.

My thinking is that once you think it's ready, we get this merged in main @changhiskhan and @AyushExel, so that we can take stock of what can be fixed once we see it live (we can always make smaller PRs to fix any outstanding issues).

[AyushExel]

Agree. I think we should merge this to main. Smaller improvements can come later on a case by case basis

[prrao87]

Awesome! I was trying to fix the broken bits for CI, let's see how this goes, fix them and merge when finished 🙏🏽

@AyushExel do you know how to address that circular imports issue for polars? it's from the recent PR from Chang, but he seems to not have that issue on main.

[AyushExel]

Awesome! I was trying to fix the broken bits for CI, let's see how this goes, fix them and merge when finished 🙏🏽

@AyushExel do you know how to address that circular imports issue for polars? it's from the recent PR from Chang, but he seems to not have that issue on main.

Yeah the circular import issue seems weird not happening even in the branch.. just occurs during docs test

[AyushExel]

hmm..failing even after excluding the polars tests..Can investigate tomorrow

[wjones127]

you can scale compute up and down as needed depending on the size of your data and its associated index

This doesn't seem quite right. IIUC users don't set the scale; we automatically scale for optimal performance. Right @QianZhu ?

[QianZhu]

Correct. Users don't set the scale. The elastic scale is taken care by us.

[prrao87]

How would you frame it using that snippet @QianZhu ?

[prrao87]

How about this?

Because it's serverless, you only pay for the storage you use, and we take care of scaling compute up and down as needed depending on the size of your data and its associated index.

[prrao87]

Will clean up the other suggestion too, thanks, Will!

[prrao87]

@AyushExel, Python CI issues fixed. there's another issue with the node CI - I can't find anything wrong in the code that might be the source of the issue. Can you check when you have a moment?