Merge remote-tracking branch 'origin/main' into 153-implement-ukv_opt…

…ion_scan_sample_k-for-ukv_scan

.devcontainer/devcontainer.json

@@ -21,7 +21,8 @@
 				"benjamin-simmonds.pythoncpp-debug",
 				"xaver.clang-format",
 				"mhutchie.git-graph",
-				"mutantdino.resourcemonitor"
+				"mutantdino.resourcemonitor",
+				"eamodio.gitlens"
 			]
 		}
 	},

.github/workflows/static.yml

@@ -8,7 +8,7 @@ on:
 
   # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
-
+ 
 # Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
 permissions:
   contents: read
@@ -33,16 +33,16 @@ jobs:
       - name: Setup Pages
         uses: actions/configure-pages@v2
       - name: Install Deps
-        run: sudo apt update && sudo apt install -y doxygen graphviz dia git
-      - name: Get Styles
-        run: git clone https://github.com/jothepro/doxygen-awesome-css && cd doxygen-awesome-css && git checkout v2.1.0
+        run: sudo apt update && sudo apt install -y doxygen graphviz dia git && pip install sphinx breathe furo sphinx_mdinclude
       - name: Build Documentation
-        run: doxygen doxygen_docs/main.mk
+        run: cd assets/docs && doxygen config.dox && make html
+      - name: Copy assets
+        run: cp -r assets build/docs/html/
       - name: Upload artifact
         uses: actions/upload-pages-artifact@v1
         with:
           # Upload entire repository
-          path: "./doxygen_docs/html/"
+          path: "./build/docs/html/"
       - name: Deploy to GitHub Pages
         id: deployment
         uses: actions/deploy-pages@v1

.vscode/tasks.json

@@ -85,6 +85,12 @@
             "command": "cmake -DCMAKE_BUILD_TYPE=Release . && make -j 6",
             "args": [],
             "type": "shell"
+        },
+        {
+            "label": "Count Lines of Code",
+            "command":"cloc $(git ls-files)",
+            "args": [],
+            "type": "shell"
         }
     ]
 }

README.md

@@ -1,84 +1,133 @@
-# Universal Keys & Values
-
-⚠️ Under active development! Not all APIs are stable!
-
-## The BLAS of CRUD
+<h1 align="center">UKV</h1>
+<h3 align="center">Universal Binary Interface</h3>
+<h3 align="center">For The Fastest DBMS Ever Built</h3>
+<h5 align="center">RocksDB • LevelDB • UnumDB • RAM</h5>
+<h5 align="center">Blobs • Documents • Graphs • Vectors</h5>
+<h5 align="center">C • C++ • Python • Java • GoLang</h5>
+<br/>
+
+<p align="center">
+<a href="https://twitter.com/unum_cloud"><img src="https://img.shields.io/badge/twitter-follow_us-1d9bf0.svg?color=purple&style=flat-square"></a>
+&nbsp;&nbsp;
+<a href="https://www.linkedin.com/company/unum-cloud/"><img src="https://img.shields.io/badge/linkedin-connect_with_us-0a66c2.svg?color=purple&style=flat-square"></a>
+&nbsp;&nbsp;
+<a href="https://www.github.com/unum-cloud/"><img src="https://img.shields.io/github/issues-closed-raw/unum-cloud/ukv?color=purple&style=flat-square"></a>
+&nbsp;&nbsp;
+<a href="https://www.github.com/unum-cloud/"><img src="https://img.shields.io/github/stars/unum-cloud/ukv?color=purple&style=flat-square"></a>
+</p>
+
+## What is UKV?
+
+UKV is an open C-layer binary standard for "Create, Read, Update, Delete" operations, or CRUD for short.
+Many databases exist today, providing similar functionality and performance under different interfaces.
+We approach the problem from a different angle.
+If databases do similar things, let's standardize the interface and compete for the best implementation.
+That way migrations are easier and the system can be modularized into different parts, to give users the absolute flexibility in choosing between different [Data Forms ~ Modalities](#modalities) and [Key-Value Store ~ Engines](#engines) implementations, as well as different Distribution forms and [Client SDKs](#frontends).
+
+## The [BLAS][blas] of [CRUD][crud]
+
+Such generic standard exists in computing since 1979.
+It is called [BLAS][blas] and was the locomotive of Numerical Methods across all disciplines in the past 50 years.
+Every deep-learning you use relies of BLAS.
+What is the standard that your DBMS can rely on?
 
 ![Universal Key Values by Unum](assets/UKV.png)
 
-Imagine having a standardized cross-lingual interface for all your things "Data":
+We haven't passed the test of time, like BLAS, but we can go beyond them in modularity and a great reference implementation.
+Today, Intel, Nvidia, AMD, GraphCore, Cerebras and many others ship optimized implementations of BLAS for their hardware, but people rarely use the open-source reference design.
+
+---
 
-* Storing binary blobs
-* Building up graphs & indexes
-* Querying structured documents
-* [ACID](https://en.wikipedia.org/wiki/ACID) transactions across tables, docs & graphs via one API!
-* [Apache Arrow](https://arrow.apache.org/) interop and [Flight RPC](https://arrow.apache.org/docs/format/Flight.html)
-* Familiar high-level [drivers](#frontends) for tabular & graph analytics
-* Handling JSON, [BSON](https://www.mongodb.com/json-and-bson), [MsgPack](https://msgpack.org/index.html)
-* [JSON-Pointers](https://datatracker.ietf.org/doc/html/rfc6901) & [Field-level Patches](https://datatracker.ietf.org/doc/html/rfc6902), no custom Query Languages
-* Packing Tensors for [PyTorch](https://pytorch.org/) and [TensorFlow](tensorflow.org)
+## Modularity
 
-UKV does just that, abstracting away the implementation from the user.
-In under 20K LOC you get a reference implementation in C++, support for any classical backend, and bindings for [Python](#python), [GoLang](#golang), [Java](#java).
-You can combine every [engine](#engines) with every modality, [frontend](#frontends) and distribution form:
+Our reference implementation aims to be faster and more scalable, than established DBs, like MongoDB, Neo4J, Redis, ETCD and Postgres.
+You can compose your own "X" DBMS killer from any combination of components.
 
-| Engine  | Modality | Distribution                    | Frontend                        |
-| :------ | :------- | :------------------------------ | :------------------------------ |
+| Engine  | Modality |          Distribution           |            Frontend             |
+| :-----: | :------: | :-----------------------------: | :-----------------------------: |
 |         |          |                                 |                                 |
-| RAM     | Blobs    | Embedded                        | C and C++                       |
-| LevelDB | Docs     | Standalone                      | Python                          |
-| RocksDB | Graphs   | Distributed <sup>*coming*</sup> | GoLang <sup>*in-progress*</sup> |
-| UnumKV  |          |                                 | Java <sup>*in-progress*</sup>   |
+|   RAM   |  Blobs   |            Embedded             |            C and C++            |
+| LevelDB |   Docs   |        Client-Server RPC        |             Python              |
+| RocksDB |  Graphs  |     Arrow Flight RPC Server     | GoLang <sup>*in-progress*</sup> |
+| UnumKV  | Vectors  | Distributed <sup>*coming*</sup> |  Java <sup>*in-progress*</sup>  |
+
+Run `cloc $(git ls-files)` and you will see, that UKV fits into just around 20'000 Lines fo Code.
+Compare this to bulky 1'491'985 LOC in Postgres and 4'466'967 LOC for MySQL. [*HN*][dbms-cloc].
+They target just the tabular workloads.
+After standardizing the API, we can make the system a lot reacher in features!
+
+<h5 align="center">ACID transactions across many collections • Snapshots  • Operation-level WATCHes</h4>
+<h5 align="center">BSON, JSON, MessagePack documents support • JSON Patches & Merrge-Patches • JSON Pointers Addressing</h4>
+<h5 align="center">Native Apache Arrow format support in all APIs • Arrow Flight RPC Server • Bulk Scans • Random Samping</h4>
+<h5 align="center">Pandas Tabular API • NetworkX Graph API • PyTorch & TensorFlow Data-Loaders</h4>
+
+---
+
+## Usecases
+
+Let's start with the simplest and work our way up.
+
+1. Getting a Python, GoLang, Java wrapper for vanilla RocksDB or LevelDB.
+2. Serving them over a network via Apache Arrow Flight RPC.
+3. Embedded Document and GraphDB, that will avoid networking overheads.
+4. Semlessly Tiering Multi-Modal DBMS between RAM and persistent backends. 
+
+Even with just a single node, in a 2U chassis today we can easily get 24x 16 TB of NVMe storage connected to 2x CPU sockets, totalling at 384 TB of space, capable of yielding ~120 GB/s of read throughput, out of which, ~70 GB/s our in-house engine can already sustain.
+Combining it with numerous features above and GPU acceleration, once can get an all-one Data-Lake with the feel of Pandas, speed of Rapids, scale of [Hadoop][hadoop] and consistency of Postgres.
 
-This would produce hundreds of binaries for all kinds of use cases, like:
+![UKV Data-lake](assets/UKV_Combo.png)
 
-* Python, GoLang, Java and other high-level bindings for [RocksDB](rocksdb.org) and [LevelDB](https://github.com/google/leveldb).
-* Performant embedded store in the foundation of your in-house storage solution.
-* Document store, that is simpler and faster than putting JSONs in MongoDB or Postgres.
-* Graph database, with the feel of [NetworkX](https://networkx.org), ~~soon~~ speed of [GunRock](http://gunrock.github.io) and scale of [Hadoop](https://hadoop.apache.org).
-* Low-latency media storage for games, CDNs and ML/BI pipelines.
+<h3 align="left"><s>One ring to rule them all.</s></h3>
+<h3 align="left">One lake to server them all.</h3>
 
-But more importantly, if you choose backends that support transactions and collections, you can get an all-in one solution:
+---
 
-![UKV Monolithic Data-lake](assets/UKV_Combo.png)
+## The Simplicity of Consistency
 
-It is normal to have a separate Postgres for your transactional data, a MongoDB for your large flexible-schema document collections, a Neo4J instance for your graphs, and an S3 storage bucket for your media data, all serving the different data needs of a single business.
+It is normal to have a separate Postgres for your transactional data, a MongoDB for your large flexible-schema document collections, a Neo4J instance for your graphs, and an [S3][s3] storage bucket for your media data, all serving the different data needs of a single business.
 
 > Example: a social network, storing passwords in Postgres, posts in MongoDB, user relations in Neo4J and post attachments in S3.
 
 So when the data is updated, you have to apply changes across all those instances, manually rolling them back if one of the parts failed.
 Needless to say, every system has a different API, different guarantees, and runtime constraints.
-UKV provides something far more uniform, simple, and performant *with the right backend*.
-When picking the UnumKV backend, we bring our entire IO stack, bypassing the Linux kernel for storage and networking operations.
-This yields speedups not just for small-ish OLTP and mid-size OLAP, but even streaming-out Gigabyte-sized videos.
-**One ~~ring~~ data-lake to rule them all.**
+
+---
+
+## Binary = Performance
+
+Over the years we broke speed limits on CPUs and GPUs using SIMD, branch-less computing and innovation in parallel algorithm design.
+We deliberately minimize the dynamic allocations in all modality implementations.
+Our engine implementations - "RAM" and "KV" follow the same tradition, but in "LevelDB" and "RocksDB" we are forced to re-allocate, as the library authors didn't design a proper binary interface.
+The one they provide heavily depends on C++ Standard Templates Library and the use fo standard allocators.
+
+Our interface is very fast, has no dynamic polymorphism and throws no exceptions, to match the speed and the quality of fast underlying engines and work for months uninterrupted!
+We have numerous detailed blog posts on performance:
 
 ## Engines
 
 Backends differ in their functionality and purposes.
 The underlying embedded key value stores include:
 
-| Name    |  Speed   |       OS        | Transact | Collections | Persistent | [Snapshots][2] | [Watches][1] |
-| :------ | :------: | :-------------: | :------: | :---------: | :--------: | :------------: | :----------: |
-| RAM     | **10x**  | POSIX + Windows |    ✅     |      ✅      |     ❌      |       ❌        |      ✅       |
-| LevelDB |   0.5x   | POSIX + Windows |    ❌     |      ❌      |     ✅      |       ❌        |      ❌       |
-| RocksDB |    1x    | POSIX + Windows |    ✅     |      ✅      |     ✅      |       ✅        |      ✅       |
-| UnumKV  | **3-5x** |      Linux      |    ✅     |      ✅      |     ✅      |       ✅        |      ✅       |
+| Name     |  Speed   |       OS        | Transact | Collections | Persistent | [Snapshots][snap] | [Watches][watch] |
+| :------- | :------: | :-------------: | :------: | :---------: | :--------: | :---------------: | :--------------: |
+| LevelDB  |   0.5x   | POSIX + Windows |    ❌     |      ❌      |     ✅      |         ❌         |        ❌         |
+| RocksDB  |    1x    | POSIX + Windows |    ✅     |      ✅      |     ✅      |         ✅         |        ✅         |
+| Unum RAM | **10x**  | POSIX + Windows |    ✅     |      ✅      |     ❌      |         ❌         |        ✅         |
+| Unum KV  | **3-5x** |      Linux      |    ✅     |      ✅      |     ✅      |         ✅         |        ✅         |
 
 
-* RAM in-memory backend was originally served educational purposes. Then it was superseeded by the [`consistent_set`][consistent_set] and can now be considered the fastest in-memory Key-Value Store, superior to Redis, MemCached or ETCD.
+* RAM in-memory backend originally served educational purposes. Then it was superseeded by the [`consistent_set`][consistent_set] and can now be considered the fastest in-memory Key-Value Store, superior to Redis, MemCached or ETCD.
 * LevelDB was originally designed at Google and extensively used across the industry, thanks to its simplicity.
 * RocksDB improves over LevelDB, extending its functionality with transactions, named collections, and higher performance.
-* UnumKV is our proprietary in-house implementation with superior concurrency and kernel-bypass techniques, as well as, GPU acceleration.
+* UnumKV is our proprietary in-house implementation with superior concurrency-control mechnisms and Linux kernel-bypass techniques, as well as, GPU acceleration. The first of its kind.
 
 All of those backends were [benchmarked for weeks](https://unum.cloud/ucsb) using [UCSB](https://github.com/unum-cloud/ucsb), so you can choose the best stack for you specific use case.
 
 ![UCSB 10 TB Results](https://unum.cloud/assets/post/2022-09-13-ucsb-10tb/ucsb-10tb-duration.png)
 
-[1]: https://redis.io/commands/watch/
-[2]: https://github.com/facebook/rocksdb/wiki/Snapshot
-[acid]: https://en.wikipedia.org/wiki/ACID
-[consistent_set]: https://github.com/ashvardanian/consistent_set
+We have published the results for BLOB-layer abstractions for [10 TB][ucsb-10], and, previously, [1 TB][ucsb-1] collections.
+Above binary layer, in logic, those numbers are further multiplied.
+Where MongoDB does 2'000 operations/s, our Community Edition does 10'000 ops/s and the Pro Version yields 50'000 ops/s.
 
 ## Frontends
 
@@ -121,14 +170,8 @@ Once you have a good enough shared interface, it is relatively easy to build on
 * Vectors, and
 * Paths.
 
-But why even bother mixing all of it in one DBMS?
-
-There are too extremes these days: consistency and scalability, especially when working with heavily linked flexible schema data.
-The consistent camp would take a tabular/relational DBMS and add a JSON column and additional columns for every relationship they want to maintain.
-The others would take 2 different DBMS solutions - one for large collections of entries and one for the links between them, often - MongoDB and Neo4J.
-In that case, every DBMS will have a custom modality-specific scaling, sharding, and replication strategy, but synchronizing them would be impossible in mutable conditions.
-This makes it hard for the developers to choose a future-proof solution for their projects.
-By putting different modality collections in one DBMS, we allow operation-level consistency controls giving the users all the flexibility one can get.
+Is there something else you need?
+Submit a feature request!
 
 ## Installation & Deployment
 
@@ -209,3 +252,29 @@ conan create . ukv/testing --build=missing
 * Transactions are ACI(D) by-default. [What does it mean?](ukv_transaction_t)
 * Why not use LevelDB or RocksDB interface? [](ukv.h)
 * Why not use SQL, MQL or Cypher? [](ukv.h)
+
+
+[blas]: https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms
+[crud]: https://en.wikipedia.org/wiki/Create,_read,_update_and_delete
+[acid]: https://en.wikipedia.org/wiki/ACID
+[arrow]: https://arrow.apache.org/
+[patch]: https://datatracker.ietf.org/doc/html/rfc6902
+[mpack]: https://msgpack.org/index.html
+[flight]: https://arrow.apache.org/docs/format/Flight.html
+[pointer]: https://datatracker.ietf.org/doc/html/rfc6901
+[bson]: https://www.mongodb.com/json-and-bson
+[pytorch]: https://pytorch.org/
+[tensorflow]: https://tensorflow.org
+[rocksdb]: https://rocksdb.org
+[leveldb]: https://github.com/google/leveldb
+[hadoop]: https://hadoop.apache.org
+[networkx]: https://networkx.org
+[gunrock]: https://gunrock.github.io
+[s3]: https://aws.amazon.com/s3
+[dbms-cloc]: https://news.ycombinator.com/item?id=24813239
+[ucsb-10]: https://unum.cloud/post/2022-03-22-ucsb/
+[ucsb-1]: https://unum.cloud/post/2021-11-25-ycsb/
+[watch]: https://redis.io/commands/watch/
+[snap]: https://github.com/facebook/rocksdb/wiki/Snapshot
+[acid]: https://en.wikipedia.org/wiki/ACID
+[consistent_set]: https://github.com/ashvardanian/consistent_set

assets/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = ../../build/docs
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

assets/docs/benchmarks/index.rst

@@ -0,0 +1,5 @@
+============
+Benchmarks
+============
+
+.. mdinclude:: ../../../benchmarks/README.md

assets/docs/capi/files.rst

@@ -0,0 +1,37 @@
+API Referance
+===============
+
+===============
+db
+===============
+.. doxygenfile:: db.h
+
+===============
+ukv
+===============
+.. doxygenfile:: ukv.h
+
+===============
+docs
+===============
+.. doxygenfile:: docs.h
+
+===============
+graph
+===============
+.. doxygenfile:: graph.h
+
+===============
+media
+===============
+.. doxygenfile:: media.h
+
+===============
+paths
+===============
+.. doxygenfile:: paths.h
+
+===============
+arrow
+===============
+.. doxygenfile:: arrow.h

assets/docs/capi/index.rst

@@ -0,0 +1,5 @@
+========
+Tutorial
+========
+
+.. mdinclude:: ../../../include/ukv/README.md

assets/docs/conf.py

@@ -0,0 +1,31 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = 'UKV'
+copyright = '2022, Unum'
+author = 'Unum'
+release = open('../../VERSION', 'r').read()
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = ['breathe', 'sphinx_mdinclude']
+
+templates_path = ['_templates']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_logo = '../black.png'
+html_theme = 'furo'
+
+
+breathe_projects = {"UKV": "../../build/xml"}
+breathe_default_project = "UKV"