Add readme and license

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Guillermo Rodriguez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,81 @@
+# arrowbatch-nodejs
+
+`yarn add @guilledk/arrowbatch-nodejs`
+
+## Protocol
+
+The ArrowBatch v1 protocol is a binary format that allows for streaming new rows to a file in a way that the files can be arbitrarily large while still retaining fast random access properties. It achieves this by sequentially appending random access Apache Arrow tables of a specific batch size plus a small header before each, to form a bigger table.
+
+## File Structure
+
+The ArrowBatch v1 file structure consists of a global header followed by a sequence of batches, each containing a batch header and the Arrow random access file bytes.
+
+```
++-----------------+
+|  Global Header  |
++-----------------+
+|  Batch 0 Header |
++-----------------+
+|    Arrow Table  |
+|     Batch 0     |
++-----------------+
+|  Batch 1 Header |
++-----------------+
+|    Arrow Table  |
+|     Batch 1     |
++-----------------+
+|      ...        |
++-----------------+
+```
+
+### Global Header
+
+The global header contains the following information:
+
+- Version constant: ASCII string "ARROW-BATCH1"
+
+### Batch Header
+
+Each batch header contains the following fields:
+
+- Batch header constant: ASCII string "ARROW-BATCH-TABLE"
+- Batch byte size: 64-bit unsigned integer representing the size of the Arrow table batch in bytes
+- Compression: 8-bit unsigned integer indicating the compression method used (0 for uncompressed, 1 for zstd)
+
+## Streaming
+
+The ArrowBatch v1 format can be streamed easily by reading each batch header as it comes in, then expecting to read the new full Arrow random access batch sent.
+
+## Random Access
+
+To perform random access on a disk ArrowBatch v1 file:
+
+1. Read the global header.
+2. Before reading any actual Arrow table data, read all Arrow batch headers in order by seeking around on the file using the specified metadata values on the batch headers.
+3. Once the batch that contains the desired row is reached, read that batch and perform queries that only affect that small batch.
+
+## Compression
+
+The ArrowBatch v1 protocol supports compression of the Arrow table batches. The supported compression methods are:
+
+- 0: Uncompressed
+- 1: zstd (Zstandard compression)
+
+The compression method is specified in the batch header.
+
+## Data Model
+
+The ArrowBatch v1 protocol defines a data model using Arrow table mappings. Each table mapping specifies the name, data type, and additional properties of the fields in the table.
+
+The supported data types include:
+
+- Unsigned integers (u8, u16, u32, u64, uintvar)
+- Signed integers (i64)
+- Bytes (string, bytes, base64)
+- Digests (checksum160, checksum256)
+
+The table mappings also allow specifying optional fields, fixed-length fields, arrays, and references to other tables.
+
+## Conclusion
+
+The ArrowBatch v1 protocol provides an efficient way to store and access large amounts of structured data while supporting streaming and random access. By leveraging the Apache Arrow format and incorporating compression, it offers flexibility and performance for various data processing scenarios.