docs

docs/tech.v3.libs.arrow.html

@@ -4,4 +4,4 @@
 <pre><code class="clojure">[org.apache.arrow/arrow-memory-netty "1.0.0"]
 [org.apache.arrow/arrow-memory-core "1.0.0"]
 [org.apache.arrow/arrow-vector "1.0.0" :exclusions [commons-codec]]
-</code></pre></div></div><div class="public anchor" id="var-message-seq"><h3>message-seq</h3><div class="usage"><code>(message-seq data)</code></div><div class="doc"><div class="markdown"><p>Given a native buffer of arrow stream data, produce a sequence of flatbuf messages</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L75">view source</a></div></div><div class="public anchor" id="var-parse-message"><h3>parse-message</h3><h4 class="type">multimethod</h4><div class="usage"></div><div class="doc"><div class="markdown"><p>Given a message, parse it just a bit into a more interpretable datastructure.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L141">view source</a></div></div><div class="public anchor" id="var-parse-message-printable"><h3>parse-message-printable</h3><div class="usage"><code>(parse-message-printable msg)</code></div><div class="doc"><div class="markdown"><p>Parse the message and return something that you can look at in the repl.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L164">view source</a></div></div><div class="public anchor" id="var-read-stream-dataset-copying"><h3>read-stream-dataset-copying</h3><div class="usage"><code>(read-stream-dataset-copying path options)</code><code>(read-stream-dataset-copying path)</code></div><div class="doc"><div class="markdown"><p>Read a single record batch and return a dataset. It is an error if there are more record batches in the file.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L385">view source</a></div></div><div class="public anchor" id="var-read-stream-dataset-inplace"><h3>read-stream-dataset-inplace</h3><div class="usage"><code>(read-stream-dataset-inplace fname &amp; [options])</code></div><div class="doc"><div class="markdown"><p>Loads data up to and including the first data record. Returns the dataset and the memory-mapped file. This method is expected to be called from within a stack resource context.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L319">view source</a></div></div><div class="public anchor" id="var-stream-.3Edataset-seq-copying"><h3>stream-&gt;dataset-seq-copying</h3><div class="usage"><code>(stream-&gt;dataset-seq-copying path options)</code><code>(stream-&gt;dataset-seq-copying path)</code></div><div class="doc"><div class="markdown"><p>Read a complete arrow file lazily. Each data record is copied into an independent dataset. Stream is closed when the last dataset is loaded.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L374">view source</a></div></div><div class="public anchor" id="var-stream-.3Edataset-seq-inplace"><h3>stream-&gt;dataset-seq-inplace</h3><div class="usage"><code>(stream-&gt;dataset-seq-inplace fname &amp; [options])</code></div><div class="doc"><div class="markdown"><p>Loads data up to and including the first data record. Returns the a lazy sequence of datasets. Datasets use mmapped data, however, so realizing the entire sequence is usually safe, even for datasets that are larger than available RAM. This method is expected to be called from within a stack resource context unless options include {:resource-type :gc}. See documentation for tech.v3.datatype.mmap/mmap-file.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L301">view source</a></div></div><div class="public anchor" id="var-visualize-arrow-stream"><h3>visualize-arrow-stream</h3><div class="usage"><code>(visualize-arrow-stream fname &amp; [options])</code></div><div class="doc"><div class="markdown"><p>Loads an arrow file via mmap pathway and parses the file into a lower-level description that prints well to the REPL. Useful for quickly seeing what is in an Arrow stream. Returned value can be used to construct datasets via in-place/parse-next-dataset. See source code to stream-&gt;dataset-seq-inplace.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow.clj#L29">view source</a></div></div><div class="public anchor" id="var-write-dataset-seq-to-stream.21"><h3>write-dataset-seq-to-stream!</h3><div class="usage"><code>(write-dataset-seq-to-stream! ds-seq path options)</code><code>(write-dataset-seq-to-stream! ds path)</code></div><div class="doc"><div class="markdown"><p>Write a sequence of datasets to a stream. Datasets are written with doseq. All datasets must be amenable to being written into vectors of the type dictated by the schema of the first dataset. Each dataset is written to a separate batch.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L208">view source</a></div></div><div class="public anchor" id="var-write-dataset-to-stream.21"><h3>write-dataset-to-stream!</h3><div class="usage"><code>(write-dataset-to-stream! ds path options)</code><code>(write-dataset-to-stream! ds path)</code></div><div class="doc"><div class="markdown"><p>Write a dataset as an arrow stream file. File will contain one record set.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L188">view source</a></div></div></div></body></html>
+</code></pre></div></div><div class="public anchor" id="var-message-seq"><h3>message-seq</h3><div class="usage"><code>(message-seq data)</code></div><div class="doc"><div class="markdown"><p>Given a native buffer of arrow stream data, produce a sequence of flatbuf messages</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L75">view source</a></div></div><div class="public anchor" id="var-parse-message"><h3>parse-message</h3><h4 class="type">multimethod</h4><div class="usage"></div><div class="doc"><div class="markdown"><p>Given a message, parse it just a bit into a more interpretable datastructure.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L141">view source</a></div></div><div class="public anchor" id="var-parse-message-printable"><h3>parse-message-printable</h3><div class="usage"><code>(parse-message-printable msg)</code></div><div class="doc"><div class="markdown"><p>Parse the message and return something that you can look at in the repl.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L164">view source</a></div></div><div class="public anchor" id="var-read-stream-dataset-copying"><h3>read-stream-dataset-copying</h3><div class="usage"><code>(read-stream-dataset-copying path options)</code><code>(read-stream-dataset-copying path)</code></div><div class="doc"><div class="markdown"><p>Read a single record batch and return a dataset. It is an error if there are more record batches in the file.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L385">view source</a></div></div><div class="public anchor" id="var-read-stream-dataset-inplace"><h3>read-stream-dataset-inplace</h3><div class="usage"><code>(read-stream-dataset-inplace fname &amp; [options])</code></div><div class="doc"><div class="markdown"><p>Loads data up to and including the first data record. Returns the dataset and the memory-mapped file. This method is expected to be called from within a stack resource context.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L319">view source</a></div></div><div class="public anchor" id="var-stream-.3Edataset-seq-copying"><h3>stream-&gt;dataset-seq-copying</h3><div class="usage"><code>(stream-&gt;dataset-seq-copying path options)</code><code>(stream-&gt;dataset-seq-copying path)</code></div><div class="doc"><div class="markdown"><p>Read a complete arrow file lazily. Each data record is copied into an independent dataset. Stream is closed when the last dataset is loaded.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L374">view source</a></div></div><div class="public anchor" id="var-stream-.3Edataset-seq-inplace"><h3>stream-&gt;dataset-seq-inplace</h3><div class="usage"><code>(stream-&gt;dataset-seq-inplace fname &amp; [options])</code></div><div class="doc"><div class="markdown"><p>Loads data up to and including the first data record. Returns the a lazy sequence of datasets. Datasets use mmapped data, however, so realizing the entire sequence is usually safe, even for datasets that are larger than available RAM. This method is expected to be called from within a stack resource context unless options include {:resource-type :gc}. See documentation for tech.v3.datatype.mmap/mmap-file.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/in_place.clj#L301">view source</a></div></div><div class="public anchor" id="var-visualize-arrow-stream"><h3>visualize-arrow-stream</h3><div class="usage"><code>(visualize-arrow-stream fname &amp; [options])</code></div><div class="doc"><div class="markdown"><p>Loads an arrow file via mmap pathway and parses the file into a lower-level description that prints well to the REPL. Useful for quickly seeing what is in an Arrow stream. Returned value can be used to construct datasets via in-place/parse-next-dataset. See source code to stream-&gt;dataset-seq-inplace.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow.clj#L30">view source</a></div></div><div class="public anchor" id="var-write-dataset-seq-to-stream.21"><h3>write-dataset-seq-to-stream!</h3><div class="usage"><code>(write-dataset-seq-to-stream! ds-seq path options)</code><code>(write-dataset-seq-to-stream! ds path)</code></div><div class="doc"><div class="markdown"><p>Write a sequence of datasets to a stream. Datasets are written with doseq. All datasets must be amenable to being written into vectors of the type dictated by the schema of the first dataset. Each dataset is written to a separate batch.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L208">view source</a></div></div><div class="public anchor" id="var-write-dataset-to-stream.21"><h3>write-dataset-to-stream!</h3><div class="usage"><code>(write-dataset-to-stream! ds path options)</code><code>(write-dataset-to-stream! ds path)</code></div><div class="doc"><div class="markdown"><p>Write a dataset as an arrow stream file. File will contain one record set.</p></div></div><div class="src-link"><a href="https://github.com/techascent/tech.ml.dataset/blob/main/src/tech/v3/libs/arrow/copying.clj#L188">view source</a></div></div></div></body></html>