Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy reader support for s-expressions #627

Merged
merged 28 commits into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
e0a83d8
Top-level nulls, bools, ints
Jul 16, 2023
89f79aa
Consolidate impls of AsUtf8 w/helper fn
Jul 25, 2023
840be4d
Improved TextBufferView docs, removed DataSource
Jul 25, 2023
5db1ff0
Adds lazy text floats
Jul 27, 2023
07d4a70
Adds LazyRawTextReader support for comments
Jul 27, 2023
181e0a5
Adds LazyRawTextReader support for reading strings
Jul 28, 2023
357ca8f
clippy fixes
Jul 28, 2023
716ff34
Fix a couple of unit tests
Jul 29, 2023
e29fec5
Less ambitious float eq comparison
Jul 29, 2023
8f79a36
Adds LazyRawTextReader support for reading symbols
Aug 1, 2023
4cb9b2b
Adds more doc comments
Aug 1, 2023
54470d2
More doc comments
Aug 1, 2023
78014e7
Adds `LazyRawTextReader` support for reading lists
Aug 3, 2023
a6a3aa8
Adds `LazyRawTextReader` support for structs
Aug 10, 2023
4fc9078
More doc comments
Aug 10, 2023
11174ac
Adds `LazyRawTextReader` support for reading IVMs
Aug 10, 2023
719dbaa
Initial impl of a LazyRawAnyReader
Aug 11, 2023
f603872
Improved comments.
Aug 11, 2023
4696ca5
Adds LazyRawTextReader support for annotations
Aug 11, 2023
c7129ac
Adds lazy reader support for timestamps
Aug 14, 2023
44435ea
Lazy reader support for s-expressions
Aug 18, 2023
d50e05b
Fixed doc comments
Aug 18, 2023
8283422
Fix internal doc link
Aug 18, 2023
4b53bb3
Merge remote-tracking branch 'origin/main' into lazy-timestamps
Aug 23, 2023
60d5a17
Incorporates review feedback
Aug 23, 2023
db9718d
Matcher recognizes +00:00 as Zulu
Aug 23, 2023
37264a3
Merge remote-tracking branch 'origin/lazy-timestamps' into lazy-sexps
Aug 23, 2023
f935c64
Merge remote-tracking branch 'origin/main' into lazy-sexps
Aug 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ num-bigint = "0.4.3"
num-integer = "0.1.44"
num-traits = "0.2"
arrayvec = "0.7"
smallvec = "1.9.0"
smallvec = {version ="1.9.0", features = ["const_generics"]}
digest = { version = "0.9", optional = true }
sha2 = { version = "0.9", optional = true }

Expand Down
21 changes: 12 additions & 9 deletions examples/lazy_read_all_values.rs
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗺️ cargo fmt rearranged the imports a bit in this file.

Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
#[cfg(feature = "experimental-lazy-reader")]
use ion_rs::IonResult;

#[cfg(not(feature = "experimental-lazy-reader"))]
fn main() {
println!("This example requires the 'experimental-lazy-reader' feature to work.");
}

#[cfg(feature = "experimental-lazy-reader")]
use ion_rs::IonResult;

#[cfg(feature = "experimental-lazy-reader")]
fn main() -> IonResult<()> {
lazy_reader_example::read_all_values()
}

#[cfg(feature = "experimental-lazy-reader")]
mod lazy_reader_example {
use std::fs::File;
use std::process::exit;

use memmap::MmapOptions;

use ion_rs::lazy::r#struct::LazyBinaryStruct;
use ion_rs::lazy::reader::LazyBinaryReader;
use ion_rs::lazy::sequence::LazyBinarySequence;
use ion_rs::lazy::value::LazyBinaryValue;
use ion_rs::lazy::value_ref::ValueRef;
use ion_rs::IonResult;
use memmap::MmapOptions;
use std::fs::File;
use std::process::exit;

pub fn read_all_values() -> IonResult<()> {
let args: Vec<String> = std::env::args().collect();
Expand Down Expand Up @@ -53,14 +53,17 @@ mod lazy_reader_example {
fn count_value_and_children(lazy_value: &LazyBinaryValue) -> IonResult<usize> {
use ValueRef::*;
let child_count = match lazy_value.read()? {
List(s) | SExp(s) => count_sequence_children(&s)?,
List(s) => count_sequence_children(s.iter())?,
SExp(s) => count_sequence_children(s.iter())?,
Comment on lines +56 to +57
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗺️ In binary Ion, the parsing logic for the bodies of lists and for s-expressions is identical. However, in text the parsing is substantially different. Not only do they have different delimiters for opening ([ vs (), closing (] vs ), and between values (, vs whitespace-or-nothing), but s-expressions also allow for the special grammar production of operators.

In order to accommodate these differences without introducing runtime overhead via branching or dynamic dispatch, I had to breaking the LazySequence type into LazyList and LazySExp types that could house their own logic. This change was also made in the raw level and accounts for the majority of this diff.

Meanwhile, the Sequence type is still unified in the Element API. I think this is reasonable, as the Element API is intentionally divorced from the reading logic needed to deserialize a stream into Elements.

Struct(s) => count_struct_children(&s)?,
_ => 0,
};
Ok(1 + child_count)
}

fn count_sequence_children(lazy_sequence: &LazyBinarySequence) -> IonResult<usize> {
fn count_sequence_children<'a, 'b>(
lazy_sequence: impl Iterator<Item = IonResult<LazyBinaryValue<'a, 'b>>>,
) -> IonResult<usize> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗺️ This example program takes an iterator to generically handle either a list or sexp. We can always introduce a LazySequence trait later if desired.

let mut count = 0;
for value in lazy_sequence {
count += count_value_and_children(&value?)?;
Expand Down
6 changes: 3 additions & 3 deletions src/binary/binary_writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ impl<W: Write> IonWriter for BinaryWriter<W> {
panic!("Cannot set symbol ID ${symbol_id} as annotation. It is undefined.");
}
}
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text),
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text.as_ref()),
};
self.raw_writer.add_annotation(symbol_id);
}
Expand All @@ -145,7 +145,7 @@ impl<W: Write> IonWriter for BinaryWriter<W> {
));
}
}
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text),
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text.as_ref()),
};
self.raw_writer.write_symbol(symbol_id)
}
Expand All @@ -159,7 +159,7 @@ impl<W: Write> IonWriter for BinaryWriter<W> {
panic!("Cannot set symbol ID ${symbol_id} as field name. It is undefined.");
}
}
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text),
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text.as_ref()),
};
self.raw_writer.set_field_name(text);
}
Expand Down