Add BlobSchema V2 with ZeroTrie #4207

sffc · 2023-10-22T05:53:28Z

Part of #3865

Relates to #2699

Need to investigate the test failures but this should be mostly ready.

robertbastian

We should also add ZeroHashMap to these comparisons. For example the key lookup even for ZeroAsciiTrie can be a ZHM.
Speed up resource lookup and fallback at runtime #2699 is not in the 1.4 milestone. If you want to submit this non-experimentally it should be

sffc · 2023-10-24T21:51:34Z

I added this PR to the 1.4 milestone for tracking the new feature, left a comment on #2699 about investigating ZeroHashMap, and moved the issue to backlog.

sffc · 2023-10-24T21:53:31Z

Also filed #4216 for further discussion.

Related: #4216, #4207, #2699 See the numbers in locale_aux_test.rs

sffc · 2023-11-02T19:52:05Z

Bench results:

provider/construct/v1   time:   [58.462 ns 58.652 ns 58.854 ns]
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe

provider/construct/v2   time:   [58.448 ns 58.615 ns 58.781 ns]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

provider/read/v1        time:   [2.1391 µs 2.1436 µs 2.1498 µs]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high severe

provider/read/v2        time:   [1.9701 µs 1.9753 µs 1.9838 µs]
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) low mild
  1 (1.00%) high mild
  2 (2.00%) high severe

provider/blob/Cargo.toml

robertbastian · 2023-11-02T20:02:40Z

provider/blob/src/blob_data_provider.rs

+        let mut version = 1;

-        {
-            let mut exporter = BlobExporter::new_with_sink(Box::new(&mut blob));
+        while version <= 2 {


Suggested change

let mut version = 1;

{

let mut exporter = BlobExporter::new_with_sink(Box::new(&mut blob));

while version <= 2 {

for version in [1, 2] {

robertbastian · 2023-11-02T20:03:43Z

provider/blob/src/blob_data_provider.rs

-            exporter.close().unwrap();
+        let mut version = 1;
+
+        while version <= 2 {


robertbastian · 2023-11-02T20:06:34Z

provider/blob/benches/blob_version_bench.rs

+        b.iter(|| {
+            for locale in black_box(&locales).iter() {
+                let _: DataResponse<HelloWorldV1Marker> = black_box(&provider)
+                    .as_deserializing()


nit: I'd prefer benching this as a BufferProvider without deserialization

provider/blob/src/blob_data_provider.rs

robertbastian · 2023-11-02T20:11:09Z

provider/blob/src/blob_schema.rs

+            .locales
+            .get(idx0)
+            .ok_or_else(|| DataError::custom("Invalid blob bytes").with_req(key, req))?;
+        // TODO: Add a lookup function to zerotrie so we don't need to stringify


robertbastian · 2023-11-02T20:11:40Z

provider/blob/src/blob_schema.rs

+            .ok_or_else(|| DataError::custom("Invalid blob bytes").with_req(key, req))?;
+        // TODO: Add a lookup function to zerotrie so we don't need to stringify
+        let locale_str = req.locale.write_to_string();
+        let idx1 = ZeroTrieSimpleAscii::from_store(zerotrie)


blob_index?

robertbastian · 2023-11-02T20:14:57Z

provider/blob/tests/test_versions.rs

+    run_driver(exporter).unwrap();
+    assert_eq!(BLOB_V1, blob.as_slice());
+
+    let blob_privider = BlobDataProvider::try_new_from_blob(blob.into_boxed_slice()).unwrap();


provider/blob/src/export/blob_exporter.rs

robertbastian · 2023-11-02T20:17:36Z

provider/blob/src/export/blob_exporter.rs

    pub fn new_with_sink(sink: Box<dyn std::io::Write + Sync + 'w>) -> Self {
        Self {
-            resources: Mutex::new(Vec::new()),
+            resources: Mutex::new(BTreeMap::new()),


nit: change to Default::default() where possible

sffc · 2023-11-02T20:51:27Z

New bench results: after switching to testing only load_buffer:

provider/construct/v1   time:   [58.328 ns 58.559 ns 58.827 ns]
                        change: [-0.2722% +0.7391% +2.1320%] (p = 0.28 > 0.05)
                        No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  6 (6.00%) high severe

provider/construct/v2   time:   [58.070 ns 58.497 ns 59.049 ns]
                        change: [-0.6234% -0.1470% +0.4131%] (p = 0.60 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

provider/read/v1        time:   [1.4992 µs 1.5020 µs 1.5051 µs]
                        change: [-30.699% -30.154% -29.699%] (p = 0.00 < 0.05)
                        Performance has improved.

provider/read/v2        time:   [1.1215 µs 1.1232 µs 1.1246 µs]
                        change: [-43.400% -43.242% -43.088%] (p = 0.00 < 0.05)
                        Performance has improved.

Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>

provider/blob/src/export/blob_exporter.rs

Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>

sffc mentioned this pull request Oct 22, 2023

Split DateSymbols data #3865

Open

robertbastian reviewed Oct 24, 2023

View reviewed changes

sffc mentioned this pull request Oct 24, 2023

Speed up resource lookup and fallback at runtime #2699

Open

sffc added this to the 1.4 Blocking ⟨P1⟩ milestone Oct 24, 2023

sffc mentioned this pull request Oct 24, 2023

Migrate blob data to ZeroTrie #4216

Closed

sffc mentioned this pull request Oct 24, 2023

ZeroTrie: Add some tests for locale with aux key #4206

Merged

sffc added a commit that referenced this pull request Oct 25, 2023

ZeroTrie: Add some tests for locale with aux key (#4206)

e122add

Related: #4216, #4207, #2699 See the numbers in locale_aux_test.rs

sffc added 3 commits November 1, 2023 11:02

Add BlobSchema v2 with ZeroTrie (reader only)

aaa3855

Add exporter for v2

27aa3a5

Regen hello world data

b45b5fe

sffc force-pushed the blobv2 branch from f37b8a5 to b45b5fe Compare November 1, 2023 18:02

sffc added 4 commits November 1, 2023 11:49

Fix handling of empty and singleton keys; set default back to 1

42cbfa3

Add blob2 to datagen CLI; regen test postcard

1fcd2cf

Use the new postscard files in tests

e28d655

Add benches, implement new constructor APIs with docs, clippy

4399055

sffc requested a review from robertbastian November 2, 2023 19:50

sffc marked this pull request as ready for review November 2, 2023 19:50

sffc requested review from Manishearth and a team as code owners November 2, 2023 19:50

robertbastian reviewed Nov 2, 2023

View reviewed changes

sffc and others added 2 commits November 2, 2023 13:51

Update provider/blob/src/export/blob_exporter.rs

9655004

Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>

Review feedback

aa54e9d

sffc requested a review from robertbastian November 2, 2023 20:53

robertbastian previously approved these changes Nov 2, 2023

View reviewed changes

provider/blob/src/export/blob_exporter.rs Outdated Show resolved Hide resolved

provider/blob/src/export/blob_exporter.rs Outdated Show resolved Hide resolved

sffc dismissed robertbastian’s stale review via d2b0a5d November 2, 2023 21:32

Apply suggestions from code review

d2b0a5d

Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>

sffc requested a review from robertbastian November 2, 2023 21:37

robertbastian approved these changes Nov 2, 2023

View reviewed changes

Manishearth approved these changes Nov 2, 2023

View reviewed changes

sffc merged commit f223e11 into unicode-org:main Nov 2, 2023
28 checks passed

sffc deleted the blobv2 branch November 2, 2023 22:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BlobSchema V2 with ZeroTrie #4207

Add BlobSchema V2 with ZeroTrie #4207

sffc commented Oct 22, 2023 •

edited

Loading

robertbastian left a comment

sffc commented Oct 24, 2023

sffc commented Oct 24, 2023

sffc commented Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

robertbastian Nov 2, 2023

sffc Nov 2, 2023

sffc commented Nov 2, 2023

Add BlobSchema V2 with ZeroTrie #4207

Add BlobSchema V2 with ZeroTrie #4207

Conversation

sffc commented Oct 22, 2023 • edited Loading

robertbastian left a comment

Choose a reason for hiding this comment

sffc commented Oct 24, 2023

sffc commented Oct 24, 2023

sffc commented Nov 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sffc commented Nov 2, 2023

sffc commented Oct 22, 2023 •

edited

Loading