Enable datagen fallbacking with full set of options #3640

sffc · 2023-07-05T15:45:33Z

Toward #2683

sffc · 2023-07-05T15:50:45Z

The errors are because I need to gate this on feature = "alloc"

robertbastian

It's unclear to me how this is useful for both us and general users. Maybe you can just include this in the datagen PR?

components/locid_transform/src/fallback/mod.rs

sffc · 2023-07-05T16:26:02Z

OK I'll not merge this separately but continue working on the deduplication in this PR.

components/list/data/data/macros/list_unit_v1.data.rs

…Hybrid}

sffc · 2023-07-25T02:16:35Z

The two most recent commits don't change behavior but have most of the new configuration options and docs.

provider/datagen/src/options.rs

provider/core/src/datagen/mod.rs

provider/datagen/src/options.rs

provider/core/src/datagen/mod.rs

robertbastian

Lgtm, not approving my own changes though

provider/datagen/src/options.rs

provider/datagen/src/source.rs

robertbastian · 2023-08-08T09:39:35Z

I generated the icu_testdata diff vs 1.2.0, but can't post it because the diff itself is over 1MB and I haven't found a service that would accept that. It seems a bit excessive, but I don't really care, the Postcard file is 5.8MB vs 4.4MB before.

The only removals are sr-Cyrl, which is a slight diff for no-fallback providers, and then there are ~18k additions.

robertbastian

Happy with this, just some small things

robertbastian · 2023-08-08T08:31:46Z

tools/make/data.toml

@@ -88,15 +88,21 @@ exit_on_error true
 if array_is_empty ${@}
    exec --fail-on-error cargo build -p icu_datagen --no-default-features --features rayon,provider_baked,bin,use_wasm,networking,icu_casemap,icu_displaynames,icu_relativetime,icu_compactdecimal --release
    bin = set "target/release/icu4x-datagen"
+    verbose = set false


While this seems useful for you, when making changes to a component we probably don't want to be verbose, and I think the logging slows down datagen.

Most people run cargo make bakeddata which is usually faster than cargo make bakeddata component/foo even without this. But for the sake of expediency I'll put this in a separate PR.

robertbastian · 2023-08-08T08:43:41Z

provider/datagen/Cargo.toml

@@ -70,7 +70,7 @@ syn = {version = "2", features = ["parsing"], optional = true }

 # Other external dependencies
 displaydoc = { version = "0.2.3", default-features = false }
-elsa = "1.7"
+elsa = { git = "https://github.com/Manishearth/elsa", rev = "56d05375eea36596432a0843721d26edf3b0ec75" }


Can you either move this to dev dependencies, or use a mutex in the test, or get Manish to do a release? I don't want to submit git hashes.

provider/datagen/src/lib.rs

robertbastian · 2023-08-08T13:05:36Z

provider/adapters/tests/data/config.json

@@ -16,6 +16,6 @@
            "path": "blob.postcard"
        }
    },
-    "fallback": "Legacy",
+    "fallback": "Hybrid",


This should probably be RuntimeManual, as this is literally used for testing manual fallback.

There is no diff if I make this change, so I'll make it because if anything it helps cover RuntimeManual.

However, note that locales: All, fallback: Hybrid is the way to run a pass-through datagen.

We don't want to run pass-through datagen though. We're testing LocaleFallbackAdapter, which should be tested with data generated for manual fallback.

robertbastian · 2023-08-08T13:24:16Z

provider/datagen/src/lib.rs

-        }
+        // TODO: Consider figuring out the cases where we don't need the fallbacker.
+        // We need it most of the time, so just pre-compute it here.
+        provider.source.fallbacker = Some(LocaleFallbacker::try_new_unstable(&provider)?);


Note that in #2683 (comment) we decided that preresolved+explicit would not include ancestors and descendants. There should be at least one mode that allows you to exactly control which locales get included. We're hitting this problem with hello world, clients will hit it as well.

provider/datagen/src/lib.rs

Conflicts: components/datetime/data/data/macros/datetime_skeletons_v1.data.rs provider/datagen/tests/data/postcard/fingerprints.csv tools/make/data.toml

robertbastian · 2023-08-10T08:24:54Z

provider/datagen/src/lib.rs

+                        }
+                        None => (),
+                    };
+                    option_iter.get_or_insert_with(|| {


You can return the error here if you don't use get_or_insert_with, then you don't have to deal with it in three other locations.

sffc · 2023-08-10T16:45:59Z

@robertbastian Do you approve this PR other than the last comment you made about code style? If I fix that comment during the day can I merge this with a rubber stamp from Manish?

robertbastian · 2023-08-10T17:16:58Z

Yes

Conflicts: components/datetime/data/data/macros/datetime_skeletons_v1.data.rs

Add pub fn sort_locales_into_groups

1d7ad7e

sffc requested review from dminor and zbraniecki as code owners July 5, 2023 15:45

sffc requested review from robertbastian and removed request for dminor and zbraniecki July 5, 2023 15:45

robertbastian reviewed Jul 5, 2023

View reviewed changes

components/locid_transform/src/fallback/mod.rs Outdated Show resolved Hide resolved

components/locid_transform/src/fallback/mod.rs Outdated Show resolved Hide resolved

sffc marked this pull request as draft July 5, 2023 16:26

sffc added 7 commits July 6, 2023 15:05

Merge branch 'main' into fb2

a0bbfd2

include Vec

a7d588a

Re-write filtering function

7ec0bc4

Fix

7622ca2

Regen decimal/symbols@1

94b75de

Regen the other components

535e7ae

Merge branch 'main' into fb2

718cff2

robertbastian reviewed Jul 19, 2023

View reviewed changes

components/list/data/data/macros/list_unit_v1.data.rs Outdated Show resolved Hide resolved

sffc added 3 commits July 23, 2023 09:59

Merge branch 'main' into fb2

9c3f13e

Add plumbing for FallbackMode::{Runtime, RuntimeManual, Preresolved, …

f37be7d

…Hybrid}

Rename variants of icu_provider::datagen::FallbackMode

51b99a5

sffc changed the title ~~Add pub fn sort_locales_into_groups~~ Enable datagen fallbacking with full set of options Jul 25, 2023

sffc commented Jul 25, 2023

View reviewed changes

provider/datagen/src/options.rs Outdated Show resolved Hide resolved

provider/core/src/datagen/mod.rs Outdated Show resolved Hide resolved

provider/core/src/datagen/mod.rs Outdated Show resolved Hide resolved

provider/core/src/datagen/mod.rs Show resolved Hide resolved

sffc added 3 commits July 25, 2023 17:23

Feedback from bikeshedding with Rob

74b6577

Fix build

96124d4

Very rough checkpoint

4619124

robertbastian reviewed Jul 26, 2023

View reviewed changes

provider/core/src/datagen/mod.rs Outdated Show resolved Hide resolved

provider/datagen/src/options.rs Outdated Show resolved Hide resolved

provider/core/src/datagen/mod.rs Outdated Show resolved Hide resolved

sffc added 2 commits July 26, 2023 15:21

Change to git tag version of elsa

c6612e0

Make LocaleFallbackerWithConfig be Copy

6d74a74

robertbastian added 2 commits August 5, 2023 10:01

fix select_locales_for_key

f8d3a51

reduce diff

778a49d

robertbastian reviewed Aug 5, 2023

View reviewed changes

provider/datagen/src/options.rs Show resolved Hide resolved

provider/datagen/src/source.rs Outdated Show resolved Hide resolved

tidy

3efc3a6

robertbastian mentioned this pull request Aug 7, 2023

Setting CompactDecimalPatternDataV1 extension #3785

Merged

Revert unrelated changes in options.rs and source.rs

ee8c323

sffc mentioned this pull request Aug 8, 2023

Cleanup in options.rs and source.rs #3809

Closed

Merge branch 'main' into fb2

ec38363

robertbastian reviewed Aug 8, 2023

View reviewed changes

sffc added 6 commits August 9, 2023 23:07

Merge branch 'main' into fb2

5ccc7bb

Conflicts: components/datetime/data/data/macros/datetime_skeletons_v1.data.rs provider/datagen/tests/data/postcard/fingerprints.csv tools/make/data.toml

Regen

975c443

Revert changes to tools/make/data.toml

5f6cd24

Don't require CLDR in locales: "All" or fallback: "Preresolved"

49365bd

Remove obsolete clippy suppression

d73f8d4

Use RuntimeManual somewhere

04023ba

sffc requested a review from robertbastian August 10, 2023 07:41

clippy

20009d2

robertbastian reviewed Aug 10, 2023

View reviewed changes

robertbastian previously approved these changes Aug 10, 2023

View reviewed changes

sffc added 3 commits August 10, 2023 14:10

Merge branch 'main' into fb2

adc4d2e

Conflicts: components/datetime/data/data/macros/datetime_skeletons_v1.data.rs

regen

7897484

Catch fallbacker error sooner

cfa4600

sffc dismissed robertbastian’s stale review via cfa4600 August 10, 2023 21:23

Manishearth approved these changes Aug 10, 2023

View reviewed changes

sffc merged commit 3806e3a into unicode-org:main Aug 10, 2023
26 checks passed

sffc deleted the fb2 branch August 10, 2023 22:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable datagen fallbacking with full set of options #3640

Enable datagen fallbacking with full set of options #3640

sffc commented Jul 5, 2023

sffc commented Jul 5, 2023

robertbastian left a comment

sffc commented Jul 5, 2023

sffc commented Jul 25, 2023

robertbastian left a comment

robertbastian commented Aug 8, 2023 •

edited

Loading

robertbastian left a comment

robertbastian Aug 8, 2023

sffc Aug 10, 2023

robertbastian Aug 8, 2023

robertbastian Aug 8, 2023

sffc Aug 10, 2023

robertbastian Aug 10, 2023

robertbastian Aug 8, 2023

robertbastian Aug 10, 2023

sffc commented Aug 10, 2023

robertbastian commented Aug 10, 2023

Enable datagen fallbacking with full set of options #3640

Enable datagen fallbacking with full set of options #3640

Conversation

sffc commented Jul 5, 2023

sffc commented Jul 5, 2023

robertbastian left a comment

Choose a reason for hiding this comment

sffc commented Jul 5, 2023

sffc commented Jul 25, 2023

robertbastian left a comment

Choose a reason for hiding this comment

robertbastian commented Aug 8, 2023 • edited Loading

robertbastian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sffc commented Aug 10, 2023

robertbastian commented Aug 10, 2023

robertbastian commented Aug 8, 2023 •

edited

Loading