Removing auto-derived Ord impl for Locale/LangId #2142

snktd · 2022-06-30T01:34:21Z

I had to provide custom Ord implementation for LanguageIdentifier and ResourceOptions as these are being used widely across the codebase. For example, in many places LangId is being used inside a LiteMap and cause errors if Ord is not implemented for LangId.

Also, I am not entirely sure if this is the correct way to implement Ord. Let me know if I missed anything!

sffc · 2022-06-30T02:58:17Z

components/locid/src/langid.rs

@@ -351,6 +351,18 @@ impl FromStr for LanguageIdentifier {
    }
 }

+impl Ord for LanguageIdentifier {


Issue: we can't agree on what the behavior of impl Ord for LanguageIdentifier should be, so we don't want any impl of it. What things in ICU4X require it?

These are the list of errors that I am running into if there is no Ord impl for LanguageIdentifier. This is probably not complete and most if not all of these errors are related to LiteMap.

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/calendar/japanese.rs#L60

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/decimal/mod.rs#L82

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/list/mod.rs#L42

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/locale_canonicalizer/aliases.rs#L79

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/plurals/mod.rs#L61

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/ca.rs#L165

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/likely_subtags.rs#L16

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/list_patterns.rs#L52

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/numbers.rs#L110

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/parent_locales.rs#L16

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/time_zones/time_zone_names.rs#L164

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/cldr_serde/time_zones/time_zone_names.rs#L164

https://github.com/unicode-org/icu4x/blob/main/provider/datagen/src/transform/cldr/datetime/mod.rs#L89

sffc · 2022-06-30T02:59:46Z

provider/core/src/resource.rs

@@ -383,6 +383,18 @@ impl fmt::Display for ResourceOptions {
    }
 }

+impl Ord for ResourceOptions {
+    fn cmp(&self, other: &Self) -> Ordering {
+        self.strict_cmp(other.to_string().as_bytes())


Issue: we should not call to_string(), which is an expensive function, in cmp, which should be efficient since it may be called many times.

Where do we need impl Ord for ResourceOptions? Maybe we can change the call sites instead.

Maybe we can switch to HashMap as you commented below. This is where I am getting errors for ResourceOptions if Ord impl is not provided.

https://github.com/unicode-org/icu4x/blob/main/provider/core/src/hello_world.rs#L110

https://github.com/unicode-org/icu4x/blob/main/provider/core/src/hello_world.rs#L127

Probably because HelloWorldProvider uses LiteMap: https://github.com/unicode-org/icu4x/blob/main/provider/core/src/hello_world.rs#L127

HelloWorldProvider should just use HashMap, it doesn't matter

Oh, it can't, for reasons Rob mentioned

We could instead just use a manual LiteMap, a Vec of tuples that we binary search

I moved all of datagen to LiteMap a while ago in part because LiteMap has a bunch of utility functions that are useful for us that don't exist in HashMap.

This is a place where LocaleStr would be a good type to have.

I'm not quite sure what to recommend to unblock @snktd.

I don't think any of those utility functions you mentioned are still in use. See #2152, which removes LiteMap deps and unblocks this.

Manishearth · 2022-06-30T16:40:43Z

For example, in many places LangId is being used inside a LiteMap and cause errors if Ord is not implemented for LangId.

@sffc I wonder if we should be using a regular hashmap instead here?

sffc · 2022-06-30T16:54:52Z

It would be useful to see a list of the places and decide what to do there:

Switch from LiteMap to HashMap?
Use a different type (like a string) as the key?
Something else?

Manishearth · 2022-06-30T17:06:11Z

actually yeah a string might be fine

snktd · 2022-06-30T22:10:54Z

it would be useful to see a list of the places and decide what to do there:

Added the list of code-paths that throw errors if Ord impl is removed.

robertbastian · 2022-07-01T13:20:28Z

All occurences of LiteMap<LanguageIdentifier, ...> are in datagen::transform::cldr::cldr_serde. I think we should just use HashMaps.

robertbastian · 2022-07-01T13:25:53Z

HashMap won't work in #[no_std] as it requires entropy. So for HelloWorld we'll need another solution. But we can just linearly scan through the list, it's a dummy key where it doesn't really matter.

robertbastian · 2022-07-11T06:26:06Z

#2152 should have unblocked this

snktd · 2022-07-11T22:46:44Z

#2152 should have unblocked this

Tried to fetch and update. There are still few more issues removing the Ord impl.

Looks like we are comparing LangId here: https://github.com/snktd/icu4x/blob/remove-ord-impl/provider/datagen/src/transform/cldr/locale_canonicalizer/aliases.rs#L79. Which throws error if Ord isn't implemented for LangId. Maybe we can do string comparison here?
Still can't remove Ord impl from ResourceOptions. Because a Vector containing ResourceOptions (Vec<(ResourceKey, ResourceOptions, String), Global>) is being sorted here: https://github.com/snktd/icu4x/blob/remove-ord-impl/provider/fs/src/export/fs_exporter.rs#L144. Not sure what is the best option here.

sffc · 2022-07-12T00:33:45Z

Yes, I think it is actually more correct to switch it to a string comparison anyway, given that the comment says: "Order the set of rules ... alphabetically by field"
Is being fixed by Custom eq for ResourceKey #2163

robertbastian · 2022-07-12T08:52:47Z

I don't think that's correct. We are following this algorithm, and assuming our current code is correct, changing to string comparison is not, because it changes the location of und-* ids.

sffc · 2022-07-14T22:04:36Z

provider/core/src/data_provider.rs

-#[derive(Default, Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
+#[derive(Default, Debug, PartialEq, Eq)]


I think I see what's going on. You should not remove the Clone here.

Ah right! Thanks for catching! Updated.

Removing auto-derived Ord impl for Locale/LangId

c0dd818

snktd requested review from sffc, Manishearth, zbraniecki and nciric as code owners June 30, 2022 01:34

sffc requested changes Jun 30, 2022

View reviewed changes

Manishearth removed request for zbraniecki and nciric June 30, 2022 04:00

robertbastian mentioned this pull request Jul 11, 2022

Custom eq for ResourceKey #2163

Merged

Merge branch 'unicode-org:main' into remove-ord-impl

e3870ff

snktd and others added 3 commits July 12, 2022 16:21

Merge branch 'unicode-org:main' into remove-ord-impl

4427be2

Removing custom Ord implementation.

cc904ce

Merge branch 'main' into remove-ord-impl

e6ae594

sffc reviewed Jul 14, 2022

View reviewed changes

snktd and others added 2 commits July 14, 2022 15:51

Adding the Clone impl back to data provider.

1049996

Merge branch 'unicode-org:main' into remove-ord-impl

a1ecd5a

sffc approved these changes Jul 14, 2022

View reviewed changes

Manishearth approved these changes Jul 14, 2022

View reviewed changes

sffc merged commit 1a4233e into unicode-org:main Jul 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removing auto-derived Ord impl for Locale/LangId #2142

Removing auto-derived Ord impl for Locale/LangId #2142

snktd commented Jun 30, 2022 •

edited by robertbastian

Loading

sffc Jun 30, 2022

snktd Jun 30, 2022

sffc Jun 30, 2022

snktd Jun 30, 2022

Manishearth Jul 1, 2022

Manishearth Jul 1, 2022

sffc Jul 1, 2022

robertbastian Jul 6, 2022

Manishearth commented Jun 30, 2022

sffc commented Jun 30, 2022

Manishearth commented Jun 30, 2022

snktd commented Jun 30, 2022 •

edited

Loading

robertbastian commented Jul 1, 2022

robertbastian commented Jul 1, 2022 •

edited

Loading

robertbastian commented Jul 11, 2022

snktd commented Jul 11, 2022

sffc commented Jul 12, 2022

robertbastian commented Jul 12, 2022

sffc Jul 14, 2022

snktd Jul 14, 2022

		#[derive(Default, Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
		#[derive(Default, Debug, PartialEq, Eq)]

Removing auto-derived Ord impl for Locale/LangId #2142

Removing auto-derived Ord impl for Locale/LangId #2142

Conversation

snktd commented Jun 30, 2022 • edited by robertbastian Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Manishearth commented Jun 30, 2022

sffc commented Jun 30, 2022

Manishearth commented Jun 30, 2022

snktd commented Jun 30, 2022 • edited Loading

robertbastian commented Jul 1, 2022

robertbastian commented Jul 1, 2022 • edited Loading

robertbastian commented Jul 11, 2022

snktd commented Jul 11, 2022

sffc commented Jul 12, 2022

robertbastian commented Jul 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snktd commented Jun 30, 2022 •

edited by robertbastian

Loading

snktd commented Jun 30, 2022 •

edited

Loading

robertbastian commented Jul 1, 2022 •

edited

Loading