Diagnostics `icu4x` based list formatting. #104047

crlf0710 · 2022-11-06T09:19:06Z

This adds a new kind of DiagnosticArg and add the ability to convert it to a FluentValue::Custom. When emitting fluent output, it makes use of ListFormatter from icu4x project to convert it to localized version of list following the fluent locale, as a kind of eager translation.

Tested locally with locales like en, ja, etc, and they work fine. ~~Though neither zh-CN nor zh-Hans works correctly, it seems they fallback to und locale somehow, emitting only comma-based list. I believe this is an internal issue of icu4x itself.~~(Works fine after #104047 (comment))

Would love to hear what others think.

r? @davidtwco
cc @Manishearth

rustbot · 2022-11-06T09:19:15Z

rustc_error_messages was changed

cc @davidtwco, @compiler-errors, @JohnTitor, @estebank, @TaKO8Ki

crlf0710 · 2022-11-06T09:48:59Z

Mmmm, seems there's non-thread-safe things within icu_list::ListFormatter... @Manishearth

Manishearth · 2022-11-06T16:31:55Z

@crlf0710 you need the sync feature on the icu_provider crate.

Manishearth · 2022-11-06T16:40:03Z

So I think you should simply generate for zh and zh-Hant, since zh-Hans is the default for zh. Unsure why script fallback doesn't take care of that though, cc @sffc

Manishearth · 2022-11-06T16:41:17Z

Also, cc @zbraniecki: do you think this is the right way to use list formatter with fluent? This is only for lists of untranslated placeholders.

crlf0710 · 2022-11-06T18:12:00Z

@crlf0710 you need the sync feature on the icu_provider crate.

Thanks! In the two recent commits, I updated the features flags to include sync for parallel compiler. And also included zh into datagen locale selection.

davidtwco

I can't speak to any of the icu4x usage or the baked data stuff, but the integration into DiagnosticArg and other rustc infrastructure looks good to me.

zbraniecki · 2022-11-07T16:41:40Z

Also, cc @zbraniecki: do you think this is the right way to use list formatter with fluent? This is only for lists of untranslated placeholders.

LGTM so far, but I'll wait to see how it's hooked into Fluent (this PR only hooks in Memoizable)

Manishearth

ICU4X usage seems good too

Manishearth · 2022-11-07T16:53:10Z

compiler/rustc_errors/src/diagnostic.rs

@@ -49,6 +51,7 @@ impl<'source> Into<FluentValue<'source>> for DiagnosticArgValue<'source> {
        match self {
            DiagnosticArgValue::Str(s) => From::from(s),
            DiagnosticArgValue::Number(n) => From::from(n),
+            DiagnosticArgValue::StrListSepByAnd(l) => fluent_value_from_str_list_sep_by_and(l),


@zbraniecki this is where it gets hooked in to fluent

Manishearth · 2022-11-07T16:54:07Z

@zbraniecki it's already hooked into Fluent; the lists eventually produce a FluentValue

sffc · 2022-11-07T17:17:02Z

The fallback algorithm visits zh instead of zh-Hans since Hans is the default script for zh.

Upstream issues to improve documentation and tooling around this: unicode-org/icu4x#2683, unicode-org/icu4x#2243, unicode-org/icu4x#834

crlf0710 · 2022-11-12T07:08:06Z

Rebased forward.

davidtwco

r=me after last comment

compiler/rustc_baked_icu_data/src/lib.rs

Manishearth · 2022-11-16T15:33:56Z

@bors r=davidtwco

bors · 2022-11-16T15:33:58Z

📌 Commit d75c76d has been approved by davidtwco

It is now in the queue for this repository.

…, r=davidtwco Diagnostics `icu4x` based list formatting. This adds a new kind of `DiagnosticArg` and add the ability to convert it to a `FluentValue::Custom`. When emitting fluent output, it makes use of `ListFormatter` from `icu4x` project to convert it to localized version of list following the fluent locale, as a kind of eager translation. Tested locally with locales like `en`, `ja`, etc, and they work fine. <del>Though neither `zh-CN` nor `zh-Hans` works correctly, it seems they fallback to `und` locale somehow, emitting only comma-based list. I believe this is an internal issue of `icu4x` itself.</del>(Works fine after rust-lang#104047 (comment)) Would love to hear what others think. r? `@davidtwco` cc `@Manishearth`

…, r=davidtwco Diagnostics `icu4x` based list formatting. This adds a new kind of `DiagnosticArg` and add the ability to convert it to a `FluentValue::Custom`. When emitting fluent output, it makes use of `ListFormatter` from `icu4x` project to convert it to localized version of list following the fluent locale, as a kind of eager translation. Tested locally with locales like `en`, `ja`, etc, and they work fine. <del>Though neither `zh-CN` nor `zh-Hans` works correctly, it seems they fallback to `und` locale somehow, emitting only comma-based list. I believe this is an internal issue of `icu4x` itself.</del>(Works fine after rust-lang#104047 (comment)) Would love to hear what others think. r? ``@davidtwco`` cc ``@Manishearth``

Manishearth · 2022-11-18T22:28:53Z

@bors rollup=iffy

this will break naïve rollups because it introduces a new dep

…, r=davidtwco Diagnostics `icu4x` based list formatting. This adds a new kind of `DiagnosticArg` and add the ability to convert it to a `FluentValue::Custom`. When emitting fluent output, it makes use of `ListFormatter` from `icu4x` project to convert it to localized version of list following the fluent locale, as a kind of eager translation. Tested locally with locales like `en`, `ja`, etc, and they work fine. <del>Though neither `zh-CN` nor `zh-Hans` works correctly, it seems they fallback to `und` locale somehow, emitting only comma-based list. I believe this is an internal issue of `icu4x` itself.</del>(Works fine after rust-lang#104047 (comment)) Would love to hear what others think. r? `@davidtwco` cc `@Manishearth`

Manishearth · 2022-11-18T22:48:00Z

@bors r=davidtwco rollup=maybe

lockfile problems were inherent to this PR after merge, did a force push with fixes

bors · 2022-11-18T22:48:02Z

📌 Commit 2721e40 has been approved by davidtwco

It is now in the queue for this repository.

…earth Rollup of 8 pull requests Successful merges: - rust-lang#102977 (remove HRTB from `[T]::is_sorted_by{,_key}`) - rust-lang#103378 (Fix mod_inv termination for the last iteration) - rust-lang#103456 (`unchecked_{shl|shr}` should use `u32` as the RHS) - rust-lang#103701 (Simplify some pointer method implementations) - rust-lang#104047 (Diagnostics `icu4x` based list formatting.) - rust-lang#104338 (Enforce that `dyn*` coercions are actually pointer-sized) - rust-lang#104498 (Edit docs for `rustc_errors::Handler::stash_diagnostic`) - rust-lang#104556 (rustdoc: use `code-header` class to format enum variants) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

rustbot assigned davidtwco Nov 6, 2022

This comment has been minimized.

Sign in to view

davidtwco approved these changes Nov 7, 2022

View reviewed changes

Manishearth approved these changes Nov 7, 2022

View reviewed changes

davidtwco mentioned this pull request Nov 8, 2022

Diagnostic Translation #100717

Open

84 tasks

This comment was marked as resolved.

Sign in to view

crlf0710 force-pushed the icu_based_list_formatting branch from ad990e0 to 4c24176 Compare November 12, 2022 07:07

davidtwco approved these changes Nov 14, 2022

View reviewed changes

compiler/rustc_baked_icu_data/src/lib.rs Outdated Show resolved Hide resolved

Manishearth reviewed Nov 14, 2022

View reviewed changes

compiler/rustc_baked_icu_data/src/lib.rs Outdated Show resolved Hide resolved

crlf0710 force-pushed the icu_based_list_formatting branch 2 times, most recently from 177cfbd to d75c76d Compare November 16, 2022 02:52

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 16, 2022

Manishearth mentioned this pull request Nov 16, 2022

Rollup of 6 pull requests #104495

Closed

Manishearth mentioned this pull request Nov 18, 2022

Rollup of 9 pull requests #104585

Closed

Manishearth mentioned this pull request Nov 18, 2022

Rollup of 8 pull requests #104589

Closed

crlf0710 and others added 8 commits November 18, 2022 14:46

Add rustc_baked_icu_data crate.

42d3bda

Migrate diagnostics list output to use icu list formatter.

a775004

Import icu locale fallback data

bde2f98

Enable icu sync feature for parallel compiler

d15b020

Include zh locale in icu data

707c035

Fix compilation issue after rebase

b21e0b8

Update crate documentation of rustc_baked_icu_data crate

b22cb90

Fix lockfile

2721e40

Manishearth force-pushed the icu_based_list_formatting branch from d75c76d to 2721e40 Compare November 18, 2022 22:47

Manishearth mentioned this pull request Nov 18, 2022

Rollup of 8 pull requests #104591

Merged

bors merged commit 3181006 into rust-lang:master Nov 19, 2022

rustbot added this to the 1.67.0 milestone Nov 19, 2022

RalfJung mentioned this pull request Nov 19, 2022

More deriving on packed structs #104429

Merged

davidtwco mentioned this pull request Nov 23, 2022

Migrate rustc_mir_build diagnostics #104417

Merged

mejrs mentioned this pull request Jan 18, 2023

Let Vec be used as an argument in fluent strings #106986

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Diagnostics `icu4x` based list formatting. #104047

Diagnostics `icu4x` based list formatting. #104047

crlf0710 commented Nov 6, 2022 •

edited

Loading

rustbot commented Nov 6, 2022

This comment has been minimized.

crlf0710 commented Nov 6, 2022

Manishearth commented Nov 6, 2022

Manishearth commented Nov 6, 2022 •

edited

Loading

Manishearth commented Nov 6, 2022

crlf0710 commented Nov 6, 2022

davidtwco left a comment

zbraniecki commented Nov 7, 2022

Manishearth left a comment

Manishearth Nov 7, 2022

Manishearth commented Nov 7, 2022

sffc commented Nov 7, 2022

This comment was marked as resolved.

crlf0710 commented Nov 12, 2022

davidtwco left a comment

Manishearth commented Nov 16, 2022

bors commented Nov 16, 2022

Manishearth commented Nov 18, 2022

Manishearth commented Nov 18, 2022

bors commented Nov 18, 2022

Diagnostics icu4x based list formatting. #104047

Diagnostics icu4x based list formatting. #104047

Conversation

crlf0710 commented Nov 6, 2022 • edited Loading

rustbot commented Nov 6, 2022

This comment has been minimized.

crlf0710 commented Nov 6, 2022

Manishearth commented Nov 6, 2022

Manishearth commented Nov 6, 2022 • edited Loading

Manishearth commented Nov 6, 2022

crlf0710 commented Nov 6, 2022

davidtwco left a comment

Choose a reason for hiding this comment

zbraniecki commented Nov 7, 2022

Manishearth left a comment

Choose a reason for hiding this comment

Manishearth Nov 7, 2022

Choose a reason for hiding this comment

Manishearth commented Nov 7, 2022

sffc commented Nov 7, 2022

This comment was marked as resolved.

crlf0710 commented Nov 12, 2022

davidtwco left a comment

Choose a reason for hiding this comment

Manishearth commented Nov 16, 2022

bors commented Nov 16, 2022

Manishearth commented Nov 18, 2022

Manishearth commented Nov 18, 2022

bors commented Nov 18, 2022

Diagnostics `icu4x` based list formatting. #104047

Diagnostics `icu4x` based list formatting. #104047

crlf0710 commented Nov 6, 2022 •

edited

Loading

Manishearth commented Nov 6, 2022 •

edited

Loading