Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(common): Update standards data #11003

Merged
merged 2 commits into from
May 23, 2024
Merged

chore(common): Update standards data #11003

merged 2 commits into from
May 23, 2024

Conversation

ermshiperete
Copy link
Contributor

@ermshiperete ermshiperete commented Mar 14, 2024 •
edited

@keymanapp-test-bot skip

@keymanapp-test-bot keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Mar 14, 2024
@keymanapp-test-bot
Copy link

keymanapp-test-bot bot commented Mar 14, 2024 •
edited

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

@keymanapp-test-bot keymanapp-test-bot bot added this to the B17S3 milestone Mar 14, 2024
@ermshiperete ermshiperete changed the base branch from master to beta March 14, 2024 21:29
@ermshiperete ermshiperete changed the title chore/standards data chore(common): Update standards data Mar 14, 2024
@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-missing User tests have not yet been defined for the PR label Mar 14, 2024
@mcdurdin
Copy link
Member

We just have to be a little careful with testing in case metadata formats have changed.

@ermshiperete
Copy link
Contributor Author

One thing that changed is in iso639-3.tab the Language_Type column now has H (Historical) instead of A (Ancient) for several languages. But I'm not sure that information is used anywhere in our code.

Other than that it's only updated data AFAICT.

@darcywong00 darcywong00 modified the milestones: B17S3, B17S4 Mar 16, 2024
@ermshiperete ermshiperete marked this pull request as draft March 16, 2024 15:12
@ermshiperete ermshiperete modified the milestones: B17S4, A18S1 Mar 16, 2024
@ermshiperete ermshiperete changed the base branch from beta to master April 19, 2024 07:20
@github-actions github-actions bot added the common/resources/ Build infrastructure label Apr 23, 2024
@rc-swag
Copy link
Contributor

rc-swag commented May 3, 2024

Here are some observations using the Delphi debugger. However, I really don't understand the whole code.
There is an error about inserting dulplicates

class procedure TISO6393ToBCP47Map.Fill(dict: TDictionary<string, string>);
begin
  dict.Add('aar','aa');
  dict.Add('abk','ab');

For the failing test of sqt-YE the following if statement in CanonicalLanguageCodeUtils.FindBestTag evaluates to true and exits. For the master branch this if statement evaluates to false.

if not TLangTagsMap.LangTags.TryGetValue(t.Tag, LangTag) then
   begin
     // Not a known tag but perhaps it's a custom language
     // We'll make no further assumptions
     Exit(t.Tag);
   end;

This was referenced May 7, 2024
Closed
Closed
@mcdurdin mcdurdin mentioned this pull request May 7, 2024
Closed
@rc-swag
Copy link
Contributor

rc-swag commented May 7, 2024

The duplicate insertion area was the key here, due to duplicates in the langtag.json source, this meant the dictionary was not completely loaded resulting in these other tests failing. In other words those failed tests where a side effect of the duplicate data error.

@mcdurdin
Copy link
Member

mcdurdin commented May 7, 2024

Failing tests due to data issues in langtags.json, reported at:

Note that most of the failing tests are caused by a cascade of errors rather than these root causes.

@ermshiperete ermshiperete force-pushed the chore/standards-data branch 2 times, most recently from ba20fbe to 15a924e Compare May 7, 2024 08:58
mcdurdin
mcdurdin reviewed May 8, 2024
View reviewed changes
resources/standards-data/langtags/langtags.json Outdated
@@ -93906,7 +94074,7 @@
"full": "pht-Thai-TH",
"iana": [ "Phu Thai" ],
"iso639_3": "pht",
"latnnames": [ "Phu Thai or Phuu Thai", "Phu Thai or Phuu Thai" ],
"latnnames": [ "Phu Thai", "Phuu Thai" ],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"latnnames": [ "Phu Thai", "Phuu Thai" ],
"latnnames": [ "Phu Thai or Phuu Thai", "Phu Thai or Phuu Thai" ],

Per discussion in silnrsi/langtags#17 (comment), this is valid.

@darcywong00 darcywong00 modified the milestones: A18S1, A18S2 May 11, 2024
@ermshiperete
Copy link
Contributor Author

Updated to the latest standards data which has the data fixed that previously required to handcraft langtags.json.

@ermshiperete
chore: Update standards data
b208edc 
`langtags.json` is directly downloaded from master branch at the GitHub repo
silnrsi/langtags@68cf829
because the version served at ldml.api.sil.org still has a duplicate.
@ermshiperete
chore(windows): Split tests to easier see which ones fail
4e27f64
@ermshiperete ermshiperete marked this pull request as ready for review May 22, 2024 16:24
rc-swag
rc-swag approved these changes May 22, 2024
View reviewed changes
Copy link
Contributor

@rc-swag rc-swag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ermshiperete ermshiperete merged commit 315c60a into master May 23, 2024
25 checks passed
@ermshiperete ermshiperete deleted the chore/standards-data branch May 23, 2024 07:45
@keyman-server
Copy link
Collaborator

Changes in this pull request will be available for download in Keyman version 18.0.42-alpha

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mcdurdin mcdurdin mcdurdin left review comments

@darcywong00 darcywong00 darcywong00 left review comments

@rc-swag rc-swag rc-swag approved these changes

@jahorton jahorton Awaiting requested review from jahorton jahorton is a code owner

@sgschantz sgschantz Awaiting requested review from sgschantz

Assignees
No one assigned
Labels
chore common/resources/ Build infrastructure common/ windows/
Milestone
A18S2
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@ermshiperete @mcdurdin @rc-swag @keyman-server @darcywong00