New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collaboration with Globalize on CLDR? #1241

Open
jzaefferer opened this Issue Oct 31, 2013 · 16 comments

Comments

Projects
None yet
10 participants
@jzaefferer

jzaefferer commented Oct 31, 2013

This is not a code issue, but StackOverflow is inappropriate for this request and I didn't yet want to email committers individually.

I'm on the jQuery Team, and we're working on a project called Globalize. Globalize has a lot of overlap with Moment.js. One major difference is the source for locales. As far as I can tell, Moment builds on user-contributed translations, by now 63 in total. Globalize started by importing locale data from .Net, giving as about 350 "cultures" (naming is hard...). So far not too interesting.

We're currently in the process of moving to CLDR, using their database of locales as the new base for Globalize. This requires a lot of groundwork to make the data usable in JavaScript. We're also reporting back issues to them, like missing documentation.

I've had a bit of communication with Tim Wood in the past (see #315) about using CLDR. That was a year ago, since then it looks like the project is now standing on more shoulders, so hopefully adopting CLDR, along with some form of collaboration with the jQuery Team is something worth revisiting.

On our end, @rxaviers and @scottgonzalez are the other two people most involved. We're generally around on IRC (Freenode/#jqueryui-dev) for some ad-hoc discussions. I can also make introductions per email, if that is preferred over this issue tracker (my email address is in my github profile).

@icambron

This comment has been minimized.

Show comment
Hide comment
@icambron

icambron Nov 5, 2013

Member

I'd love to get better CLDR support. A couple of ways we've looked at this are to create a plugin that packages that data (see #786), and also simply updating our language data to be manually consistent with it (there was going to be a pull request from @hagmandan, but I seem to have lost track of that thread).

From my perspective, I just don't know that much about CLDR; we've been pretty ad-hock about approach to language and I'm not even sure we're thinking about the whole subject in the right way. As Tim mentioned in that #315 thread, one of the hardest parts has been getting the grammar right, which we do through callbacks, and it's not obvious how the CLDR data would interact with that.

But I think it's fair to say we're more than open to collaborating on this. I'm pretty slammed now and I'm pretty sure @ichernev is too, but I'll try to make some time to get on IRC so we can just chat about it.

Member

icambron commented Nov 5, 2013

I'd love to get better CLDR support. A couple of ways we've looked at this are to create a plugin that packages that data (see #786), and also simply updating our language data to be manually consistent with it (there was going to be a pull request from @hagmandan, but I seem to have lost track of that thread).

From my perspective, I just don't know that much about CLDR; we've been pretty ad-hock about approach to language and I'm not even sure we're thinking about the whole subject in the right way. As Tim mentioned in that #315 thread, one of the hardest parts has been getting the grammar right, which we do through callbacks, and it's not obvious how the CLDR data would interact with that.

But I think it's fair to say we're more than open to collaborating on this. I'm pretty slammed now and I'm pretty sure @ichernev is too, but I'll try to make some time to get on IRC so we can just chat about it.

@hagmandan

This comment has been minimized.

Show comment
Hide comment
@hagmandan

hagmandan Nov 18, 2013

Contributor

Hey all - the thread I started hadn't been lost, just slowed down due to equal slammage on my end. However, @Krinkle has reached out further interest on CLDR support as well. My team was also thinking of a method to somehow pull and parse data straight from CLDR's json files; either periodically and store them statically, or dynamically when a locale is requested (this has obvious concerns with performance and reliability).

Some initial thoughts on helping with CLDR standardization through the manual method:

  1. all the language files should have the nominative ("stand-alone" from CLDR) and possessive ("format" from CLDR) available in each, regardless of if there are differences
  2. this would apply to months, abbreviated months; weekdays and abbreviated weekdays
  3. additionally, CLDR and moment.js's naming convention of patterns don't match up 100% - (compare
    http://cldr.unicode.org/translation/date-time with http://momentjs.com/docs/#/displaying/format/); while I understand that is a HUGE change, I feel it's something to consider heavily if CLDR is becoming the standard across the board.

My current branch for CLDR has a few changes in the code that aren't necessarily related to CLDR compliance (eg, unique to a project I'm working on: localization support for different formats instead of just 'LLLL', 'l', in case users want month-day or month-year instead of month-day-year) - I will work on separating these out this week so my CLDR branch will be more aligned with what it should be.

I'll try to jump onto IRC, however email or github notifications have a better chance of reaching me.

Contributor

hagmandan commented Nov 18, 2013

Hey all - the thread I started hadn't been lost, just slowed down due to equal slammage on my end. However, @Krinkle has reached out further interest on CLDR support as well. My team was also thinking of a method to somehow pull and parse data straight from CLDR's json files; either periodically and store them statically, or dynamically when a locale is requested (this has obvious concerns with performance and reliability).

Some initial thoughts on helping with CLDR standardization through the manual method:

  1. all the language files should have the nominative ("stand-alone" from CLDR) and possessive ("format" from CLDR) available in each, regardless of if there are differences
  2. this would apply to months, abbreviated months; weekdays and abbreviated weekdays
  3. additionally, CLDR and moment.js's naming convention of patterns don't match up 100% - (compare
    http://cldr.unicode.org/translation/date-time with http://momentjs.com/docs/#/displaying/format/); while I understand that is a HUGE change, I feel it's something to consider heavily if CLDR is becoming the standard across the board.

My current branch for CLDR has a few changes in the code that aren't necessarily related to CLDR compliance (eg, unique to a project I'm working on: localization support for different formats instead of just 'LLLL', 'l', in case users want month-day or month-year instead of month-day-year) - I will work on separating these out this week so my CLDR branch will be more aligned with what it should be.

I'll try to jump onto IRC, however email or github notifications have a better chance of reaching me.

@hagmandan

This comment has been minimized.

Show comment
Hide comment
@hagmandan

hagmandan Nov 18, 2013

Contributor

I would definitely be open to collaboration - there was an ongoing thread
for this that I've mentioned you in.

As I'm also slammed at work, availability may be limited at first, but
brainstorming some next steps and plan scope could be hugely advantageous
to us in the coming weeks (the holidays may yield some "free" time to work
on this!)

On Fri, Nov 15, 2013 at 10:06 AM, Timo Tijhof notifications@github.comwrote:

As an engineer working with the Wikimedia Foundationhttps://wikimediafoundation.org/(the non-profit organisation that powers Wikipedia), I'm interested in
exploring Moment.js as our library of choice for date/time parsing,
formatting etc. It would be included in the MediaWikihttps://www.mediawiki.org/software. At this time this is purely from my personal interest and
preference and not representative of the organisation in any way, we might
be looking into other libraries or develop our own if needed.

I personally really like Moment.js, but the main blocker at this point is
CLDR support and completeness (Wikipedia supports over 286 languages).

Would you be open to collaboration with us (I'd be willing to do most of
the work) in making this happen?


Reply to this email directly or view it on GitHubhttps://github.com/moment/moment/issues/1241#issuecomment-28590120
.

Contributor

hagmandan commented Nov 18, 2013

I would definitely be open to collaboration - there was an ongoing thread
for this that I've mentioned you in.

As I'm also slammed at work, availability may be limited at first, but
brainstorming some next steps and plan scope could be hugely advantageous
to us in the coming weeks (the holidays may yield some "free" time to work
on this!)

On Fri, Nov 15, 2013 at 10:06 AM, Timo Tijhof notifications@github.comwrote:

As an engineer working with the Wikimedia Foundationhttps://wikimediafoundation.org/(the non-profit organisation that powers Wikipedia), I'm interested in
exploring Moment.js as our library of choice for date/time parsing,
formatting etc. It would be included in the MediaWikihttps://www.mediawiki.org/software. At this time this is purely from my personal interest and
preference and not representative of the organisation in any way, we might
be looking into other libraries or develop our own if needed.

I personally really like Moment.js, but the main blocker at this point is
CLDR support and completeness (Wikipedia supports over 286 languages).

Would you be open to collaboration with us (I'd be willing to do most of
the work) in making this happen?


Reply to this email directly or view it on GitHubhttps://github.com/moment/moment/issues/1241#issuecomment-28590120
.

@jzaefferer

This comment has been minimized.

Show comment
Hide comment
@jzaefferer

jzaefferer Nov 27, 2013

A quick update on what we discussed elsewhere: We had an IRC meeting with @ichernev. Since then he and @rxaviers have discussed more CLDR details.

A few things we discussed:

  • Neither project has any interest spending more time on maintaining a custom locale database, we both want to be able to respond to issues with "go report it to CLDR"
  • CLDR tools (like json converter) and docs are flawed, but trying to use them and reporting docs issues should help get those improved
  • We can at least share the tool Rafael wrote for dealing with CLDR in JavaScript, which is independent of Globalize: https://github.com/rxaviers/cldr
  • Maybe Moment can use the date parsing and formatting implementation from Globalize, along with implementing other features directly on top of CLDR

jzaefferer commented Nov 27, 2013

A quick update on what we discussed elsewhere: We had an IRC meeting with @ichernev. Since then he and @rxaviers have discussed more CLDR details.

A few things we discussed:

  • Neither project has any interest spending more time on maintaining a custom locale database, we both want to be able to respond to issues with "go report it to CLDR"
  • CLDR tools (like json converter) and docs are flawed, but trying to use them and reporting docs issues should help get those improved
  • We can at least share the tool Rafael wrote for dealing with CLDR in JavaScript, which is independent of Globalize: https://github.com/rxaviers/cldr
  • Maybe Moment can use the date parsing and formatting implementation from Globalize, along with implementing other features directly on top of CLDR
@schmod

This comment has been minimized.

Show comment
Hide comment
@schmod

schmod Jan 24, 2014

I'm strongly in favor of starting work on this. Having access to the CLDR language data would make it easier to add locale-dependent features in the future.

A few disjointed thoughts about our current state of affairs, and where we should go

  • Twitter has a fairly comprehensive library for doing CLDR stuff in JavaScript. It may be worth looking at their work as we develop this further.
  • Just as the package maintainers have little interest in maintaining a custom locale database, web developers have little desire to load separate locale files for multiple libraries. It would be fantastic if I could grab a single CLDR JSON file that all of my applications/libraries could consume. I don't want to have to load separate language packs for moment, globalize, angular, etc....
  • Given that quite a few libraries and frameworks implement this functionality, it may be worth discussing how we can collaborate effectively, and share code. While it would be a great start for Moment to use CLDR's localization data, it'd be really awesome if we could split out all parsing/formatting functions into an independent module or library that could be consumed by other projects that only require this functionality.

schmod commented Jan 24, 2014

I'm strongly in favor of starting work on this. Having access to the CLDR language data would make it easier to add locale-dependent features in the future.

A few disjointed thoughts about our current state of affairs, and where we should go

  • Twitter has a fairly comprehensive library for doing CLDR stuff in JavaScript. It may be worth looking at their work as we develop this further.
  • Just as the package maintainers have little interest in maintaining a custom locale database, web developers have little desire to load separate locale files for multiple libraries. It would be fantastic if I could grab a single CLDR JSON file that all of my applications/libraries could consume. I don't want to have to load separate language packs for moment, globalize, angular, etc....
  • Given that quite a few libraries and frameworks implement this functionality, it may be worth discussing how we can collaborate effectively, and share code. While it would be a great start for Moment to use CLDR's localization data, it'd be really awesome if we could split out all parsing/formatting functions into an independent module or library that could be consumed by other projects that only require this functionality.
@rxaviers

This comment has been minimized.

Show comment
Hide comment
@rxaviers

rxaviers Jan 27, 2014

@schmod, I am thrilled to let you know that some of your wishes have already come true.

Cldr.js does exactly what you are looking for. It's being used by jQuery Globalize (on its recent complete-rewrite), whose goal was to provide a set of tools that leverage the official CLDR JSON data, allow users to load as much or as little data as they need, avoid duplicating data if using multiple i18n libraries that leverage CLDR, run in browsers and node.js.

Moment.js has a migration work going on, and we (@ichernev and I) are actively collaborating on this, making sure we don't duplicate any efforts. @ichernev can correct me or give more details if needed. We also have ongoing talks with other companies (eg. Wikipedia), so hopefully more and more libs will share our same dream.

Just let us know if you have any questions.

Twitter has a fairly comprehensive library for doing CLDR stuff in JavaScript. It may be worth looking at their work as we develop this further.

Check this out twitter/twitter-cldr-js#20

rxaviers commented Jan 27, 2014

@schmod, I am thrilled to let you know that some of your wishes have already come true.

Cldr.js does exactly what you are looking for. It's being used by jQuery Globalize (on its recent complete-rewrite), whose goal was to provide a set of tools that leverage the official CLDR JSON data, allow users to load as much or as little data as they need, avoid duplicating data if using multiple i18n libraries that leverage CLDR, run in browsers and node.js.

Moment.js has a migration work going on, and we (@ichernev and I) are actively collaborating on this, making sure we don't duplicate any efforts. @ichernev can correct me or give more details if needed. We also have ongoing talks with other companies (eg. Wikipedia), so hopefully more and more libs will share our same dream.

Just let us know if you have any questions.

Twitter has a fairly comprehensive library for doing CLDR stuff in JavaScript. It may be worth looking at their work as we develop this further.

Check this out twitter/twitter-cldr-js#20

@rxaviers

This comment has been minimized.

Show comment
Hide comment
@rxaviers

rxaviers Jan 27, 2014

Given that quite a few libraries and frameworks implement this functionality, it may be worth discussing how we can collaborate effectively, and share code. While it would be a great start for Moment to use CLDR's localization data, it'd be really awesome if we could split out all parsing/formatting functions into an independent module or library that could be consumed by other projects that only require this functionality.

I forgot to comment about the "split out all parsing/formatting functions into an independent module or library that could be consumed by other projects" that you've mentioned... The idea is that it will be provided by Globalize's Date module. Note that Globalize now provides modularized modules for consumption (date module, number module, etc). Also note that Globalize's implementation is based on Unicode standards for processing the CLDR data (eg. UTS TR35).

rxaviers commented Jan 27, 2014

Given that quite a few libraries and frameworks implement this functionality, it may be worth discussing how we can collaborate effectively, and share code. While it would be a great start for Moment to use CLDR's localization data, it'd be really awesome if we could split out all parsing/formatting functions into an independent module or library that could be consumed by other projects that only require this functionality.

I forgot to comment about the "split out all parsing/formatting functions into an independent module or library that could be consumed by other projects" that you've mentioned... The idea is that it will be provided by Globalize's Date module. Note that Globalize now provides modularized modules for consumption (date module, number module, etc). Also note that Globalize's implementation is based on Unicode standards for processing the CLDR data (eg. UTS TR35).

@masklinn

This comment has been minimized.

Show comment
Hide comment
@masklinn

masklinn Aug 27, 2014

coming from #315, wanted to note that

It would be awesome to set up a build script that can keep moment in sync with CLDR, although moment does have timeago and I didn't see anything equivalent in CLDR.

CLDR 24 (September 2013) added support for relative date/time fields: http://cldr.unicode.org/index/downloads/cldr-24#TOC-Enhanced-fields-structure

masklinn commented Aug 27, 2014

coming from #315, wanted to note that

It would be awesome to set up a build script that can keep moment in sync with CLDR, although moment does have timeago and I didn't see anything equivalent in CLDR.

CLDR 24 (September 2013) added support for relative date/time fields: http://cldr.unicode.org/index/downloads/cldr-24#TOC-Enhanced-fields-structure

@jzaefferer

This comment has been minimized.

Show comment
Hide comment
@jzaefferer

jzaefferer Oct 8, 2014

@ichernev or anyone else, is there any progress on actually implementing moment on top of cldr?

jzaefferer commented Oct 8, 2014

@ichernev or anyone else, is there any progress on actually implementing moment on top of cldr?

@ichernev

This comment has been minimized.

Show comment
Hide comment
@ichernev

ichernev Oct 9, 2014

Contributor

@jzaefferer I haven't looked at the cldr or globalize recently. The final decision was to make a thin wrapper around an actual library to work with moment, that would be separate than the current locale business.

Contributor

ichernev commented Oct 9, 2014

@jzaefferer I haven't looked at the cldr or globalize recently. The final decision was to make a thin wrapper around an actual library to work with moment, that would be separate than the current locale business.

@jzaefferer

This comment has been minimized.

Show comment
Hide comment
@jzaefferer

jzaefferer Oct 9, 2014

a thin wrapper around an actual library to work with moment

I don't understand this, could you explain that in a bit more detail? Maybe a usage example?

Anyway, do you have any plans to work on that (again)?

jzaefferer commented Oct 9, 2014

a thin wrapper around an actual library to work with moment

I don't understand this, could you explain that in a bit more detail? Maybe a usage example?

Anyway, do you have any plans to work on that (again)?

@Nate-Wilkins

This comment has been minimized.

Show comment
Hide comment
@Nate-Wilkins

Nate-Wilkins Jan 8, 2015

@ichernev @jzaefferer Does anyone know how far along this is?

If nobody has started work on this I might be able to take a look?

Nate-Wilkins commented Jan 8, 2015

@ichernev @jzaefferer Does anyone know how far along this is?

If nobody has started work on this I might be able to take a look?

@rxaviers

This comment has been minimized.

Show comment
Hide comment
@rxaviers

rxaviers Jan 8, 2015

@ichernev can give better details, but he started https://github.com/ichernev/moment-cldr

rxaviers commented Jan 8, 2015

@ichernev can give better details, but he started https://github.com/ichernev/moment-cldr

@Nate-Wilkins

This comment has been minimized.

Show comment
Hide comment
@Nate-Wilkins

Nate-Wilkins Jan 9, 2015

Great thanks! @rxaviers Do you know if the intent of this issue was to also combine Globalize's date formatter?

Nate-Wilkins commented Jan 9, 2015

Great thanks! @rxaviers Do you know if the intent of this issue was to also combine Globalize's date formatter?

@rxaviers

This comment has been minimized.

Show comment
Hide comment
@rxaviers

rxaviers Jan 9, 2015

Yes, the intent was not to duplicate efforts by using Globalize as an underlying CLDR layer provider (or more precisely, i18n library that leverage CLDR). But, again @ichernev can provide better details.

rxaviers commented Jan 9, 2015

Yes, the intent was not to duplicate efforts by using Globalize as an underlying CLDR layer provider (or more precisely, i18n library that leverage CLDR). But, again @ichernev can provide better details.

@Boycce

This comment has been minimized.

Show comment
Hide comment
@Boycce

Boycce Sep 5, 2015

Be awesome if moment took JSON like objects for parsing new locales in. eg. http://amsul.ca/pickadate.js/date/#translations
Any logic that a particular language needs, such as pluralisation would be detected by what is parsed into moment by conforming to CLDR naming conventions.

This will allow us to package our locales in our build processes from CLDR, and reuse the same locale data for every aspect of our software.

Boycce commented Sep 5, 2015

Be awesome if moment took JSON like objects for parsing new locales in. eg. http://amsul.ca/pickadate.js/date/#translations
Any logic that a particular language needs, such as pluralisation would be detected by what is parsed into moment by conforming to CLDR naming conventions.

This will allow us to package our locales in our build processes from CLDR, and reuse the same locale data for every aspect of our software.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment