Collaboration with Globalize on CLDR?

[jzaefferer]

This is not a code issue, but StackOverflow is inappropriate for this request and I didn't yet want to email committers individually.

I'm on the jQuery Team, and we're working on a project called Globalize. Globalize has a lot of overlap with Moment.js. One major difference is the source for locales. As far as I can tell, Moment builds on user-contributed translations, by now 63 in total. Globalize started by importing locale data from .Net, giving as about 350 "cultures" (naming is hard...). So far not too interesting.

We're currently in the process of moving to CLDR, using their database of locales as the new base for Globalize. This requires a lot of groundwork to make the data usable in JavaScript. We're also reporting back issues to them, like missing documentation.

I've had a bit of communication with Tim Wood in the past (see #315) about using CLDR. That was a year ago, since then it looks like the project is now standing on more shoulders, so hopefully adopting CLDR, along with some form of collaboration with the jQuery Team is something worth revisiting.

On our end, @rxaviers and @scottgonzalez are the other two people most involved. We're generally around on IRC (Freenode/#jqueryui-dev) for some ad-hoc discussions. I can also make introductions per email, if that is preferred over this issue tracker (my email address is in my github profile).

[icambron]

I'd love to get better CLDR support. A couple of ways we've looked at this are to create a plugin that packages that data (see #786), and also simply updating our language data to be manually consistent with it (there was going to be a pull request from @hagmandan, but I seem to have lost track of that thread).

From my perspective, I just don't know that much about CLDR; we've been pretty ad-hock about approach to language and I'm not even sure we're thinking about the whole subject in the right way. As Tim mentioned in that #315 thread, one of the hardest parts has been getting the grammar right, which we do through callbacks, and it's not obvious how the CLDR data would interact with that.

But I think it's fair to say we're more than open to collaborating on this. I'm pretty slammed now and I'm pretty sure @ichernev is too, but I'll try to make some time to get on IRC so we can just chat about it.

[hagmandan]

Hey all - the thread I started hadn't been lost, just slowed down due to equal slammage on my end. However, @Krinkle has reached out further interest on CLDR support as well. My team was also thinking of a method to somehow pull and parse data straight from CLDR's json files; either periodically and store them statically, or dynamically when a locale is requested (this has obvious concerns with performance and reliability).

Some initial thoughts on helping with CLDR standardization through the manual method:

1. all the language files should have the nominative ("stand-alone" from CLDR) and possessive ("format" from CLDR) available in each, regardless of if there are differences
2. this would apply to months, abbreviated months; weekdays and abbreviated weekdays
3. additionally, CLDR and moment.js's naming convention of patterns don't match up 100% - (compare
   http://cldr.unicode.org/translation/date-time with http://momentjs.com/docs/#/displaying/format/); while I understand that is a HUGE change, I feel it's something to consider heavily if CLDR is becoming the standard across the board.

My current branch for CLDR has a few changes in the code that aren't necessarily related to CLDR compliance (eg, unique to a project I'm working on: localization support for different formats instead of just 'LLLL', 'l', in case users want month-day or month-year instead of month-day-year) - I will work on separating these out this week so my CLDR branch will be more aligned with what it should be.

I'll try to jump onto IRC, however email or github notifications have a better chance of reaching me.

[hagmandan]

I would definitely be open to collaboration - there was an ongoing thread
for this that I've mentioned you in.

As I'm also slammed at work, availability may be limited at first, but
brainstorming some next steps and plan scope could be hugely advantageous
to us in the coming weeks (the holidays may yield some "free" time to work
on this!)

On Fri, Nov 15, 2013 at 10:06 AM, Timo Tijhof notifications@github.comwrote:

As an engineer working with the Wikimedia Foundationhttps://wikimediafoundation.org/(the non-profit organisation that powers Wikipedia), I'm interested in
exploring Moment.js as our library of choice for date/time parsing,
formatting etc. It would be included in the MediaWikihttps://www.mediawiki.org/software. At this time this is purely from my personal interest and
preference and not representative of the organisation in any way, we might
be looking into other libraries or develop our own if needed.

I personally really like Moment.js, but the main blocker at this point is
CLDR support and completeness (Wikipedia supports over 286 languages).

Would you be open to collaboration with us (I'd be willing to do most of
the work) in making this happen?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1241#issuecomment-28590120
.

[jzaefferer]

A quick update on what we discussed elsewhere: We had an IRC meeting with @ichernev. Since then he and @rxaviers have discussed more CLDR details.

A few things we discussed:

• Neither project has any interest spending more time on maintaining a custom locale database, we both want to be able to respond to issues with "go report it to CLDR"
• CLDR tools (like json converter) and docs are flawed, but trying to use them and reporting docs issues should help get those improved
• We can at least share the tool Rafael wrote for dealing with CLDR in JavaScript, which is independent of Globalize: https://github.com/rxaviers/cldr
• Maybe Moment can use the date parsing and formatting implementation from Globalize, along with implementing other features directly on top of CLDR

[schmod]

I'm strongly in favor of starting work on this. Having access to the CLDR language data would make it easier to add locale-dependent features in the future.

A few disjointed thoughts about our current state of affairs, and where we should go

• Twitter has a fairly comprehensive library for doing CLDR stuff in JavaScript. It may be worth looking at their work as we develop this further.
• Just as the package maintainers have little interest in maintaining a custom locale database, web developers have little desire to load separate locale files for multiple libraries. It would be fantastic if I could grab a single CLDR JSON file that all of my applications/libraries could consume. I don't want to have to load separate language packs for moment, globalize, angular, etc....
• Given that quite a few libraries and frameworks implement this functionality, it may be worth discussing how we can collaborate effectively, and share code. While it would be a great start for Moment to use CLDR's localization data, it'd be really awesome if we could split out all parsing/formatting functions into an independent module or library that could be consumed by other projects that only require this functionality.

[rxaviers]

@schmod, I am thrilled to let you know that some of your wishes have already come true.

Cldr.js does exactly what you are looking for. It's being used by jQuery Globalize (on its recent complete-rewrite), whose goal was to provide a set of tools that leverage the official CLDR JSON data, allow users to load as much or as little data as they need, avoid duplicating data if using multiple i18n libraries that leverage CLDR, run in browsers and node.js.

Moment.js has a migration work going on, and we (@ichernev and I) are actively collaborating on this, making sure we don't duplicate any efforts. @ichernev can correct me or give more details if needed. We also have ongoing talks with other companies (eg. Wikipedia), so hopefully more and more libs will share our same dream.

Just let us know if you have any questions.

Twitter has a fairly comprehensive library for doing CLDR stuff in JavaScript. It may be worth looking at their work as we develop this further.

Check this out twitter/twitter-cldr-js#20

[rxaviers]

Given that quite a few libraries and frameworks implement this functionality, it may be worth discussing how we can collaborate effectively, and share code. While it would be a great start for Moment to use CLDR's localization data, it'd be really awesome if we could split out all parsing/formatting functions into an independent module or library that could be consumed by other projects that only require this functionality.

I forgot to comment about the "split out all parsing/formatting functions into an independent module or library that could be consumed by other projects" that you've mentioned... The idea is that it will be provided by Globalize's Date module. Note that Globalize now provides modularized modules for consumption (date module, number module, etc). Also note that Globalize's implementation is based on Unicode standards for processing the CLDR data (eg. UTS TR35).

[masklinn]

coming from #315, wanted to note that

It would be awesome to set up a build script that can keep moment in sync with CLDR, although moment does have timeago and I didn't see anything equivalent in CLDR.

CLDR 24 (September 2013) added support for relative date/time fields: http://cldr.unicode.org/index/downloads/cldr-24#TOC-Enhanced-fields-structure

[jzaefferer]

@ichernev or anyone else, is there any progress on actually implementing moment on top of cldr?

[ichernev]

@jzaefferer I haven't looked at the cldr or globalize recently. The final decision was to make a thin wrapper around an actual library to work with moment, that would be separate than the current locale business.

[jzaefferer]

a thin wrapper around an actual library to work with moment

I don't understand this, could you explain that in a bit more detail? Maybe a usage example?

Anyway, do you have any plans to work on that (again)?

[Nate-Wilkins]

@ichernev @jzaefferer Does anyone know how far along this is?

If nobody has started work on this I might be able to take a look?

[rxaviers]

@ichernev can give better details, but he started https://github.com/ichernev/moment-cldr

[Nate-Wilkins]

Great thanks! @rxaviers Do you know if the intent of this issue was to also combine Globalize's date formatter?

[rxaviers]

Yes, the intent was not to duplicate efforts by using Globalize as an underlying CLDR layer provider (or more precisely, i18n library that leverage CLDR). But, again @ichernev can provide better details.

[boycce]

Be awesome if moment took JSON like objects for parsing new locales in. eg. http://amsul.ca/pickadate.js/date/#translations
Any logic that a particular language needs, such as pluralisation would be detected by what is parsed into moment by conforming to CLDR naming conventions.

This will allow us to package our locales in our build processes from CLDR, and reuse the same locale data for every aspect of our software.

[marwahaha]

it looks like Globalize has been out for some time: https://github.com/globalizejs/globalize

I'm not sure how Moment.js would collaborate at this time.

If people have ideas, they can post here or in a new issue.