Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CLDR data #826

Open
7 tasks
noviluni opened this issue Oct 30, 2020 · 1 comment
Open
7 tasks

Update CLDR data #826

noviluni opened this issue Oct 30, 2020 · 1 comment

Comments

@noviluni
Copy link
Collaborator

noviluni commented Oct 30, 2020

Once we merge this: #825

we will be able to upgrade the CLDR data easily. By doing it we will improve this library as we will be able to add support for more locales, fix old language bugs, etc.

I will try to explain how to do it.

  1. Increase the cldr_version number (in dateparser_scripts.utils.get_raw_data). We should go version by version to avoid too much files to check.
  2. Run python dateparser_scripts/get_cldr_data.py. This will download the new JSON files.
  3. Check the new files. If there are new locales supported (new files or new regions), we should add them to the docs and probably add tests. We can add some languages to the avoid_languages lists or remove some from it if the language is now fully supported.
  4. Run python dateparser_scripts/write_complete_data.py. This will create the new .py files by merging the json files with the yaml files.
  5. Run python dateparser_scripts/order_languages.py to sort them and update the languages_info.py
  6. Run the tests (tox).
  7. If tests fail it could be because some old valid words have been removed from the new files. In this case, we can change/remove the test or add the old words to the yaml files.
  8. Run dateparser_scripts/update_supported_languages_and_locales.py to update the supported locales in the docs.

Next versions to update:

  • 32.0.0 (Nov 2017)
  • 33.0.0 (Mar 2018)
  • 34.0.0 (Oct 2018)
  • 35.0.0 (Mar 2019)
  • 35.1.0 (Apr 2019)
  • 36.0.0 (Oct 2019)
  • 37.0.0 (Oct 2020)
@noviluni noviluni added this to the 1.1.0 milestone Oct 30, 2020
@noviluni
Copy link
Collaborator Author

I can handle the first update to give an example of how to do it. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant