remove Python dependency

[stevengj]

Closes #1: calls the libgettext library (provided by Gettext_jll) directly rather than via Python.

Also closes #7: moves _"..." evaluation to runtime.

[inkydragon]

I found that libintl-8.dll on Windows exported the required functions.

• textdomain
• bindtextdomain
• gettext
• dgettext
• ngettext
• dngettext

But libgettextlib-0-21.dll does not.
Maybe we need to update the Gettext_jll.

libgettextlib-0-21.dll exports: https://gist.github.com/inkydragon/0208104e9d0844073adbf32282867591

Dump of file libintl-8.dll

>dumpbin /exports libintl-8.dll

File Type: DLL

  Section contains the following exports for libintl-8.dll

    00000000 characteristics
    60A6E265 time date stamp Fri May 21 06:27:49 2021
        0.00 version
           1 ordinal base
          96 number of functions
          96 number of names

    ordinal hint RVA      name

          1    0 00006150 DllMain
          2    1 0000F320 __imp_gl_get_setlocale_null_lock
          3    2 0000F340 __imp_libintl_version
          4    3 0000B1C0 __printf__
          5    4 000035F0 _nl_expand_alias
          6    5 00003F30 _nl_explode_name
          7    6 00001A50 _nl_find_domain
          8    7 000043D0 _nl_find_msg
          9    8 000065B0 _nl_language_preferences_default
         10    9 00001E10 _nl_load_domain
         11    A 000083C0 _nl_locale_name
         12    B 00006B60 _nl_locale_name_canonicalize
         13    C 000083A0 _nl_locale_name_default
         14    D 000082D0 _nl_locale_name_environ
         15    E 00006BD0 _nl_locale_name_from_win32_LANGID
         16    F 000082A0 _nl_locale_name_from_win32_LCID
         17   10 00008330 _nl_locale_name_posix
         18   11 000082C0 _nl_locale_name_thread
         19   12 000082B0 _nl_locale_name_thread_unsafe
         20   13 000084B0 _nl_log_untranslated
         21   14 00003840 _nl_make_l10nflist
         22   15 00018250 _nl_msg_cat_cntr
         23   16 00003DC0 _nl_normalize_codeset
         24   17 0000F160 _nl_state_lock
         25   18 0000BEF0 bind_textdomain_codeset
         26   19 0000BEE0 bindtextdomain
         27   1A 0000BE90 dcgettext
         28   1B 0000BEC0 dcngettext
         29   1C 0000BE80 dgettext
         30   1D 0000BEB0 dngettext
         31   1E 0000BE70 gettext
         32   1F 0000BCD0 gl_get_setlocale_null_lock
         33   20 0000C030 glwthread_mutex_destroy
         34   21 0000BF00 glwthread_mutex_init
         35   22 0000BF20 glwthread_mutex_lock
         36   23 0000BF90 glwthread_mutex_trylock
         37   24 0000C000 glwthread_mutex_unlock
         38   25 0000CA20 glwthread_once
         39   26 0000C9E0 glwthread_recmutex_destroy
         40   27 0000C800 glwthread_recmutex_init
         41   28 0000C830 glwthread_recmutex_lock
         42   29 0000C8E0 glwthread_recmutex_trylock
         43   2A 0000C980 glwthread_recmutex_unlock
         44   2B 0000C780 glwthread_rwlock_destroy
         45   2C 0000C170 glwthread_rwlock_init
         46   2D 0000C1D0 glwthread_rwlock_rdlock
         47   2E 0000C480 glwthread_rwlock_tryrdlock
         48   2F 0000C570 glwthread_rwlock_trywrlock
         49   30 0000C660 glwthread_rwlock_unlock
         50   31 0000C330 glwthread_rwlock_wrlock
         51   32 0000B3D0 libintl_asprintf
         52   33 000019D0 libintl_bind_textdomain_codeset
         53   34 00001970 libintl_bindtextdomain
         54   35 00001A00 libintl_dcgettext
         55   36 00004C50 libintl_dcigettext
         56   37 00005520 libintl_dcngettext
         57   38 00001A30 libintl_dgettext
         58   39 00005550 libintl_dngettext
         59   3A 0000B160 libintl_fprintf
         60   3B 00001A40 libintl_gettext
         61   3C 00005E30 libintl_gettext_extract_plural
         62   3D 000055A0 libintl_gettext_free_exp
         63   3E 00010620 libintl_gettext_germanic_plural
         64   3F 00005700 libintl_gettextparse
         65   40 00001DE0 libintl_hash_string
         66   41 00005570 libintl_ngettext
         67   42 0000F1C8 libintl_nl_current_default_domain
         68   43 00010180 libintl_nl_default_default_domain
         69   44 00010160 libintl_nl_default_dirname
         70   45 00018258 libintl_nl_domain_bindings
         71   46 00006200 libintl_relocate
         72   47 00006560 libintl_relocate2
         73   48 00006120 libintl_set_relocation_prefix
         74   49 0000B910 libintl_setlocale
         75   4A 0000B340 libintl_snprintf
         76   4B 0000B280 libintl_sprintf
         77   4C 00003750 libintl_textdomain
         78   4D 000086F0 libintl_vasnprintf
         79   4E 0000B360 libintl_vasprintf
         80   4F 0000F330 libintl_version
         81   50 0000B0E0 libintl_vfprintf
         82   51 0000B190 libintl_vprintf
         83   52 0000B2B0 libintl_vsnprintf
         84   53 0000B210 libintl_vsprintf
         85   54 000019A0 libintl_wbindtextdomain
         86   55 00005F10 locale_charset
         87   56 0000BEA0 ngettext
         88   57 0000BDF0 setlocale_null
         89   58 0000BD60 setlocale_null_r
         90   59 0000BED0 textdomain
         91   5A 0000E350 wcrtomb
         92   5B 0000E390 wcsrtombs
         93   5C 0000BE60 xmax
         94   5D 0000BE00 xsum
         95   5E 0000BE10 xsum3
         96   5F 0000BE30 xsum4

[stevengj]

Gettext_jll ships libintl as well as libgettext, but it currently only exports a variable for the path of the latter.

I'm confused about what libgettext is, actually. The gettext docs seem to recommend linking libintl.

(Would be good to get CI fixed in this package to use github actions, so that Windows is tested.)

[stevengj]

Hmm, it looks like libgettextlib maybe to the gettext in libc on systems where it is shipped with the system libc (e.g. GNU/Linux), and otherwise to libintl. However, on Windows this doesn't work because Windows DLLs can't depend on other DLLs.

I think that in Julia we probably never want to use the system gettext — we always want to use the GNU libintl one. JuliaPackaging/Yggdrasil#6950 should update Gettext_jll to install libintl unconditionally so that we can link to it here instead of to libgettextlib.

[nhz2]

JuliaPackaging/Yggdrasil#11038 builds libintl for all platforms.

However, I think a pure Julia version could have a much nicer API and implementation compared to libintl. Internally, libintl uses a lot of mutable global state, file system operations, and complex caching logic. Something more like the python Class Based API seems way simpler.

[stevengj]

I think the first step should just be to call libintl. If someone wants to rewrite it in pure Julia, that would be cool but is a big project, and we shouldn't wait for that before we can get non-Python-based i18n.

[nhz2]

The new GettextRuntime_jll package is registered now and provides GNU libintl

[stevengj]

I've updated this PR to use libintl from GettextRuntime_jll.

[stevengj]

Right now, @garrison and @waldyrious are the only members of the Julia-i18n org … could you guys invite some more maintainers, e.g. me and/or someone from JuliaLang like @viralshah? I'm worried about this org getting orphaned.

[waldyrious]

Sure, @stevengj, makes sense! There are actually two more members in the organization, but they have their membership set to private so they're not visible at the moment — namely, @nalimilan and @neomatrixcode (who IIRC was the org's creator). I'd urge them to change the visibility of their membership in the interests of openness, and to ensure the organization's public profile adequately reflects its membership.

I have sent invitations to @stevengj and @viralshah, both as owners.

[stevengj]

Thanks @waldyrious! Any thoughts on merging this PR?

I should probably rebase once #12 is merged, first, so that CI can run.

[nalimilan]

I have no idea how I ended up being a private member. I've just changed that. @stevengj you're also a private member so maybe that's the default in this org?

[codecov-commenter]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[stevengj]

CI is failing on Windows with:

ERROR: LoadError: could not load symbol "wbindtextdomain":
The specified procedure could not be found. 
Stacktrace:
 [1] bindtextdomain(domain::String, dir_name::String)
   @ Gettext D:\a\Gettext.jl\Gettext.jl\src\Gettext.jl:18

The wbindtextdomain function is supposed to be provided on Windows in order to return the dir_name as a wchar_t (UTF-16, which we can then convert to UTF-8 with transcode), but it doesn't seem to exist in the libintl provided by the JLL.

It's possible that this function is not needed, and bindtextdomain returns a UTF-8 pathname even on Windows? I'll have to check the source code, since the documentation isn't clear on this.

[stevengj]

Looking at the gettext source code, it looks like it calls mbstowcs to convert the provided dirname to UTF-16 on Windows if the ordinary bindtextdomain function is called, which should allow us to pass ordinary Julia-native UTF-8 strings, I think?

On the other hand, all of the Windows-specific API code is only compiled if #if defined _WIN32 && !defined __CYGWIN__ … if wbindtextdomain isn't compiled, maybe that isn't true when the JLL was compiled? But maybe if __CYGWIN__ is defined when the JLL was compiled, the ordinary UTF-8 filesystem API just works?

I should probably just change it to use bindtextdomain and then include a test with a Unicode directory name.

[waldyrious]

Thanks @waldyrious! Any thoughts on merging this PR?

I am all for reducing dependencies and setup for running software, but I'm afraid I am not familiar with this codebase, so I'd rather leave it to those who actually understand the problems at play to make a decision :)

a private member so maybe that's the default in this org?

I believe that's GitHub's default, unfortunately. If you know of a way to change the default, please do share, as I'd be interested in implementing that for other organizations I'm a member of!

[stevengj]

Hmm, I removed the call to wbindtextdomain and a test for unicode path names, and it is now failing on Windows.

Seems like we need to make sure that libintl is compiled with Windows support properly.

[stevengj]

wbindtextdomain is called libintl_wbindtextdomain in the shared library, apparently, so I added it back. The unicode pathname test is still failing on Windows, though 😕 .

[stevengj]

Progress: it's not a Unicode thing — Windows is failing even with an ascii pathname.

[stevengj]

I have a feeling that there's some Windows-specific way to set the language that we are not doing? It is giving:

Gettext.setlocale() = "English_United States.1252"

and ignoring the LANG environment variable.

Update: Yup, you have to call SetThreadLocale 😡

[stevengj]

Hooray, Windows is finally working (including Unicode pathnames).

Okay to merge?