[native-image] -H:IncludeResourceBundles only includes bundle for default locale

[bradnewman]

With native-image RC10, -H:IncludeResourceBundles includes only the bundle for the default locale at image build-time. This is useful only if just a single locale is needed for an application, and that locale is known at build-time - and that seems insufficient for most internationalization/localization use cases.

As an example, take the following Java program:

import java.util.Locale;
import java.util.ResourceBundle;

public class Main {
    private static String getMessage(final Locale locale) {
        return ResourceBundle.getBundle("message", locale).getString("msg");
    }
    public static void main(final String[] args) {
        System.out.println("user.language=" + System.getProperty("user.language"));
        System.out.println("Locale.getDefault() = " + Locale.getDefault());
        System.out.println("message in default locale = " + getMessage(Locale.getDefault()));
        System.out.println("message in English = " + getMessage(Locale.ENGLISH));
        System.out.println("message in Japanese = " + getMessage(Locale.JAPANESE));
    }
}

And let messages be a property file backed resource bundle with two files: messages_en.properties (msg=English message), and messages_ja.properties (msg=Japanese message). The example program is also available as a Gist.

Running the code under the JVM produces the expected results - ResourceBundle.getBundle(String, Locale) returns the appropriate bundle for the given locale, and the default locale is determined from the host OS or properties at run-time.

$ java -Duser.language=ja Main
user.language=ja
Locale.getDefault() = ja_US
message in default locale = Japanese message
message in English = English message
message in Japanese = Japanese message

On the other hand, when using native-image, -H:IncludeResourceBundles only includes the bundle for the locale that is the default at image build-time, and the default locale is baked into the image. The replacement of ResourceBundle.getBundle(String, Locale) ignores the locale argument, so the bundle corresponding to the default locale at build-time is returned for every getBundle() call.

For this example, the locale for native-image was en_US:

$ ./main -Duser.language=ja
user.language=ja
Locale.getDefault() = en_US
message in default locale = English message
message in English = English message
message in Japanese = English message

The default locale being baked into the image is irritating, but that could be worked around by writing application code that determines the correct locale at startup and calls Locale.setDefault(). But the current behavior of -H:IncludeResourceBundles defeats the purpose of using resource bundles in most cases - it's no better than hardcoding messages for a single locale.

At a minimum, I feel this should be clearly documented as a limitation where resource bundles are discussed (e.g. https://github.com/oracle/graal/blob/master/substratevm/RESOURCES.md).

But ideally, the example program would behave identically on the JVM and when compiled with native-image. I think that would mean that com.oracle.svm.core.jdk.LocalizationSupport would be enhanced to capture resource bundles for all locales (or at least, for a list of locales provided via an option at build-time - that would be sufficient for my needs). And SVM would ship a replacement for Locale.getDefault() that determines the correct locale at image run-time rather than image build-time.

[ludoch]

+1.
To add on this, I am quite confused about the difference between H:IncludeResourceBundles and a resource-config.json containing:

{
  "resources":[
    {"pattern":"...path_to_resource_file_localized_or_not"}
}

Can we express everything we put in H:IncludeResourceBundles in a json file? It does not see the case...

[h908714124]

Still broken in GraalVM Version 19.1.1 CE

[aesteve]

Hi and sorry to jump in.

This is a major blocker for us in order to use native images.

Pretty much every application we're designing is dealing with translations at some point.
Not including resource bundles for every language should at least be documented under the "GraalVM major restrictions" so that people don't happen to face this issue by pure accident.

Is it going to be supported at some point? Or is this a feature native image won't ever support (so that we know if that's a "no-go" for building native images for every of our applications).

Thank you.

[bradnewman]

Since this doesn't seem to be getting fixed soon, let me update it with the workarounds we're using. I don't know how portable or reliable it will prove to be:

1. Use -H:IncludeResources or -H:ResourceConfigurationFiles instead of -H:IncludeResourceBundles, so that GraalVM actually includes the message catalogs for all languages in the native image.
2. Use reflection to make accessible and call ResourceBundle.getBundleImpl(String, Locale, ClassLoader, Control) instead of any of the public ResourceBundle.getBundle() overloads. This bypasses GraalVM's broken substitution for the public getBundle() method, and allows us to call through to the standard JVM's logic, which appears to work just fine (at least for our needs) in a native image.
3. Create an implementation of Locale.initDefault() (simplified for our needs) that calls Locale.setDefault(). We call it at native image run time in order to set the correct default locale from user.language and other environment variables.

[maxum2610]

Alternatively you can use the following workaround using Features:

`@AutomaticFeature
public class LocalizationResourcesFeature implements Feature {
    
    @Override
    public void beforeAnalysis(BeforeAnalysisAccess access)  {
        
        // work around for the fact that com.oracle.svm.core.jdk.LocalizationSupport only uses the default locale.
        
        // replaces --initialize-at-build-time=features.MyLocalizationSupport
          ImageSingletons.lookup(RuntimeClassInitializationSupport.class).initializeAtBuildTime(LocalizationSupport.class, "BECAUSE I CAN!!!");
        
        // This code runs at image buildtime and is not reachable at runtime.
        LocalizationSupport bundles = new LocalizationSupport();
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.UK), Locale.UK);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.ENGLISH), Locale.ENGLISH);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.FRENCH), Locale.FRENCH);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.FRANCE), Locale.FRANCE);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", new Locale("nl", "BE")), new Locale("nl", "BE"));
        
        ImageSingletons.add(LocalizationSupport.class, bundles);
    }
    
    @Override
    public void afterRegistration(AfterRegistrationAccess access) {        
        /* This code runs during image generation => pre-initialize whatever you need */
        // avoid null pointer exception when loading at runtime (Graal issue 1645)
        DecimalFormatSymbols dfsymsUK = DecimalFormatSymbols.getInstance(Locale.UK);
        DecimalFormatSymbols dfsymsFR = DecimalFormatSymbols.getInstance(Locale.FRENCH);
        DecimalFormatSymbols dfsymsBE = DecimalFormatSymbols.getInstance(new Locale("nl", "BE"));
        Calendar calUk = Calendar.getInstance(Locale.UK);
        Calendar calFr = Calendar.getInstance(Locale.FRENCH);
        Calendar calBe = Calendar.getInstance(new Locale("nl", "BE"));
    }
    
    @Override
    public void afterCompilation(AfterCompilationAccess access) {
        LocalizationSupport support = ImageSingletons.lookup(LocalizationSupport.class);
        access.registerAsImmutable(support, LocalizationResourcesFeature::isImmutable);
    }
    
    private static boolean isImmutable(Object object) {
        if (object instanceof java.util.Locale) {
            /* These classes have a mutable hash code field. */
            return false;
        }
        if (object instanceof java.util.Map) {
            /* The maps have lazily initialized cache fields (see JavaUtilSubstitutions). */
            return false;
        }
        return true;
    }
}

public class LocalizationSupport
{
    protected final Map<String, ResourceBundle> bundles= new HashMap<>();
    
    public LocalizationSupport() {}
     
    private String getBundleLocalizedName(String bundleName, Locale locale) {
        String localizedName = new StringBuilder().append(bundleName)
        .append("_")
        //.append(locale.getCountry()) only if this is important
        //.append("_")
        .append(locale.getLanguage()).toString();
        
        System.out.println("LocalizationSupport:getBundleLocalizedName for localizedName=" + localizedName);
        return localizedName;
    }
    
    public ResourceBundle getResourceBundle(String bundleName, Locale locale)
    {
        System.out.println("LocalizationSupport:getResourceBundle for locale=" + locale);
        return (ResourceBundle)bundles.get(getBundleLocalizedName(bundleName, locale));
    }

    public LocalizationSupport addResourceBundle(ResourceBundle rb, Locale locale)
    {
        System.out.println("LocalizationSupport:addResourceBundle " + rb.getBaseBundleName() + " for locale=" + locale);
        bundles.put(getBundleLocalizedName(rb.getBaseBundleName(), locale), rb);
        return this;
    }
}`

... then retrieve the required bundle at runtime like so:

            ResourceBundle bd;
            if(ImageSingletons.contains(LocalizationSupport.class)){
                // only when compiled to native
                LocalizationSupport bundles = ImageSingletons.lookup(LocalizationSupport.class);
                bd = bundles.getResourceBundle(FULLY_QUALIFIED_BUNDLE_NAME, locale);
            }
            else {
            	// normal jvm run
                bd = ResourceBundle.getBundle(FULLY_QUALIFIED_BUNDLE_NAME, locale);
            }

Hope this helps.

[mipastgt]

I really don't understand why this important limitation does not get more attention.
Native-image should have an option like --include-locales as it is already used in the JDKs jlink tool. https://docs.oracle.com/en/java/javase/13/docs/specs/man/jlink.html jlink has to solve similar problems like native-image. So, it might be a good idea to have a look at their solutions.

[christianwimmer]

@mipastgt Unfortunately this is not an easy limitation to lift. It requires a good bit of internal refactoring how resource bundles are handled in Native Image. In the current approach, all the loading happens at image build time. When doing that for all or a larger number of locales, then the image size will probably increase too much. So we probably need to switch to an approach where the locale data remains in (more compact) resources.

[mipastgt]

Is there still no progress on this issue after nine months? For some applications this is a show stopper.

[vjovanov]

This feature is set for 21.1 and will be implemented by @d-kozak.

[gilles-duboscq]

This is still open on the 21.1 milestone, is this already resolved or should it be moved to another milestone?