Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[native-image] -H:IncludeResourceBundles only includes bundle for default locale #911

Closed
bradnewman opened this issue Jan 15, 2019 · 10 comments
Closed
Assignees
Labels
Milestone

Comments

@bradnewman
Copy link

@bradnewman bradnewman commented Jan 15, 2019

With native-image RC10, -H:IncludeResourceBundles includes only the bundle for the default locale at image build-time. This is useful only if just a single locale is needed for an application, and that locale is known at build-time - and that seems insufficient for most internationalization/localization use cases.

As an example, take the following Java program:

import java.util.Locale;
import java.util.ResourceBundle;

public class Main {
    private static String getMessage(final Locale locale) {
        return ResourceBundle.getBundle("message", locale).getString("msg");
    }
    public static void main(final String[] args) {
        System.out.println("user.language=" + System.getProperty("user.language"));
        System.out.println("Locale.getDefault() = " + Locale.getDefault());
        System.out.println("message in default locale = " + getMessage(Locale.getDefault()));
        System.out.println("message in English = " + getMessage(Locale.ENGLISH));
        System.out.println("message in Japanese = " + getMessage(Locale.JAPANESE));
    }
}

And let messages be a property file backed resource bundle with two files: messages_en.properties (msg=English message), and messages_ja.properties (msg=Japanese message). The example program is also available as a Gist.

Running the code under the JVM produces the expected results - ResourceBundle.getBundle(String, Locale) returns the appropriate bundle for the given locale, and the default locale is determined from the host OS or properties at run-time.

$ java -Duser.language=ja Main
user.language=ja
Locale.getDefault() = ja_US
message in default locale = Japanese message
message in English = English message
message in Japanese = Japanese message

On the other hand, when using native-image, -H:IncludeResourceBundles only includes the bundle for the locale that is the default at image build-time, and the default locale is baked into the image. The replacement of ResourceBundle.getBundle(String, Locale) ignores the locale argument, so the bundle corresponding to the default locale at build-time is returned for every getBundle() call.

For this example, the locale for native-image was en_US:

$ ./main -Duser.language=ja
user.language=ja
Locale.getDefault() = en_US
message in default locale = English message
message in English = English message
message in Japanese = English message

The default locale being baked into the image is irritating, but that could be worked around by writing application code that determines the correct locale at startup and calls Locale.setDefault(). But the current behavior of -H:IncludeResourceBundles defeats the purpose of using resource bundles in most cases - it's no better than hardcoding messages for a single locale.

At a minimum, I feel this should be clearly documented as a limitation where resource bundles are discussed (e.g. https://github.com/oracle/graal/blob/master/substratevm/RESOURCES.md).

But ideally, the example program would behave identically on the JVM and when compiled with native-image. I think that would mean that com.oracle.svm.core.jdk.LocalizationSupport would be enhanced to capture resource bundles for all locales (or at least, for a list of locales provided via an option at build-time - that would be sufficient for my needs). And SVM would ship a replacement for Locale.getDefault() that determines the correct locale at image run-time rather than image build-time.

@ludoch
Copy link

@ludoch ludoch commented May 16, 2019

+1.
To add on this, I am quite confused about the difference between H:IncludeResourceBundles and a resource-config.json containing:

{
  "resources":[
    {"pattern":"...path_to_resource_file_localized_or_not"}
}

Can we express everything we put in H:IncludeResourceBundles in a json file? It does not see the case...

@h908714124
Copy link

@h908714124 h908714124 commented Jul 21, 2019

Still broken in GraalVM Version 19.1.1 CE

@aesteve
Copy link

@aesteve aesteve commented Oct 15, 2019

Hi and sorry to jump in.

This is a major blocker for us in order to use native images.

Pretty much every application we're designing is dealing with translations at some point.
Not including resource bundles for every language should at least be documented under the "GraalVM major restrictions" so that people don't happen to face this issue by pure accident.

Is it going to be supported at some point? Or is this a feature native image won't ever support (so that we know if that's a "no-go" for building native images for every of our applications).

Thank you.

@bradnewman
Copy link
Author

@bradnewman bradnewman commented Oct 15, 2019

Since this doesn't seem to be getting fixed soon, let me update it with the workarounds we're using. I don't know how portable or reliable it will prove to be:

  1. Use -H:IncludeResources or -H:ResourceConfigurationFiles instead of -H:IncludeResourceBundles, so that GraalVM actually includes the message catalogs for all languages in the native image.
  2. Use reflection to make accessible and call ResourceBundle.getBundleImpl(String, Locale, ClassLoader, Control) instead of any of the public ResourceBundle.getBundle() overloads. This bypasses GraalVM's broken substitution for the public getBundle() method, and allows us to call through to the standard JVM's logic, which appears to work just fine (at least for our needs) in a native image.
  3. Create an implementation of Locale.initDefault() (simplified for our needs) that calls Locale.setDefault(). We call it at native image run time in order to set the correct default locale from user.language and other environment variables.

@maxum2610
Copy link

@maxum2610 maxum2610 commented Oct 17, 2019

Alternatively you can use the following workaround using Features:

`@AutomaticFeature
public class LocalizationResourcesFeature implements Feature {
    
    @Override
    public void beforeAnalysis(BeforeAnalysisAccess access)  {
        
        // work around for the fact that com.oracle.svm.core.jdk.LocalizationSupport only uses the default locale.
        
        // replaces --initialize-at-build-time=features.MyLocalizationSupport
          ImageSingletons.lookup(RuntimeClassInitializationSupport.class).initializeAtBuildTime(LocalizationSupport.class, "BECAUSE I CAN!!!");
        
        // This code runs at image buildtime and is not reachable at runtime.
        LocalizationSupport bundles = new LocalizationSupport();
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.UK), Locale.UK);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.ENGLISH), Locale.ENGLISH);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.FRENCH), Locale.FRENCH);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", Locale.FRANCE), Locale.FRANCE);
        bundles.addResourceBundle(ResourceBundle.getBundle("translations.Bundle", new Locale("nl", "BE")), new Locale("nl", "BE"));
        
        ImageSingletons.add(LocalizationSupport.class, bundles);
    }
    
    @Override
    public void afterRegistration(AfterRegistrationAccess access) {        
        /* This code runs during image generation => pre-initialize whatever you need */
        // avoid null pointer exception when loading at runtime (Graal issue 1645)
        DecimalFormatSymbols dfsymsUK = DecimalFormatSymbols.getInstance(Locale.UK);
        DecimalFormatSymbols dfsymsFR = DecimalFormatSymbols.getInstance(Locale.FRENCH);
        DecimalFormatSymbols dfsymsBE = DecimalFormatSymbols.getInstance(new Locale("nl", "BE"));
        Calendar calUk = Calendar.getInstance(Locale.UK);
        Calendar calFr = Calendar.getInstance(Locale.FRENCH);
        Calendar calBe = Calendar.getInstance(new Locale("nl", "BE"));
    }
    
    @Override
    public void afterCompilation(AfterCompilationAccess access) {
        LocalizationSupport support = ImageSingletons.lookup(LocalizationSupport.class);
        access.registerAsImmutable(support, LocalizationResourcesFeature::isImmutable);
    }
    
    private static boolean isImmutable(Object object) {
        if (object instanceof java.util.Locale) {
            /* These classes have a mutable hash code field. */
            return false;
        }
        if (object instanceof java.util.Map) {
            /* The maps have lazily initialized cache fields (see JavaUtilSubstitutions). */
            return false;
        }
        return true;
    }
}

public class LocalizationSupport
{
    protected final Map<String, ResourceBundle> bundles= new HashMap<>();
    
    public LocalizationSupport() {}
     
    private String getBundleLocalizedName(String bundleName, Locale locale) {
        String localizedName = new StringBuilder().append(bundleName)
        .append("_")
        //.append(locale.getCountry()) only if this is important
        //.append("_")
        .append(locale.getLanguage()).toString();
        
        System.out.println("LocalizationSupport:getBundleLocalizedName for localizedName=" + localizedName);
        return localizedName;
    }
    
    public ResourceBundle getResourceBundle(String bundleName, Locale locale)
    {
        System.out.println("LocalizationSupport:getResourceBundle for locale=" + locale);
        return (ResourceBundle)bundles.get(getBundleLocalizedName(bundleName, locale));
    }

    public LocalizationSupport addResourceBundle(ResourceBundle rb, Locale locale)
    {
        System.out.println("LocalizationSupport:addResourceBundle " + rb.getBaseBundleName() + " for locale=" + locale);
        bundles.put(getBundleLocalizedName(rb.getBaseBundleName(), locale), rb);
        return this;
    }
}`

... then retrieve the required bundle at runtime like so:

            ResourceBundle bd;
            if(ImageSingletons.contains(LocalizationSupport.class)){
                // only when compiled to native
                LocalizationSupport bundles = ImageSingletons.lookup(LocalizationSupport.class);
                bd = bundles.getResourceBundle(FULLY_QUALIFIED_BUNDLE_NAME, locale);
            }
            else {
            	// normal jvm run
                bd = ResourceBundle.getBundle(FULLY_QUALIFIED_BUNDLE_NAME, locale);
            }

Hope this helps.

@mipastgt
Copy link

@mipastgt mipastgt commented Dec 31, 2019

I really don't understand why this important limitation does not get more attention.
Native-image should have an option like --include-locales as it is already used in the JDKs jlink tool. https://docs.oracle.com/en/java/javase/13/docs/specs/man/jlink.html jlink has to solve similar problems like native-image. So, it might be a good idea to have a look at their solutions.

@christianwimmer
Copy link
Member

@christianwimmer christianwimmer commented Jan 7, 2020

@mipastgt Unfortunately this is not an easy limitation to lift. It requires a good bit of internal refactoring how resource bundles are handled in Native Image. In the current approach, all the loading happens at image build time. When doing that for all or a larger number of locales, then the image size will probably increase too much. So we probably need to switch to an approach where the locale data remains in (more compact) resources.

@eginez eginez added spring feature labels Mar 12, 2020
@eginez eginez added this to To do in Native Image via automation Mar 12, 2020
@eginez eginez self-assigned this May 1, 2020
@eginez eginez moved this from To do to In progress in Native Image May 5, 2020
@mipastgt
Copy link

@mipastgt mipastgt commented Sep 11, 2020

Is there still no progress on this issue after nine months? For some applications this is a show stopper.

@vjovanov
Copy link
Contributor

@vjovanov vjovanov commented Dec 15, 2020

This feature is set for 21.1 and will be implemented by @d-kozak.

murfffi added a commit to murfffi/zebra4j that referenced this issue Apr 5, 2021
The native build of the CLI is fully functional, except for oracle/graal#911.
Add graalvm_docker.ps1 for starting a GraalVM container on Windows and native-image configuration files created with

java -agentlib:native-image-agent=config-output-dir=./native -jar target/zebra4j-0.7-SNAPSHOT-shaded.jar demo

The CL also simplifies the implementation of AtHouse.TYPE#getAttributes and removes Demo from the public API.
@gilles-duboscq
Copy link
Member

@gilles-duboscq gilles-duboscq commented Jun 15, 2021

This is still open on the 21.1 milestone, is this already resolved or should it be moved to another milestone?

@christianwimmer christianwimmer moved this from In progress to Done in Native Image Jun 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Native Image
  
Done
Development

No branches or pull requests