Different output for native image and JVM applications

[sshark]

When the application runs under the JVM, it performs correctly. My images are moved to the organized/ directory. If I run it in its native image, my images are moved to invalid/ directory. I am using GraalVM CE 20.0.0 Java 11 for both in Windows and Linux.

[AlexITC]

Please try generating the native image locally (sbt nativeImage), which gets stored on target/native-image/my-photo-timeline.

Are you trying v1.0.2? because on a previous version, there was a similar bug.

[sshark]

Yes, I am using source tag v1.0.2 / master. What was the previous issue did you face?

[AlexITC]

Did you tried generating the native image locally to see if you can reproduce the issue? Which OS are you trying?

The previous issue was similar to what you report, moving everything to the invalid folder.

[sshark]

I tried: -

1. Building from source and run the native image in Windows
2. Building from source and run the native image in Linux
3. Running the native image I downloaded from GitHub

The results were the same. The images were moved to the invalid/ directory.

[AlexITC]

Interesting, by trying with other data, it seems I was able to reproduce the problem.

I'll work in a fix and keep you updated, thanks for reporting.

[AlexITC]

Ok, I found the issue, turns out that the metadata-extractor library used to detect the creation date on the files, uses reflexion, hence, those classes doesn't get into the native image.

So, there are two fixes:

1. Allow providing a debug flag, so that we can understand why something isn't working as expected.
2. Add the necessary classes from the previous library to the native image.

I'll work on the fixes tomorrow.

[sshark]

I submitted a PR for a configuration to include the reflection classes into the native image build. It works for me and I hope it is useful for you too.

[AlexITC]

I'll do a pre-release soon which should get better behaviour but still different, I have also included the option to print debug logs.

There is an issue I haven't been able to fix, which is preventing the native build to work with other files, in case you have any idea:

DEBUG: Failed to find metadata date from filename on /home/dell/Pictures/data/output/organized/2020/01-january/IMG_2198.JPG
java.util.MissingResourceException: Could not load any resource bundle by com.sun.org.apache.xerces.internal.impl.msg.SAXMessages
	at jdk.xml.internal.SecuritySupport.lambda$getResourceBundle$5(SecuritySupport.java:274)
	at java.security.AccessController.doPrivileged(AccessController.java:83)
	at jdk.xml.internal.SecuritySupport.getResourceBundle(SecuritySupport.java:267)
	at com.sun.org.apache.xerces.internal.util.SAXMessageFormatter.formatMessage(SAXMessageFormatter.java:59)
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.setFeature(DOMParser.java:462)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.setFeatures(DocumentBuilderImpl.java:245)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.<init>(DocumentBuilderImpl.java:231)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:74)
	at com.adobe.internal.xmp.impl.XMPMetaParser.parseInputSource(XMPMetaParser.java:322)
	at com.adobe.internal.xmp.impl.XMPMetaParser.parseXmlFromBytebuffer(XMPMetaParser.java:226)
	at com.adobe.internal.xmp.impl.XMPMetaParser.parseXml(XMPMetaParser.java:151)
	at com.adobe.internal.xmp.impl.XMPMetaParser.parse(XMPMetaParser.java:87)
	at com.adobe.internal.xmp.XMPMetaFactory.parseFromBuffer(XMPMetaFactory.java:169)
	at com.drew.metadata.xmp.XmpReader.extract(XmpReader.java:170)
	at com.drew.metadata.xmp.XmpReader.extract(XmpReader.java:150)
	at com.drew.metadata.xmp.XmpReader.extract(XmpReader.java:140)
	at com.drew.metadata.xmp.XmpReader.readJpegSegments(XmpReader.java:111)
	at com.drew.imaging.jpeg.JpegMetadataReader.processJpegSegmentData(JpegMetadataReader.java:134)
	at com.drew.imaging.jpeg.JpegMetadataReader.process(JpegMetadataReader.java:126)
	at com.drew.imaging.jpeg.JpegMetadataReader.readMetadata(JpegMetadataReader.java:77)
	at com.drew.imaging.jpeg.JpegMetadataReader.readMetadata(JpegMetadataReader.java:84)
	at com.drew.imaging.ImageMetadataReader.readMetadata(ImageMetadataReader.java:146)
	at com.drew.imaging.ImageMetadataReader.readMetadata(ImageMetadataReader.java:124)
	at com.drew.imaging.ImageMetadataReader.readMetadata(ImageMetadataReader.java:204)
	at net.wiringbits.myphototimeline.MetadataService.f$2(MetadataService.scala:39)
	at net.wiringbits.myphototimeline.MetadataService.getCreationDate(MetadataService.scala:53)
	at net.wiringbits.myphototimeline.FileOrganizerService.$anonfun$load$1(FileOrganizerService.scala:37)
	at scala.collection.immutable.ArraySeq.$anonfun$map$1(ArraySeq.scala:71)
	at scala.collection.immutable.ArraySeq.$anonfun$map$1$adapted(ArraySeq.scala:71)
	at scala.collection.immutable.ArraySeq$.tabulate(ArraySeq.scala:288)
	at scala.collection.immutable.ArraySeq$.tabulate(ArraySeq.scala:267)
	at scala.collection.ClassTagIterableFactory$AnyIterableDelegate.tabulate(Factory.scala:664)
	at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:71)
	at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35)
	at net.wiringbits.myphototimeline.FileOrganizerService.load(FileOrganizerService.scala:32)
	at net.wiringbits.myphototimeline.FileOrganizerTask.run(FileOrganizerTask.scala:37)
	at net.wiringbits.myphototimeline.CommandAppHelper$.run(CommandAppHelper.scala:111)
	at net.wiringbits.myphototimeline.Main$$anonfun$$lessinit$greater$1.apply(Main.scala:25)
	at net.wiringbits.myphototimeline.Main$$anonfun$$lessinit$greater$1.apply(Main.scala:24)
	at cats.SemigroupalArityFunctions.$anonfun$map4$1(SemigroupalArityFunctions.scala:60)
	at cats.data.Validated.ap(Validated.scala:182)
	at cats.data.ValidatedApplicative.ap(Validated.scala:527)
	at cats.data.ValidatedApplicative.ap(Validated.scala:520)
	at cats.ComposedApply.$anonfun$ap$2(Composed.scala:33)
	at cats.Monad.$anonfun$map$1(Monad.scala:16)
	at cats.instances.Function0Instances$$anon$4.$anonfun$flatMap$1(function.scala:75)
	at cats.instances.Function0Instances$$anon$4.$anonfun$flatMap$1(function.scala:75)
	at com.monovore.decline.Parser.evalResult(Parser.scala:30)
	at com.monovore.decline.Parser.consumeAll(Parser.scala:107)
	at com.monovore.decline.Parser.apply(Parser.scala:19)
	at com.monovore.decline.Command.parse(opts.scala:18)
	at com.monovore.decline.CommandApp.main(CommandApp.scala:52)
	at net.wiringbits.myphototimeline.Main.main(Main.scala)
`` 

[sshark]

If you can share the photo with me I can take a look

[AlexITC]

I found one, I find it weird because I have two photos that were taken consecutively, one which is processed properly, and this one which fails while exacting the metadata, from what I saw, the problem relates while parsing XMP related metadata which should help to fix the issue:

The actual photo

[sshark]

Just tried the photo. The application crashed by the runtime exception. I am looking for another photo metadata library which is not using reflection unless, of course, this is the only Java library available.

[AlexITC]

Can you print the stracktrace that crashed it? locally, I get the metadata parsing exception logged but the app continues looking on the remaining files.

Another library could be an alternative.

[sshark]

The stack trace I got was exactly the one you had earlier, java.util.MissingResourceException: Could not load any resource bundle by com.sun.org.apache.xerces.internal.impl.msg.SAXMessages. Let's see what I can resolve this

[AlexITC]

Given the discussion on the metadata-extractor library, it seems reflection isn't used on purpose, but most likely inherited from xmpcore, so, including the whole classes on com.adobe.internal.xmp could hopefully solve the issue, I'll likely try it at night.

[AlexITC]

Yesterday, I looked a bit into the error, unfortunately, no progress so far.

I tried including the whole xmpcore library, but it made no difference, then, looking a bit more into the error message, it looks like the problem may not be related to the reflection involved in such library, but a missing resource (I don't know which kind of resource yet).

It seems like the error will be a bit time consuming to debug, so, I don't expect to do much progress until the weekend, I will try compiling the library locally to add more debug logs, so that we understand what resource is being tried to load, and why,

If you have a better idea, please let me know.

[sshark]

I have a hunch that the native-image didn't include the SAXMessages.properties into the executable image. I am trying to get a plain class to work with properties files in the executable form. https://app.slack.com/client/TN37RDLPK/CNBFR78F9/thread/CNBFR78F9-1601398294.078500

[AlexITC]

You have posted a private link.

[sshark]

Perhaps this might help,
https://graalvm.slack.com/archives/CNBFR78F9/p1601398294078500

However, it does not matter much now as I have understood and fixed the problem

[AlexITC]

After merging your PR, my whole test data works on the native image, I'm closing the issue for now.

Thanks!