Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8286122: [macos]: App bundle cannot upload to Mac App Store due to info.plist embedded in java exe #8666

Closed
wants to merge 2 commits into from

Conversation

sashamatveev
Copy link
Member

@sashamatveev sashamatveev commented May 11, 2022

  • It is not possible to support native JDK commands such as "java" inside Mac App Store bundles due to embedded info.plist. Workarounds suggested in JDK-8286122 does not seems to be visible.
  • With proposed fix we will enforce "--strip-native-commands" option for jlink, so native JDK commands are not included when generating Mac App Store bundles.
  • Custom runtime provided via --runtime-image should not contain native commands as well, otherwise jpackage will throw error.
  • Added two tests to validate fix.

Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed (1 review required, with at least 1 reviewer)

Issue

  • JDK-8286122: [macos]: App bundle cannot upload to Mac App Store due to info.plist embedded in java exe

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/8666/head:pull/8666
$ git checkout pull/8666

Update a local copy of the PR:
$ git checkout pull/8666
$ git pull https://git.openjdk.java.net/jdk pull/8666/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 8666

View PR using the GUI difftool:
$ git pr show -t 8666

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/8666.diff

@bridgekeeper
Copy link

bridgekeeper bot commented May 11, 2022

👋 Welcome back almatvee! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr label May 11, 2022
@openjdk
Copy link

openjdk bot commented May 11, 2022

@sashamatveev The following label will be automatically applied to this pull request:

  • core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the core-libs label May 11, 2022
@mlbridge
Copy link

mlbridge bot commented May 11, 2022

Webrevs

@Test
public static void test() throws Exception {
JPackageCommand cmd = JPackageCommand.helloAppImage();
cmd.addArguments("--mac-app-store", "--runtime-image", getRuntimeImage(true));

cmd.executeAndAssertHelloAppImageCreated();
}

@Test
public static void test2() throws Exception {
JPackageCommand cmd = JPackageCommand.helloAppImage();
cmd.addArguments("--mac-app-store", "--runtime-image", getRuntimeImage(false));

cmd.execute(1);
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Test
@Parameter("true")
@Parameter("false")
public static void test(boolean stripNativeCommands) throws Exception {
    JPackageCommand cmd = JPackageCommand.helloAppImage();
    cmd.addArguments("--mac-app-store", "--runtime-image", getRuntimeImage(stripNativeCommands));

    if (stripNativeCommands) {
        cmd.executeAndAssertHelloAppImageCreated();
    } else {
        cmd.execute(1);
    }
}

Copy link
Member Author

@sashamatveev sashamatveev May 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

public class MacAppStoreJLinkOptionsTest {

@Test
public static void test() throws Exception {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd give some more descriptive names to test functions than test and test2. Something like testWithStripNativeCommands and testWithoutStripNativeCommands maybe?

Copy link
Member Author

@sashamatveev sashamatveev May 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@mlbridge
Copy link

mlbridge bot commented May 11, 2022

Mailing list message from Michael Hall on core-libs-dev:

Is this restricted somehow to Mac App Store applications?

Is a warning issued as stripping native commands may break application functionality?

Is it not possible for the user to provide their own Info.plist with a different bundle identifier that doesn?t collide?

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?

@mlbridge
Copy link

mlbridge bot commented May 11, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 11, 2022, at 5:17 PM, Michael Hall <mik3hall at gmail.com> wrote:

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?

Or just change it in the current build so it doesn?t collide? With the application or whatever one it is colliding with.

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Alexander Matveev on core-libs-dev:

Hi Michael,

On May 11, 2022, at 3:17 PM, Michael Hall <mik3hall at gmail.com> wrote:

Is this restricted somehow to Mac App Store applications?

Yes, helper tools (in our case JDK native commands) in Mac App Store applications cannot use same bundle ID as another application. Since we have bundle ID embedded in native commands multiple applications cannot use it without updating each native command with unique bundle ID and since they embedded inside executable I do not see how it would be possible to change during packaging.

Is a warning issued as stripping native commands may break application functionality?

No, error will be issues and jpackage will exit with error if ?mac-app-store arguments is used and provided runtime has native tools or user did not specified --strip-native-commands to jlink.

Is it not possible for the user to provide their own Info.plist with a different bundle identifier that doesn?t collide?

Not possible due to native commands have embedded Info.plist.

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?

Yes, it should be possible and in theory such JDK with native commands can be used. However, current fix will not allow such JDK build, since we checking for presence of ?bin? folder and not if ID is actually unique. To work around this limitation user can package without native commands first, then add native commands and re-sign application.

Or just change it in the current build so it doesn?t collide? With the application or whatever one it is colliding with.

Not possible, since ID should be unique per application.

Thanks,
Alexander

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Michael Hall on core-libs-dev:

Alexander

On May 11, 2022, at 11:38 PM, Alexander Matveev <alexander.matveev at oracle.com> wrote:

Hi Michael,

On May 11, 2022, at 3:17 PM, Michael Hall <mik3hall at gmail.com> wrote:

Is this restricted somehow to Mac App Store applications?
Yes, helper tools (in our case JDK native commands) in Mac App Store applications cannot use same bundle ID as another application. Since we have bundle ID embedded in native commands multiple applications cannot use it without updating each native command with unique bundle ID and since they embedded inside executable I do not see how it would be possible to change during packaging.

Possibly not during packaging but during the jdk build of the native commands? Somehow the Info.plist must be getting embedded and a bundle id provided. I?m not sure where or how. But one way to get uniqueness if the id is provided to embed the Info.plist in the native command executable compile/link would be to include the command as part of the identifier. Something like CFBundleID = java.native.cmd or CFBundleID = javac.native.cmd.

Is a warning issued as stripping native commands may break application functionality?
No, error will be issues and jpackage will exit with error if ?mac-app-store arguments is used and provided runtime has native tools or user did not specified --strip-native-commands to jlink.

A warning might be sufficient if the commands are included accidentally by the user unintentionally omitting the ?strip-native-commands with the jlink options. Then just force the ?strip-native-commands option. It might be the application will just lose some non-critical functionality even if they meant to have the native command included.

Is it not possible for the user to provide their own Info.plist with a different bundle identifier that doesn?t collide?
Not possible due to native commands have embedded Info.plist.

Maybe I?m misunderstanding the conflict. Are the commands conflicting with the application level Info.plist or with each other? I believe jpackage usually has a mechanism where a substitute Info.plist can be used if the conflict is with the application level. If multiple commands are conflicting with each other this won?t work.

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?
Yes, it should be possible and in theory such JDK with native commands can be used. However, current fix will not allow such JDK build, since we checking for presence of ?bin? folder and not if ID is actually unique. To work around this limitation user can package without native commands first, then add native commands and re-sign application.

Signing is currently an integrated part of the process. I thought about making an enhancement request to have it possible to do it stepped. First build the application, then allow the user to post-process that. Maybe run a tool/script to modify the application level Info.plist file. Then allow a second final step that does the signing. I don?t remember if I actually made that enhancement request. But currently there is no way with jpackage to standalone sign the application is there? I think this is somewhat involved with the application bundle being iterated and for one thing each jar being separately code signed? I think it took a release or two of the command to get this correct. So I doubt Xcode could manage it. Is there a way to do the signing standalone with jpackage that I am not aware of?

If I didn?t do an enhancement request on this I could, if you think jpackage could manage it?

Or just change it in the current build so it doesn?t collide? With the application or whatever one it is colliding with.
Not possible, since ID should be unique per application.

Possibly you are misunderstanding me here I think you already indicated the could be done ?in theory?. It still seems possibly the correct ?fix?. Change the build so uniqueness is not an issue.

Thanks,
Alexander

Thanks,
Mike

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Magnus Ihse Bursie on core-libs-dev:

(cc:ing build-dev.)

On 2022-05-12 00:17, Michael Hall wrote:

Is this restricted somehow to Mac App Store applications?

Is a warning issued as stripping native commands may break application functionality?

Is it not possible for the user to provide their own Info.plist with a different bundle identifier that doesn?t collide?

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?

This problem seem to come about as a clash between the the OpenJDK
packages themselves being submitted to Apple, as well as a
developer-specific jpackage containing JDK binaries, such as java.

The well-written response from Apple tech support in the original web
bug is very instructive. I'll quote it in its entirety:

----

I?ve dealt with this message before, and it has a long and interesting
history. The Mac App Store prevents two developers from submitting
executables with the same bundle identifier. This is an important
security check: We don?t want app A impersonating app B.

This check applies to all executables, not just the app?s main
executable. Historically the Mac App Store ingestion process had bugs
that caused it to apply to other types of code, like frameworks and
bundles. When I first saw this incident I was worried that these bugs
had come back.

However, that?s not the case. Let?s look at the main executables in
your app:

% find APP_NAME.app -type f -print0 | xargs -0 file | grep
"Mach-O.*executable"
APP_NAME.app/Contents/MacOS/APP_NAME: Mach-O 64-bit executable x86_64
APP_NAME.app/Contents/runtime/Contents/Home/bin/java: Mach-O 64-bit
executable x86_64
APP_NAME.app/Contents/runtime/Contents/Home/bin/keytool: Mach-O 64-bit
executable x86_64
APP_NAME.app/Contents/runtime/Contents/Home/lib/jspawnhelper: Mach-O
64-bit executable x86_64

Based on the error message it?s obvious we need to focus on the `java`
executable. However, it?s placed in a location that doesn?t have a
corresponding `Info.plist` file:

% find APP_NAME.app -name "Info.plist"
APP_NAME.app/Contents/Info.plist
APP_NAME.app/Contents/runtime/Contents/Info.plist

The first file here applies to your entire app and the second file
applies to the Java runtime package as a whole. Moreover, neither of
these have a bundle ID that matches the error message:

% plutil -extract CFBundleIdentifier raw -o -
"APP_NAME.app/Contents/Info.plist"
UNIQUE.BUNDLE.ID
% plutil -extract CFBundleIdentifier raw -o -
"APP_NAME.app/Contents/runtime/Contents/Info.plist"
com.oracle.java.com.UNIQUE.BUNDLE.ID

So where is this bundle ID coming from?

* * *

Some further spelunking reveals the issue. Consider this:

% otool -s __TEXT __info_plist -v
APP_NAME.app/Contents/runtime/Contents/Home/bin/java
?
<dict>
<key>CFBundleIdentifier</key>
<string>net.java.openjdk.java</string>
?
</dict>
</plist>

This is an obscure corner case in the macOS bundle story: A standalone
tool, like `java`, which isn?t in a bundle structure, and thus can?t
have a standalone `Info.plist` file, can put its information property
list in a `__TEXT` / `__info_plist` section within its executable. And
it seems that the folks who created your Java runtime did just that.

Given that, the Mac App Store error is valid: You are trying to submit
an executable whose bundle ID matches some existing app.

To get around this check you?ll need to give `java` a new bundle ID.
That?s not easy to do. This `__TEXT` / `__info_plist` section is set
up by a linker option (see `-sectcreate` in <x-man-page://1/ld&gt;)
<x-man-page://1/ld>)> and there?s no good way to modify it after the
fact [1].

At this point my advice is that you escalate this with your Java
runtime vendor. I suspect that they?ve added this information property
list to get around a TCC check ? the only other interesting property
in there is `NSMicrophoneUsageDescription` ? and they probably didn?t
notice this Mac App Store submission issue. There?s a couple of ways
they could resolve this [2] but it?s really their issue to resolve.

[1] And by ?good? I mean ?Using a standard tool supplied by Apple.?
The Mach-O file format is reasonably well documented and thus you
could build a custom tool to do that, but even that?s not easy. There
are a couple of problems:

* The new information property list may be larger than the previous one.

* The `__info_plist` section can appear anywhere in the `__TEXT` segment.

If you increase the size of the section then subsequent sections need
to move up to accommodate it and, depending on which sections you have
to move, that might require other cross-section links to be fixed up.
Yergh!

[2] The ones that spring to mind are:

* Package the `java` executable in a standard bundle, replacing
`runtime/Contents/Home/bin/java` with a symlink to that.

* Add an `__info_plist` section with a bunch of padding and then build
a tool to update the bundle ID in that section, taking advantage of
that padding to avoid the need to move subsequent sections in the
`__TEXT` segment.

----

So maybe a better, or at least alternative, way of dealing with this
issue is changing how the OpenJDK binaries are built.

I will need to do some reading up on the macOS bundle format to fully
grok what our options are here.

/Magnus

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 12, 2022, at 4:42 AM, Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com> wrote:

Some further spelunking reveals the issue. Consider this:

% otool -s __TEXT __info_plist -v APP_NAME.app/Contents/runtime/Contents/Home/bin/java
?
<dict>
<key>CFBundleIdentifier</key>
<string>net.java.openjdk.java</string>
?
</dict>
</plist>

This is an obscure corner case in the macOS bundle story: A standalone tool, like `java`, which isn?t in a bundle structure, and thus can?t have a standalone `Info.plist` file, can put its information property list in a `__TEXT` / `__info_plist` section within its executable. And it seems that the folks who created your Java runtime did just that.

Given that, the Mac App Store error is valid: You are trying to submit an executable whose bundle ID matches some existing app.

To get around this check you?ll need to give `java` a new bundle ID. That?s not easy to do. This `__TEXT` / `__info_plist` section is set up by a linker option (see `-sectcreate` in <x-man-page://1/ld&gt;) <x-man-page://1/ld%3E)> and there?s no good way to modify it after the fact [1].

I had read this but possibly didn?t grok the issue myself. If I understand correctly now the conflict isn?t within the application but across applications to some other application that has already been added to the Mac App Store that included the commands with the given CFBundleIdentifier. A solution like including a bundle identifier something like net.java.openjdk.MYAPP.java would be possible at packaging time but not at build time.
To fix this at build time you would need to generate a name unique to each installed jdk. Including release net.java.openjdk.JDK_RELEASE.java might avoid some conflicts but would still be open to conflict for two applications at the same release. So it can?t be addressed ?before the fact? either. The only thing I am currently thinking of that might work would be include a replaceable part in the identifier. So something like net.java.openjdk.java.XXXXXXXXXXXXXXXXXXXXXX
Where jpackage could include something to change the XXXXX?. to a unique application name. If you don?t change the string size you could probably avoid some of the resizing issues Apple DTS mentions. Whether there is a standard Apple tool to do this I don?t know.

@magicus
Copy link
Member

magicus commented May 12, 2022

/label build

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Magnus Ihse Bursie on core-libs-dev:

On 2022-05-12 13:17, Michael Hall wrote:

I had read this but possibly didn?t grok the issue myself. If I
understand correctly now the conflict isn?t within the application but
across applications to some other application that has already been
added to the Mac App Store that included the commands with the given
CFBundleIdentifier.

Yes, that is indeed how I also interpret this. Presumably, the very
first developer to submit a jpackaged application to App Store (from
what I can tell now OpenJDK itself is not present there, which I
mistakenly believed before) got everything working without troubles, but
blocked all other developers from submitting their apps.

A solution like including a bundle identifier something like
net.java.openjdk.MYAPP.java would be possible at packaging time but
not at build time.
To fix this at build time you would need to generate a name unique to
each installed jdk. Including release
net.java.openjdk.JDK_RELEASE.java might avoid some conflicts but would
still be open to conflict for two applications at the same release. So
it can?t be addressed ?before the fact? either. The only thing I am
currently thinking of that might work would be include a replaceable
part in the identifier. So something like
net.java.openjdk.java.XXXXXXXXXXXXXXXXXXXXXX
Where jpackage could include something to change the XXXXX?. to a
unique application name. If you don?t change the string size you could
probably avoid some of the resizing issues Apple DTS mentions. Whether
there is a standard Apple tool to do this I don?t know.

As you say, we're a bit in a bind since the java executable needs to be
created when the JDK is built, but the bundle ID needs to be determined
when jpackage is run (and a suitable, per-application ID can be
created), and there is no standard tooling to update the bundle ID after
build time.

I believe what you are describing is exactly solution #2 suggested by
the Apple engineer. I don't think that would be horribly difficult to
achieve, though. Sure, it's a bit of a hack, but not the ugliest I've
seen in my days. If we go down this route, I suppose we do something
like this:

1) When building the JDK, we create an additional copy of the java
executable, and store it with the jdk.jpackage resources. Let's call it
java.template, for the sake of it. This is identical to the real
bin/java except for the fact that it contains a different bundle ID,
with a large enough padding field, like XXXXX...? This way, we don't
have to mess around with the real java executable for the JDK.

2) At jpackage time, this java.template file is installed instead of
bin/java, and the padding field is replaced by a unique value. The java
executable is small (33kB on macOS, currently) so a simple search
through the binary field for the pattern is likely to work alright. As
long as there are no checksums being broken, this should be straightforward.

My primary suggestion would to be to use an UUID for the unique ID. They
are of fixed length, are for all intents and purposes unique and you can
conjure them up from your hat. (An alternative is that the user needs to
specify a unique ID, but that is probably a less ideal solution.)
Presumably, we can have some kind of prefix like "org.openjdk.jpackage."
before the UUID to make them a bit understandable, if someone where to
inspect the package metadata.

This seems like a fully workable solution to me. However, I'd really
like to understand option #1 better, which was: "Package the `java`
executable in a standard bundle, replacing
`runtime/Contents/Home/bin/java` with a symlink to that."

I don't know what a "standard bundle" is, or how you would go around to
package the java executable in one. But on the surface, it sounds much
nicer than binary editing.

I also don't understand if that means that the standard bundle needs to
be created at jpackage time, so it gives us the chance to set a proper
ID, or if the standard bundle can be created at build time, and then the
existence of this bundle just makes Apple avoid the bundle ID collision
checks. Or if I'm just misunderstanding it all...

/Magnus

@openjdk openjdk bot added the build label May 12, 2022
@openjdk
Copy link

openjdk bot commented May 12, 2022

@magicus
The build label was successfully added.

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Michael Hall on core-libs-dev:

My primary suggestion would to be to use an UUID for the unique ID. They are of fixed length, are for all intents and purposes unique and you can conjure them up from your hat. (An alternative is that the user needs to specify a unique ID, but that is probably a less ideal solution.) Presumably, we can have some kind of prefix like "org.openjdk.jpackage." before the UUID to make them a bit understandable, if someone where to inspect the package metadata.e

I was thinking jpackage would change the XXX to app name but a fixed size unique field would probably be better.

This seems like a fully workable solution to me. However, I'd really like to understand option #1 better, which was: "Package the `java` executable in a standard bundle, replacing `runtime/Contents/Home/bin/java` with a symlink to that."

I don't know what a "standard bundle" is, or how you would go around to package the java executable in one. But on the surface, it sounds much nicer than binary editing.

I also don't understand if that means that the standard bundle needs to be created at jpackage time, so it gives us the chance to set a proper ID, or if the standard bundle can be created at build time, and then the existence of this bundle just makes Apple avoid the bundle ID collision checks. Or if I'm just misunderstanding it all...

/Magnus

I may again not understanding but I was thinking they were talking about something like symlinks to a machine installed JDK and this seemed to me to defeat some of the purpose of embedding the jdk. But he could be thinking else. Something external to the application anyhow it seemed.

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from erik.joelsson at oracle.com on core-libs-dev:

On 2022-05-12 04:58, Magnus Ihse Bursie wrote:

On 2022-05-12 13:17, Michael Hall wrote:

A solution like including a bundle identifier something like
net.java.openjdk.MYAPP.java would be possible at packaging time but
not at build time.
To fix this at build time you would need to generate a name unique to
each installed jdk. Including release
net.java.openjdk.JDK_RELEASE.java might avoid some conflicts but
would still be open to conflict for two applications at the same
release. So it can?t be addressed ?before the fact? either. The only
thing I am currently thinking of that might work would be include a
replaceable part in the identifier. So something like
net.java.openjdk.java.XXXXXXXXXXXXXXXXXXXXXX
Where jpackage could include something to change the XXXXX?. to a
unique application name. If you don?t change the string size you
could probably avoid some of the resizing issues Apple DTS mentions.
Whether there is a standard Apple tool to do this I don?t know.

As you say, we're a bit in a bind since the java executable needs to
be created when the JDK is built, but the bundle ID needs to be
determined when jpackage is run (and a suitable, per-application ID
can be created), and there is no standard tooling to update the bundle
ID after build time.

I believe what you are describing is exactly solution #2 suggested by
the Apple engineer. I don't think that would be horribly difficult to
achieve, though. Sure, it's a bit of a hack, but not the ugliest I've
seen in my days. If we go down this route, I suppose we do something
like this:

1) When building the JDK, we create an additional copy of the java
executable, and store it with the jdk.jpackage resources. Let's call
it java.template, for the sake of it. This is identical to the real
bin/java except for the fact that it contains a different bundle ID,
with a large enough padding field, like XXXXX...? This way, we don't
have to mess around with the real java executable for the JDK.

Aren't we embedding this bundle ID in every launcher, so you would need
a <launcher>.template for each possible launcher that could be included
in an app?

What I think we need to do is first evaluate if we actually need to
embed this bundle ID in the executables at all, or rather, what would
the consequences be if we didn't. My understanding is that we only do
this to be able to run them outside of a bundle directory structure, but
this would need some pretty thorough testing of course. If we are to
generate a special set of launchers for jpackage, maybe all we need to
do is not embed any bundle ID in them, as they are meant to only be used
within a jpackaged app, so they should be covered by Info.plist files
anyway.

/Erik

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Scott Palmer on core-libs-dev:

On May 12, 2022, at 8:10 AM, Michael Hall <mik3hall at gmail.com> wrote:

?

My primary suggestion would to be to use an UUID for the unique ID. They are of fixed length, are for all intents and purposes unique and you can conjure them up from your hat. (An alternative is that the user needs to specify a unique ID, but that is probably a less ideal solution.) Presumably, we can have some kind of prefix like "org.openjdk.jpackage." before the UUID to make them a bit understandable, if someone where to inspect the package metadata.e

I was thinking jpackage would change the XXX to app name but a fixed size unique field would probably be better.

This seems like a fully workable solution to me. However, I'd really like to understand option #1 better, which was: "Package the `java` executable in a standard bundle, replacing `runtime/Contents/Home/bin/java` with a symlink to that."

I don't know what a "standard bundle" is, or how you would go around to package the java executable in one. But on the surface, it sounds much nicer than binary editing.

I also don't understand if that means that the standard bundle needs to be created at jpackage time, so it gives us the chance to set a proper ID, or if the standard bundle can be created at build time, and then the existence of this bundle just makes Apple avoid the bundle ID collision checks. Or if I'm just misunderstanding it all...

/Magnus

I may again not understanding but I was thinking they were talking about something like symlinks to a machine installed JDK and this seemed to me to defeat some of the purpose of embedding the jdk. But he could be thinking else. Something external to the application anyhow it seemed.

I thought they meant something like the embedded JDK would be like a framework bundle. Since the framework is expected to be the same in multiple apps it would be excluded from the duplicate id check. (I think that is related to the older bug that the Apple guy thought might have come back.)

Scott

@openjdk
Copy link

openjdk bot commented May 12, 2022

@sashamatveev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8286122: [macos]: App bundle cannot upload to Mac App Store due to info.plist embedded in java exe

Reviewed-by: asemenyuk, kcr

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 153 new commits pushed to the master branch:

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label May 12, 2022
@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Magnus Ihse Bursie on build-dev:

Aren't we embedding this bundle ID in every launcher, so you would
need a <launcher>.template for each possible launcher that could be
included in an app?

I naively assumed that only the java launcher was needed, since this is
about packaging a Java app, so all you'd need is an entry to your main
class. Is this not the case?

What I think we need to do is first evaluate if we actually need to
embed this bundle ID in the executables at all, or rather, what would
the consequences be if we didn't. My understanding is that we only do
this to be able to run them outside of a bundle directory structure,
but this would need some pretty thorough testing of course. If we are
to generate a special set of launchers for jpackage, maybe all we need
to do is not embed any bundle ID in them, as they are meant to only be
used within a jpackaged app, so they should be covered by Info.plist
files anyway.

What you say sounds good, although I feel I only partially understand
it. :-)

I assume the important point here is that the app get the
NSMicrophoneUsageDescription property, and afaict this can be provided
by the Info.plist file for the entire application, as you say. And
possible the problem here is that we embed metadata in the java
executable at the same time.

There seems to be a lot of guessing here. :-) I assume we either need to
read up on how this works (which might be difficult if this is seem as a
badly documented corner case even by Apple tech support), or test some
alternatives, or perhaps both.

That solution make take some time to get correct, so the jpackage team
needs to decide if they want to go with the workaround in this PR in the
meantime.

/Magnus

/Erik

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Michael Hall on build-dev:

What you say sounds good, although I feel I only partially understand it. :-)

If it?s correct that normal applications don?t need the executable included Info.plist and jpackage can somehow get versions that don?t include it that seems the easiest and most hack free solution.
Apple DTS did suggest including it might avoid some TCC check but I don?t know what that might be about.
I don?t know how many apps try to include native commands but I have at least one non-App Store one that does.
An attempt at a solution seems worthwhile rather just throwing errors when a developer tries to get theirs into the App Store.

I assume the important point here is that the app get the NSMicrophoneUsageDescription property, and afaict this can be provided by the Info.plist file for the entire application, as you say. And possible the problem here is that we embed metadata in the java executable at the same time.

Uh, this is something I again don?t understand. It seems questionable that most java applications would need this Info.plist key granting access to the microphone.
https://developer.apple.com/documentation/bundleresources/information_property_list/nsmicrophoneusagedescription <https://developer.apple.com/documentation/bundleresources/information_property_list/nsmicrophoneusagedescription>

It should be up to the developer to include this in their application Info.plist if their application requires it. That is where I thought my earlier suggestion of allowing application post-processing before standalone signing would make sense. The current jpackage solution of allowing the developer to include a separate complete alternate Info.plist in a separate directory I found somewhat awkward.
With post-processing you could include a tool like PlistBuddy
https://www.unix.com/man-page/osx/8/PLISTBUDDY/ <https://www.unix.com/man-page/osx/8/PLISTBUDDY/>

To script the Info.plist changes. I believe this is used by native OS/X applications for this purpose including from XCode.
I think it is actually an Apple tool. With a funner name than usual for them.

There seems to be a lot of guessing here. :-) I assume we either need to read up on how this works (which might be difficult if this is seem as a badly documented corner case even by Apple tech support), or test some alternatives, or perhaps both.

That solution make take some time to get correct, so the jpackage team needs to decide if they want to go with the workaround in this PR in the meantime.

OK, one last possible misunderstanding on my part. The PR simply throws an error if commands are included. It doesn?t involve a workaround does it?

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Michael Hall on build-dev:

What you say sounds good, although I feel I only partially understand it. :-)

If it?s correct that normal applications don?t need the executable included Info.plist and jpackage can somehow get versions that don?t include it that seems the easiest and most hack free solution.
Apple DTS did suggest including it might avoid some TCC check but I don?t know what that might be about.
I don?t know how many apps try to include native commands but I have at least one non-App Store one that does.
An attempt at a solution seems worthwhile rather than just throwing errors when a developer tries to get theirs into the App Store.

I assume the important point here is that the app get the NSMicrophoneUsageDescription property, and afaict this can be provided by the Info.plist file for the entire application, as you say. And possible the problem here is that we embed metadata in the java executable at the same time.

Uh, this is something I again don?t understand. It seems questionable that most java applications would need this Info.plist key granting access to the microphone.

https://developer.apple.com/documentation/bundleresources/information_property_list/nsmicrophoneusagedescription

It should be up to the developer to include this in their application Info.plist if their application requires it. That is where I thought my earlier suggestion of allowing application post-processing before doing separate standalone signing would make sense. The current jpackage solution of allowing the developer to include a separate complete alternate Info.plist in a separate directory I found somewhat awkward.
With post-processing you could include a tool like PlistBuddy

https://www.unix.com/man-page/osx/8/PLISTBUDDY/

To script the Info.plist changes. I believe this is used by native OS/X applications for this purpose including from XCode.
I think it is actually an Apple tool. With a funner name than usual for them.

There seems to be a lot of guessing here. :-) I assume we either need to read up on how this works (which might be difficult if this is seem as a badly documented corner case even by Apple tech support), or test some alternatives, or perhaps both.

That solution make take some time to get correct, so the jpackage team needs to decide if they want to go with the workaround in this PR in the meantime.

OK, one last possible misunderstanding on my part. The PR simply throws an error if commands are included. It doesn?t involve a workaround does it?

@magicus
Copy link
Member

magicus commented May 12, 2022

We have the NSMicrophoneUsageDescription permission on the java launcher in the JDK, since otherwise no Java program can access the mike, even though most won't care. I agree that the situation is different for a jpackaged app, where the developer knows if that permission is needed or not.

Yes, plistbuddy is an official Apple program.

My understanding of the PR was that native commands are removed by jlink if the user is packaging on a mac for the App Store. I thought this was a workaround that solved the immediate problem of not being able to submit the app to App Store. (However, I don't know how the app is supposed to be started without a launcher...)

@mlbridge
Copy link

mlbridge bot commented May 12, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 12, 2022, at 3:13 PM, Magnus Ihse Bursie <ihse at openjdk.java.net> wrote:

On Thu, 12 May 2022 04:15:50 GMT, Alexander Matveev <almatvee at openjdk.org> wrote:

- It is not possible to support native JDK commands such as "java" inside Mac App Store bundles due to embedded info.plist. Workarounds suggested in JDK-8286122 does not seems to be visible.
- With proposed fix we will enforce "--strip-native-commands" option for jlink, so native JDK commands are not included when generating Mac App Store bundles.
- Custom runtime provided via --runtime-image should not contain native commands as well, otherwise jpackage will throw error.
- Added two tests to validate fix.

Alexander Matveev has updated the pull request incrementally with one additional commit since the last revision:

8286122: [macos]: App bundle cannot upload to Mac App Store due to info.plist embedded in java exe [v2]

We have the `NSMicrophoneUsageDescription` permission on the `java` launcher in the JDK, since otherwise no Java program can access the mike, even though most won't care. I agree that the situation is different for a jpackaged app, where the developer knows if that permission is needed or not.

I?d have to agree with Apple DTS that this is an interesting exception.

Yes, plistbuddy is an official Apple program.

My understanding of the PR was that native commands are removed by jlink if the user is packaging on a mac for the App Store. I thought this was a workaround that solved the immediate problem of not being able to submit the app to App Store. (However, I don't know how the app is supposed to be started without a launcher?)

I thought that if the app was indicated as intended for the App Store and native commands were also indicated an error would be thrown and the app not built. It doesn?t allow apps that will fail to attempt the App Store but does nothing for getting them there. Switching from an error to a warning and forcing the native commands to be stripped would allow the app into the app store but with unknown functionality probably not working. My understanding.

@alexeysemenyukoracle
Copy link
Member

alexeysemenyukoracle commented May 12, 2022

However, I don't know how the app is supposed to be started without a launcher...

jpackage supplies an alternative launcher that doesn't have plist.

@mlbridge
Copy link

mlbridge bot commented May 17, 2022

Mailing list message from Alexander Matveev on core-libs-dev:

Hi Michael,

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?
Yes, it should be possible and in theory such JDK with native commands can be used. However, current fix will not allow such JDK build, since we checking for presence of ?bin? folder and not if ID is actually unique. To work around this limitation user can package without native commands first, then add native commands and re-sign application.

Signing is currently an integrated part of the process. I thought about making an enhancement request to have it possible to do it stepped. First build the application, then allow the user to post-process that. Maybe run a tool/script to modify the application level Info.plist file. Then allow a second final step that does the signing. I don?t remember if I actually made that enhancement request. But currently there is no way with jpackage to standalone sign the application is there? I think this is somewhat involved with the application bundle being iterated and for one thing each jar being separately code signed? I think it took a release or two of the command to get this correct. So I doubt Xcode could manage it. Is there a way to do the signing standalone with jpackage that I am not aware of?

If I didn?t do an enhancement request on this I could, if you think jpackage could manage it?

There is no a way to do the signing of standalone app image and app image will not get signed if you generating DMG or PKG by providing app image via ?app-image and specifing signing. Only PKG will get signed if signing is specified with ?app-image.

Currently jpackage will block following command:
jpackage --type app-image --app-image Test.app --mac-sign ...
Error: Option [--app-image] is not valid with type [app-image]

I think we can allow app-image type with app image input if signing is requested and command from above will just sign app image in place.

Generating DMG or PKG from ?app-image and with signing enabled will not sign app image as it currently do.

If no objections for proposed enhancement I will go ahead and file one.

Thanks,
Alexander

@mlbridge
Copy link

mlbridge bot commented May 17, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 16, 2022, at 7:09 PM, Alexander Matveev <alexander.matveev at oracle.com> wrote:

Hi Michael,

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?
Yes, it should be possible and in theory such JDK with native commands can be used. However, current fix will not allow such JDK build, since we checking for presence of ?bin? folder and not if ID is actually unique. To work around this limitation user can package without native commands first, then add native commands and re-sign application.

Signing is currently an integrated part of the process. I thought about making an enhancement request to have it possible to do it stepped. First build the application, then allow the user to post-process that. Maybe run a tool/script to modify the application level Info.plist file. Then allow a second final step that does the signing. I don?t remember if I actually made that enhancement request. But currently there is no way with jpackage to standalone sign the application is there? I think this is somewhat involved with the application bundle being iterated and for one thing each jar being separately code signed? I think it took a release or two of the command to get this correct. So I doubt Xcode could manage it. Is there a way to do the signing standalone with jpackage that I am not aware of?

If I didn?t do an enhancement request on this I could, if you think jpackage could manage it?

There is no a way to do the signing of standalone app image and app image will not get signed if you generating DMG or PKG by providing app image via ?app-image and specifing signing. Only PKG will get signed if signing is specified with ?app-image.

Currently jpackage will block following command:
jpackage --type app-image --app-image Test.app --mac-sign ...
Error: Option [--app-image] is not valid with type [app-image]

I think we can allow app-image type with app image input if signing is requested and command from above will just sign app image in place.

Generating DMG or PKG from ?app-image and with signing enabled will not sign app image as it currently do.

If no objections for proposed enhancement I will go ahead and file one.

Thanks,
Alexander

Alexander

I?m not entirely sure I?m following this. I assumed it wasn?t currently possible to do standalone signing against an application bundle but would require an enhancement request. If you are saying you will do that request yourself, that?s fine.
Again, I think this should be possible as a two step process. The first would be to generate the application bundle. Then perform some application bundle post-processing, probably independent of jpackage. Finally allow a jpackage invocation that takes the path to the completed application bundle and performs the signing.
If this is what you are thinking we are on the same page.

Thanks
Mike

@mlbridge
Copy link

mlbridge bot commented May 17, 2022

Mailing list message from Alexander Matveev on core-libs-dev:

Hi Michael,

I filed following enhancement for signing user provided app image and yes it will be two step process. Invoke jpackage to generate image, then user can modified it and then invoke jpackage again to sign it.
https://bugs.openjdk.java.net/browse/JDK-8286850

For JDK-8286122, I will integrate proposed fix as is. Error will be thrown if user tries to include native commands for Mac App Store image.

Still not sure if we will provide solution for including native commands with Mac App Store. Including special versions of native commands with jpackage will work only if runtime is generated by jpackage. It will not solve issue with user provided runtime. Also, it is not clear if app updates will require same unique ID based on UUID across app versions.

Also, many functionality provided by native JDK commands can be used via ToolProvider. For example if you include jdk.jpackage module with your app, you should able to use jpackage functionality from your app via ToolProvider without actual jpackage native command.

Thanks,
Alexander

On May 16, 2022, at 6:03 PM, Michael Hall <mik3hall at gmail.com> wrote:

On May 16, 2022, at 7:09 PM, Alexander Matveev <alexander.matveev at oracle.com> wrote:

Hi Michael,

I?m not real familiar with the build process. But would it be possible for the user to build their own jdk that substitutes something else for the colliding identifier that gets embedded?
Yes, it should be possible and in theory such JDK with native commands can be used. However, current fix will not allow such JDK build, since we checking for presence of ?bin? folder and not if ID is actually unique. To work around this limitation user can package without native commands first, then add native commands and re-sign application.

Signing is currently an integrated part of the process. I thought about making an enhancement request to have it possible to do it stepped. First build the application, then allow the user to post-process that. Maybe run a tool/script to modify the application level Info.plist file. Then allow a second final step that does the signing. I don?t remember if I actually made that enhancement request. But currently there is no way with jpackage to standalone sign the application is there? I think this is somewhat involved with the application bundle being iterated and for one thing each jar being separately code signed? I think it took a release or two of the command to get this correct. So I doubt Xcode could manage it. Is there a way to do the signing standalone with jpackage that I am not aware of?

If I didn?t do an enhancement request on this I could, if you think jpackage could manage it?

There is no a way to do the signing of standalone app image and app image will not get signed if you generating DMG or PKG by providing app image via ?app-image and specifing signing. Only PKG will get signed if signing is specified with ?app-image.

Currently jpackage will block following command:
jpackage --type app-image --app-image Test.app --mac-sign ...
Error: Option [--app-image] is not valid with type [app-image]

I think we can allow app-image type with app image input if signing is requested and command from above will just sign app image in place.

Generating DMG or PKG from ?app-image and with signing enabled will not sign app image as it currently do.

If no objections for proposed enhancement I will go ahead and file one.

Thanks,
Alexander

Alexander

I?m not entirely sure I?m following this. I assumed it wasn?t currently possible to do standalone signing against an application bundle but would require an enhancement request. If you are saying you will do that request yourself, that?s fine.
Again, I think this should be possible as a two step process. The first would be to generate the application bundle. Then perform some application bundle post-processing, probably independent of jpackage. Finally allow a jpackage invocation that takes the path to the completed application bundle and performs the signing.
If this is what you are thinking we are on the same page.

Thanks
Mike

@mlbridge
Copy link

mlbridge bot commented May 17, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 17, 2022, at 12:15 AM, Alexander Matveev <alexander.matveev at oracle.com> wrote:

Hi Michael,

I filed following enhancement for signing user provided app image and yes it will be two step process. Invoke jpackage to generate image, then user can modified it and then invoke jpackage again to sign it.
https://bugs.openjdk.java.net/browse/JDK-8286850

For JDK-8286122, I will integrate proposed fix as is. Error will be thrown if user tries to include native commands for Mac App Store image.

Still not sure if we will provide solution for including native commands with Mac App Store. Including special versions of native commands with jpackage will work only if runtime is generated by jpackage. It will not solve issue with user provided runtime. Also, it is not clear if app updates will require same unique ID based on UUID across app versions.

This enhancement doesn?t directly concern this issue. Although given this enhancement it would be possible to provide a temporary ?hack? fix without building it into jpackage. Where the executables are changed to make them unique in the post-processing step. If it is not your intention to address this issue given this enhancement you would probably have to check the provided application bundle to be sure it doesn?t have the colliding Info.plist native commands. What Apple DTS provided may of had commands for this? I don?t remember. Or you would have to just fail any passed application bundles with native java commands.From https://bugs.openjdk.java.net/browse/JDK-8286850 <https://bugs.openjdk.java.net/browse/JDK-8286850>

Generating DMG or PKG from ?app-image with signing enabled will not sign app image as it currently do.

I am not sure you would want to change this. If the developer doesn?t want to do any post-processing of the application bundle then they should be able to use the current invocations unchanged to do this?
I would think if you want post-processing you would generate unsigned applications, post-process, use the enhancement to sign.

Also, many functionality provided by native JDK commands can be used via ToolProvider. For example if you include jdk.jpackage module with your app, you should able to use jpackage functionality from your app via ToolProvider without actual jpackage native command.

Thanks,
Alexander

Which might be another way this could be useful. If any 3rd parties want to generate their own application bundles they could still use jpackage as the definitive way to sign those.

Thanks
Mike

Copy link
Member

@kevinrushforth kevinrushforth left a comment

I'm just catching up with this thread. Based on the analysis, it's pretty clear that any solution that actually allows bundling native launchers is going to take more research, and is out of scope for this bug. Some combination of JDK-8286850 and/or providing Java launchers that don't have an Info.plist is likely needed to provide a complete solution.

Given that, I think that this PR seems a reasonable fix to avoid creating an app bundle that can't be uploaded to the app store.

@sashamatveev
Copy link
Member Author

sashamatveev commented May 18, 2022

/integrate

@openjdk
Copy link

openjdk bot commented May 18, 2022

Going to push as commit b523c88.
Since your change was applied there have been 173 commits pushed to the master branch:

  • 8323787: 8255439: System Tray icons get corrupted when windows scaling changes
  • cd5bfe7: 8286814: Shenandoah: RedefineRunningMethods.java test failed with Loom
  • b5a3d28: 8285097: Duplicate XML keys in XPATHErrorResources.java and XSLTErrorResources.java
  • 6b9c152: 8286366: (cs) Charset.put can use putIfAbsent instead of containsKey+put
  • 9becf7d: 8283705: Make javax.sound.midi.Track a final class
  • a03438c: 8285397: JNI exception pending in CUPSfuncs.c:250
  • 9ab29b6: 8286869: unify os::dir_is_empty across posix platforms
  • ee45a0a: 8286669: Replace MethodHandle specialization with ASM in mainline
  • d8b0b32: 8286763: [REDO] (fc) Tune FileChannel.transferFrom()
  • ac7e019: 8286925: Move JSON parser used in JFR tests to test library
  • ... and 163 more: https://git.openjdk.java.net/jdk/compare/54e33082105dcbcfc795839c954f6e63402edff1...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated label May 18, 2022
@openjdk openjdk bot closed this May 18, 2022
@openjdk openjdk bot removed ready rfr labels May 18, 2022
@openjdk
Copy link

openjdk bot commented May 18, 2022

@sashamatveev Pushed as commit b523c88.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@mlbridge
Copy link

mlbridge bot commented May 19, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 11, 2022, at 4:39 PM, Alexander Matveev <almatvee at openjdk.java.net> wrote:

- It is not possible to support native JDK commands such as "java" inside Mac App Store bundles due to embedded info.plist. Workarounds suggested in JDK-8286122 does not seems to be visible.

I was just thinking about this. If you wanted a workaround to suggest to the user on the original issue. You could jar the native executables, extract them to a known accessible location, and then runtime them.

Having the commands jar?d would get them past the App Store check. Runtime on the client machine I doubt would object to the duplicate bundle id?s on absolute path execution but I haven?t double checked that. Also avoids issues with quarantine xattr?s.
I?ve done something like this a couple times for interfacing an app to other languages.

For example?

		if \(Files\.exists\(rscriptCmd\)\) \{
			isR \= true\;
			System\.out\.println\(\"InitialFinance\: RScript available\"\)\;
			\/\/ Where is the finance data directory\?
			String data \= prefs\.get\(\"data\"\,\"N\/A\"\)\;
			
			if \(data\.equals\(\"N\/A\"\)\) \{
				Application app \= Application\.getApplication\(\)\;
				Path documents \= app\.getFolder\(DataTypes\.DOCUMENTS\)\;
				dataLoc \= Paths\.get\(documents\.toString\(\)\,\"finance\"\)\;
			\}
			else \{
				dataLoc \= Paths\.get\(data\)\;					
			\}
			System\.out\.println\(\"InitialFinance\: data location is \" \+ dataLoc\)\;
			Path resourceJar \= Paths\.get\(System\.getProperty\(\"app\.path\"\)\,\"resource\.jar\"\)\;	
			System\.out\.println\(\"InitialFinance\: Checking resources for updates\"\)\;
			extractArchive\(resourceJar\,dataLoc\)\;
		\}
		else \{
			System\.out\.println\(\"InitialFinance\: \" \+ rscriptCmd \+ \" for \" \+ initialRscript \+ \" does not exist\"\)\;
			isR \= false\;
		\}

\/\/ https\:\/\/stackoverflow\.com\/questions\/1529611\/how\-to\-write\-a\-java\-program\-which\-can\-extract\-a\-jar\-file\-and\-store\-its\-data\-in\-s

public static void extractArchive(Path archiveFile, Path destPath) throws IOException {
Files.createDirectories(destPath); // create dest path folder(s)
try (ZipFile archive = new ZipFile(archiveFile.toFile())) {

@SuppressWarnings("unchecked")
Enumeration<ZipEntry> entries = (Enumeration<ZipEntry>) archive.entries();

// copy or create new or updated
while (entries.hasMoreElements()) {
ZipEntry entry = entries.nextElement();

if (!entry.getName().startsWith("finance/") || !(entry.getName().length() > 8)) {
continue;
}
String fileName = entry.getName().substring(8);
FileTime archiveTime = entry.getLastModifiedTime();

Path entryDest = destPath.resolve(fileName);
//if (Files.isDirectory(entryDest)) continue;
//Files.createDirectories(entryDest);
if (!Files.exists(entryDest)) {
Files.copy(archive.getInputStream(entry), entryDest, StandardCopyOption.REPLACE_EXISTING);
continue;
}
BasicFileAttributes destAttr =
Files.readAttributes(entryDest, BasicFileAttributes.class);

if (archiveTime.compareTo(destAttr.creationTime()) > 0) {
Files.copy(archive.getInputStream(entry), entryDest, StandardCopyOption.REPLACE_EXISTING);
}
}
}
catch (IOException ioex) {
throw ioex;
}
}

boolean debug = Boolean.getBoolean("R.debug");
rtexec(new String[] { RSCRIPT, script.toString() },debug);

@mlbridge
Copy link

mlbridge bot commented May 19, 2022

Mailing list message from Alexander Matveev on core-libs-dev:

Hi Michael,

I think it will be a problem to implement this for native launchers.
- If we extract native launchers inside installed app bundle it will invalidate signature and most likely will require privileged permissions to write inside app bundle.
- If we extract to some known and accessible location as you suggesting, then how our launchers will figure out which runtime to use? All other runtime files will be inside app bundle.
- If user deletes application, then how we will cleanup extracted files? Most likely it will require uninstall script in known location which user will need to run in order to cleanup extracted files.

Thanks,
Alexander

On May 18, 2022, at 6:44 PM, Michael Hall <mik3hall at gmail.com<mailto:mik3hall at gmail.com>> wrote:

On May 11, 2022, at 4:39 PM, Alexander Matveev <almatvee at openjdk.java.net<mailto:almatvee at openjdk.java.net>> wrote:

- It is not possible to support native JDK commands such as "java" inside Mac App Store bundles due to embedded info.plist. Workarounds suggested in JDK-8286122 does not seems to be visible.

I was just thinking about this. If you wanted a workaround to suggest to the user on the original issue. You could jar the native executables, extract them to a known accessible location, and then runtime them.

Having the commands jar?d would get them past the App Store check. Runtime on the client machine I doubt would object to the duplicate bundle id?s on absolute path execution but I haven?t double checked that. Also avoids issues with quarantine xattr?s.
I?ve done something like this a couple times for interfacing an app to other languages.

For example?

if (Files.exists(rscriptCmd)) {
isR = true;
System.out.println("InitialFinance: RScript available");
// Where is the finance data directory?
String data = prefs.get("data","N/A");

if (data.equals("N/A")) {
Application app = Application.getApplication();
Path documents = app.getFolder(DataTypes.DOCUMENTS);
dataLoc = Paths.get(documents.toString(),"finance");
}
else {
dataLoc = Paths.get(data);
}
System.out.println("InitialFinance: data location is " + dataLoc);
Path resourceJar = Paths.get(System.getProperty("app.path"),"resource.jar");
System.out.println("InitialFinance: Checking resources for updates");
extractArchive(resourceJar,dataLoc);
}
else {
System.out.println("InitialFinance: " + rscriptCmd + " for " + initialRscript + " does not exist");
isR = false;
}

// https://stackoverflow.com/questions/1529611/how-to-write-a-java-program-which-can-extract-a-jar-file-and-store-its-data-in-s
public static void extractArchive(Path archiveFile, Path destPath) throws IOException {
Files.createDirectories(destPath); // create dest path folder(s)
try (ZipFile archive = new ZipFile(archiveFile.toFile())) {

@SuppressWarnings("unchecked")
Enumeration<ZipEntry> entries = (Enumeration<ZipEntry>) archive.entries();

// copy or create new or updated
while (entries.hasMoreElements()) {
ZipEntry entry = entries.nextElement();

if (!entry.getName().startsWith("finance/") || !(entry.getName().length() > 8)) {
continue;
}
String fileName = entry.getName().substring(8);
FileTime archiveTime = entry.getLastModifiedTime();

Path entryDest = destPath.resolve(fileName);
//if (Files.isDirectory(entryDest)) continue;
//Files.createDirectories(entryDest);
if (!Files.exists(entryDest)) {
Files.copy(archive.getInputStream(entry), entryDest, StandardCopyOption.REPLACE_EXISTING);
continue;
}
BasicFileAttributes destAttr =
Files.readAttributes(entryDest, BasicFileAttributes.class);

if (archiveTime.compareTo(destAttr.creationTime()) > 0) {
Files.copy(archive.getInputStream(entry), entryDest, StandardCopyOption.REPLACE_EXISTING);
}
}
}
catch (IOException ioex) {
throw ioex;
}
}

boolean debug = Boolean.getBoolean("R.debug");
rtexec(new String[] { RSCRIPT, script.toString() },debug);

@mlbridge
Copy link

mlbridge bot commented May 19, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 19, 2022, at 5:38 PM, Alexander Matveev <alexander.matveev at oracle.com> wrote:

Hi Michael,

Alexander

I think it will be a problem to implement this for native launchers.

Each of the native commands is it?s own app launcher? I didn?t know this. I don?t think it was ever really indicated in the discussion why the commands needed the embedded plist. I think Apple DTS guessed it had something to do with a TCC issue.

- If we extract native launchers inside installed app bundle it will invalidate signature and most likely will require privileged permissions to write inside app bundle.

I thought maybe jpackage unsigned and resigned the jdk itself. Maybe not. I was talking about simply including a jar in input that ends up in the ?app? directory and the app then has the jar contents extracted from there. No additional write permission?s are required.

- If we extract to some known and accessible location as you suggesting, then how our launchers will figure out which runtime to use? All other runtime files will be inside app bundle.

Again this possibly my not understanding what additional requirements there are for the commands to be ?launchers?. I was thinking if the embedded plist was still a problem the developer could possibly use commands from a compatible linux distribution. OS/X and linux are both Unix right? It did occur to me that extracting native commands out of a jar might lose the executable posix permission. I?m not sure if jar has any builtin meta information to retain these permissions or if nio allows you to set executable. I haven?t had to address this.

- If user deletes application, then how we will cleanup extracted files? Most likely it will require uninstall script in known location which user will need to run in order to cleanup extracted files.

I haven?t tried to address this yet myself for my application. I?m not sure a lot of OS/X application?s do. But it would probably be up to the app to provide an uninstall of some kind. Some app?s are obviously going to need external data.

Thanks,
Alexander

Again I?m suggesting it as a roughed out workaround suggestion for the developer. There might be issues they would need to work out and might even be problems where it won?t work.
I?m not suggesting it as a fix you would try to implement.

Thanks
Mike

@mlbridge
Copy link

mlbridge bot commented May 20, 2022

Mailing list message from Michael Hall on core-libs-dev:

Alexander

I think it will be a problem to implement this for native launchers.

Basically you are telling the developer in https://bugs.openjdk.java.net/browse/JDK-8286122 <https://bugs.openjdk.java.net/browse/JDK-8286122> that it isn?t currently possible for their application to get into the Mac App Store as is.
I was thinking you could possibly in the comments mention the possibility the developer could provide these in an archive themselves that they manually manage as a workaround.
Leave them with a possible path forward to achieve this.
I have successfully done this for something similar if not identical.

@mlbridge
Copy link

mlbridge bot commented May 20, 2022

Mailing list message from Michael Hall on core-libs-dev:

On May 19, 2022, at 7:14 PM, Michael Hall <mik3hall at gmail.com> wrote:

Alexander

I think it will be a problem to implement this for native launchers.

Basically you are telling the developer in https://bugs.openjdk.java.net/browse/JDK-8286122 <https://bugs.openjdk.java.net/browse/JDK-8286122> that it isn?t currently possible for their application to get into the Mac App Store as is.
I was thinking you could possibly in the comments mention the possibility the developer could provide these in an archive themselves that they manually manage as a workaround.
Leave them with a possible path forward to achieve this.
I have successfully done this for something similar if not identical.

I think it can work for java commands as well. You have to set them executable after extraction and point to the embedded jdk lib directory

DYLD_LIBRARY_PATH=JavaCommand.app/Contents/runtime/Contents/Home/lib JavaCommand.app/Contents/MacOS/JavaCommand
java version "18" 2022-03-22
Java(TM) SE Runtime Environment (build 18+36-2087)
Java HotSpot(TM) 64-Bit Server VM (build 18+36-2087, mixed mode)

I can provide the full JavaCommand source code and jpackage invocation if of interest. But yes, it works.

From?

String v = rtexec(new String[] { Paths.get(dataLoc,"java").toString(),"-version" });
System.out.println(v);

You?d have to figure out how to provide the environment variable on the runtime invocation. That I didn?t verify, but it doesn?t seem impossible.
I added that after getting?

JavaCommand.app/Contents/MacOS/JavaCommand
dyld[5725]: Library not loaded: @rpath/libjli.dylib
Referenced from: /Users/mjh/testLoc/java
Reason: tried: '/libjli.dylib' (no such file), '/Users/mjh/JavaCommand.app/Contents/app/libjli.dylib' (no such file), '/Users/mjh/testLoc/./libjli.dylib' (no such file), '/Users/mjh/testLoc/../lib/libjli.dylib' (no such file), '/Users/mjh/testLoc/./libjli.dylib' (no such file), '/Users/mjh/testLoc/../lib/libjli.dylib' (no such file), '/usr/lib/libjli.dylib' (no such file)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build core-libs integrated
4 participants