[Moonshine SDK Installers] Automate JCE Fix and Code Updates

[JoelProminic]

@rat-moonshine reminded me about updating the App Store Helper (or built-in SDK installer for Windows) to install the latest Flex and AIR SDK version (#37).

It seems that we need to need to update the JRE/JDK with the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 Download" files in order to work around the error we have been seeing later versions of AIR:
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html

In order to keep the App Store Helper simple, we would need to automate the sestup for the JCE files. This process may look something like this:

1. SDK Installer launches
2. SDK Installer reports a message explaining that the SDK installer requires the JCE extension. If there are other implications or limitations for this, report them to the user here.
3. SDK Installer prompts the user to navigate to their JAVA_HOME directory. This may only be needed for Windows
4. Download the JCE files (should we save a copy on moonshine-ide.com?)
5. Copy the files to JAVA_HOME/lib/security and set any required permissions. (UPDATE: JAVA_HOME is the JRE home, in this case. For a JDK, this would be $JAVA_HOME/jre)
6. Start the installation.

I see these problems:

• We need to make the setup process more complicated
• We are editing the user's JRE, which make them (or the App Store reviewers) nervous
• There may be other restrictions or side effects that will make this infeasible

[JoelProminic]

Here is the README.txt file from the download, for convenience:
2017_12_21__JCE_README.txt

Some notes from my quick review:

• This extension includes support for "unlimited strength" cryptographic keys
• These cryptographic keys may be illegal for export to some countries
• The instructions recommend updating all installed JREs on Windows. I think this overkill for our needs.
• The SDK installer should automatically make backups of the existing US_export_policy.jar and local_policy.jar files

More for the Import/Export issues:
This is appropriate for most
countries. Framework vendors can create download bundles that include
jurisdiction policy files that specify cryptographic restrictions
appropriate for countries whose governments mandate restrictions. Users
in those countries can download an appropriate bundle, and the JCE
framework will enforce the specified restrictions.

You are advised to consult your export/import control counsel or
attorney to determine the exact requirements.

I remember that we were prompted about the import restrictions in either the App Store or Test Flight. So, this may change what we need to report for the App Store review.

[rat-moonshine]

We also need to updates codes to built-in Moonshine's Apache SDK Installer and Moonshine App Store Helper, per recent changes to original Apache SDK Installer to fix AIR download problems.

Some quick note from @piotrzarzycki21 on this from #37 :

@rat-moonshine repository where you can find installer sources are here. You can switch to tag "apache-flex-sdk-installer-3.3.0", folder flex-installer. I'm not sure how you are preparing installer swc, but build first ant_on_air which have most of the changes.

[rat-moonshine]

Hi @piotrzarzycki21 , I'm not familiar with the interface here. Can you describe me how to download all the source tagged inside 'apache-flex-sdk-installer-3.3.0'? Thanks!

[piotrzarzycki21]

@rat-moonshine did you checkout the repository ? What client for the Git are you using ? I'm using SourceTree. In the SourceTree it is the matter of double click. If you are using command line, once you checkout the repository you have to type:
git checkout -b tags/apache-flex-sdk-installer-3.3.0

[rat-moonshine]

By the above link it just opens me an webpage with lots of links etc. but I was unsure how to download certain tag/linked files (as there were no link to do that or any instruction related to that). I shall try with your command.

[piotrzarzycki21]

You should use that link to checkout the repository:

https://git-wip-us.apache.org/repos/asf?p=flex-utilities.git

Later you can switch to the tag.

[rat-moonshine]

While working on this, I'm getting this error when application starts:

Could not fetch the SDK download mirror URL from the CGI. Going to try the GeoIP route.

Looking into the source, this erupts from defaultLanguageLoaded() method after following call handler:

_mirrorURLUtil.getMirrorURL(Constants.APACHE_FLEX_URL + _mirrorURLCGI, getMirrorURLResultHandler);

This looks like it's building an URL http://flex.apache.org/single-mirror-url--xml.cgi after parsing data from http://flex.apache.org/installer/sdk-installer-config-4.0.xml. Calling the .cgi page giving me this error in browser:

@piotrzarzycki21 do you have any suggestion how to overcome this problem?

[piotrzarzycki21]

It looks like it is problem with update. Can you send me created SWC. Maybe we will need to somehow disable updates.

Do you have some simple instruction to create me on my own Installer SWC for Moonshine ?

[rat-moonshine]

For Moonshine App Store Helper we use a some custom scripts (per the app's custom requirements) over the original SDK installer source that you can found inside the installer folder.

The project also uses SWCs from the libs directory and source paths of common, ant_on_air etc.

At this moment Moonshine do not have support to generate SWC (which shall be a good addition point). Perhaps I can disable the updates that you've mentioned - commenting out the following line would do the job?

_mirrorURLUtil.getMirrorURL(Constants.APACHE_FLEX_URL + _mirrorURLCGI, getMirrorURLResultHandler);

[rat-moonshine]

I see the mirrorURL issue already raised to the developers' forum - http://apache-flex-development.2333347.n4.nabble.com/Apache-Flex-SDK-Installer-Broken-td54742.html.

The workaround is to not to use the mirror cgi URL from the sdk-installer-config-4.0.xml and build the correct one -

Right: http://flex.apache.org/single-mirror-url.cgi
Wrong: https://flex.apache.org/single-mirror-url--xml.cgi

This mirrorURL does use in different places so we can't cut it down from the usage easily.

[rat-moonshine]

So far, the funny thing is, the installer's InternalUtil.fetch method's URLLoader is throwing error while accessing the mirror URL but opening good on browser:

Error #2032: Stream Error. URL: http://flex.apache.org/single-mirror-url.cgi

I don't have any correct idea why it is so, needs to look into more.

[piotrzarzycki21]

Not sure what I can do. Do you have that problem when you launch installer and just download SDK ? If yes I can report it in that thread.

[rat-moonshine]

SDK download URLs prepared with the value get from this mirror URL, as suffix. The value could be http://www-eu.apache.org/dist/ or http://www-us.apache.org/dist/ etc. I can hard-code this to use http://www-us.apache.org/dist/ to overcome this problem, but the problem seems again shown up since the last discussion to the issue link I've mentioned.

[piotrzarzycki21]

I'm not sure what do you mean now. I thought that we have problem with that url -> http://flex.apache.org/single-mirror-url--xml.cgi. If yes I will simply ask on the Flex forum and later raise jira to the Apache INFRA. There is actually jira for that purpose take a look INFRA-10782

[rat-moonshine]

If you go through this discussion http://apache-flex-development.2333347.n4.nabble.com/Apache-Flex-SDK-Installer-Broken-td54742.html, you shall see some comment of using http://flex.apache.org/single-mirror-url.cgi instead of the http://flex.apache.org/single-mirror-url--xml.cgi. The former opens to the browser when the latter is not.

However, using the former one I don't know why but getting stream error while loading through the installer's URLLoader calls.

I'm trying to understand if this stream error is something part of our development environment or my system or something else.

But I think anyway its legit to report that http://flex.apache.org/single-mirror-url--xml.cgi is not working in the Apache SDK Installer source.

[piotrzarzycki21]

Maybe because we are expecting XML instead of plain text.

[rat-moonshine]

I noticed all the URLLoader calls from the installer source throws me #2032: Stream error. This was a bit odd. So I tried to do a simper tests by calling an URL like http://moonshine-ide.com/wp-content/uploads/updater.xml through its URLLoader class.

_urlLoader = new URLLoader();
_urlLoader.dataFormat = URLLoaderDataFormat.TEXT;
_urlLoader.addEventListener(Event.COMPLETE, fetchResultHandler);
_urlLoader.addEventListener(IOErrorEvent.IO_ERROR, fetchResultHandler);
_urlLoader.load(new URLRequest(fetchURL + ((args) ? "?" + args : "")));

With my surprise, the above URL also returned me the same error! I remember a few days back I changed the using SDK to Apache Flex 4.16.1 with AIR 28.0.

Once I changed the SDK back to use AIR 22.0 with Apache Flex 4.15.0 - everything is fine again!

So is it possible that URLLoader has some problem in latest AIR SDK (?)

[piotrzarzycki21]

That's interesting! I build Installer as I remember with Flex 4.16 and AIR 27 it was working there.

[rat-moonshine]

Yes, I can confirm it's (URLLoader) breaking on AIR 28.0.

I did a basic test in Moonshine using Apache Flex 4.16.1 with AIR 28.0 on my El Capitan, and immediately it turns to the IOError:

	var urlloader:URLLoader = new URLLoader();
	urlloader.addEventListener(Event.COMPLETE, onLoadCompletes);
	urlloader.addEventListener(IOErrorEvent.IO_ERROR, onIOError);
	urlloader.load(new URLRequest("http://moonshine-ide.com/wp-content/uploads/updater.xml"));
}
		
private function onLoadCompletes(event:Event):void
{
	Alert.show("positive");
}
		
private function onIOError(event:IOErrorEvent):void
{
	Alert.show("negative");
}

[piotrzarzycki21]

@rat-moonshine Can you try https url instead http ?

[piotrzarzycki21]

I'm asking because with Air 28 we need to use https urls. Take a look into the release notes -
App Transport Security.

[rat-moonshine]

That is something interesting. I shall give a try on this, soon. Thank you for this finding @piotrzarzycki21 . Just wondering this anyway worked for you with AIR 27.0 - isn't it? I will check this.

[piotrzarzycki21]

Yes it was working for me with Air 27.

[piotrzarzycki21]

It's working now. I just tested also installer by downloading Flex 4.16.1 with Air 28.0 and everything works great. Good job! Thanks!

[JoelProminic]

After revisiting the original problem, I noticed that I was able to download the latest SDKs without installing the JCE fix.

I created a simple Java application to check for the JCE unlimited length encryption keys:
2018_01_29__CheckJCE.zip

You can call it like this:

$JAVA_HOME/bin/java -jar CheckJCE.jar

The code is pretty simple:

package com.moonshine;

import java.security.NoSuchAlgorithmException;

import javax.crypto.Cipher;

/**
 * Check if the JCE unlimited strength keys are installed and enabled for this JDK/JRE.
 * Returns 0 if the keys are available.
 * Returns 1 if the keys are not available.
 * Returns 2 if an error occurs during checking
 * Check the STDOUT output for more information.
 * @author joelanderson
 *
 */
public class CheckJCE {

    public static void main(String[] args) {
        try {
            if (restrictedCryptography()) {
                System.out.println("This JRE/JDK does *not* use unlimited strength keys.");
                System.exit(1);
            }
            else {
                System.out.println("This JRE/JDK uses unlimited strength keys.");
                System.exit(0);
            }
        }
        catch (Throwable ex) {
            System.out.println("Unable to check for unlimited strength keys on this JRE/JDK.");
            ex.printStackTrace(System.out);
            System.exit(2);
        }

    }
    
    /**
     * I got this snippet from:  https://stackoverflow.com/questions/7953567/checking-if-unlimited-cryptography-is-available
     * 
     * Determines if cryptography restrictions apply.
     * Restrictions apply if the value of {@link Cipher#getMaxAllowedKeyLength(String)} returns a value smaller than {@link Integer#MAX_VALUE} if there are any restrictions according to the JavaDoc of the method.
     * This method is used with the transform <code>"AES/CBC/PKCS5Padding"</code> as this is an often used algorithm that is <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#impl">an implementation requirement for Java SE</a>.
     * 
     * @return <code>true</code> if restrictions apply, <code>false</code> otherwise
     */
    public static boolean restrictedCryptography() {
        try {
            return Cipher.getMaxAllowedKeyLength("AES/CBC/PKCS5Padding") < Integer.MAX_VALUE;
        } catch (final NoSuchAlgorithmException e) {
            throw new IllegalStateException("The transform \"AES/CBC/PKCS5Padding\" is not available (the availability of this algorithm is mandatory for Java SE implementations)", e);
        }
    }

}

However, it sounds like other countries may receive JDK builds that do not include the full encryption keys.

Does anyone know the algoritm and minimum encrption key length that are needed for the SDK installer. If I know the exact requirements, I could further refine this application.

If we want to include a JAR like this with Moonshine for checking the JDK status, then I could create a signed version. I's possible that the App Store could reject Moonshine if it runs an unknown Java JAR on startup, though.

[rat-moonshine]

Moonshine App Store Helper app distributes off-App-Store (and as a separate entity of Moonshine), so I don't think we fall in restrictions to the App Store reviewers, in macOS case.

Regarding Windows in-build Apache SDK downloader run, we probably good to bundle it anyway.

[rat-moonshine]

@piotrzarzycki21 , before we can decide on some auto-download or notification implementation to the Moonshine App Store Helper or to the in-build Apache SDK Installer (in Windows) for JCE file needs, can you prepare some words along with any technical parts for having them (.jar files) to complete latest SDK downloads successfully through the updated installer codes?

Once you prepare and reviewed by @JoelProminic , we shall probably want to have them somewhere in Moonshine website, before the release of 1.9.0.

[piotrzarzycki21]

I'm not sure what are you asking me for ? Do you want from me links to those jar files which resolve issue with downloading ? I think they are in the description of that issue.

[rat-moonshine]

@JoelProminic basically wants some information to be displayed on Moonshine website against the recent changes to SDK installer codes, and why the need of JCE fix to be present in users' system prior to download a latest SDK.

I was only asking if you can gather some words (maybe with technical details) on the above, which later can be reviewed by @JoelProminic for any correction, before a release.

[piotrzarzycki21]

To be honest I don't know why it is happen. We didn't change installer as much and in the results of that changes gets that errors. Note that it is not happen always. For example I bought one week ago new SSD and install there fresh Windows 10. I did check and download Flex SDK through Moonshine without the problem.
Not sure what kind of words we can put. Maybe some kind of Known Issue section for the release. If you experience problem with downloading it may be above issue.

[rat-moonshine]

Joel thinks (maybe) some of the JCE encryption comes bundled within JDK to some countries. Like, when he tested he able to download latest SDKs without installing JCE separately. In my case, I had to install JCE in my both systems to able to download latest SDKs successfully.

But the things is, there should be some note somewhere in the Moonshine website about the probable JCE fix needs, to overcome such problems that I had, and yes possibly with download links to JCE.

[piotrzarzycki21]

I can only suggest place where it may be. I recommend to add it in the section FAQ.

[JoelProminic]

I updated the test JAR. @piotrzarzycki21 and @rat-moonshine, please try running this with your JRE/JDKs.
2018_02_07__CheckJCE.zip

Command:

$JAVA_HOME/bin/java -jar CheckJCE.jar

For a default JDK, I get:

Maximum allowed key length:  128
This JRE/JDK does *not* use unlimited strength keys.

For a JDK with JCE, I get:

Maximum allowed key length:  2147483647
This JRE/JDK uses unlimited strength keys.

[JoelProminic]

Here is a quick writeup of the JCE issue. I was planning to put it in the Moonshine GitHub wiki, but I see this hasn't been initialized yet. I thought we had created this already...

If you encounter errors downloading SDKs through Moonshine (or Moonshine App Store Helper for macOS), you may be able to resolve them by installing JCE in your Java installation:

Java 7: http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
Java 8: http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

JCE will enable unlimited key lengths for HTTPS connnections from Java.

These downloads will contain US_export_policy.jar and local_policy.jar. These JAR files should be copied to this directory:

• JRE: %JAVA_HOME%/lib/security
• JDK: %JAVA_HOME%/jre/lib/security

Before you copy the files, make a backup of the existing US_export_policy.jar and local_policy.jar, so that you can revert later if necessary.

I'd like to have an example of the actual error that is triggered, but I don't have a way to reproduce the error at the moment. It would be hard to find a place for it in the FAQ without a better wayto describe the error.

Reviewing the F.A.Q. section on moonshine-ide.com, I feel that this is issue would not fit well in this context. It would work better in some sort of knowledgebase. Do you guys think we should set this up with the GitHub wiki, or through something on the moonshine-ide.com website?

I don't think this needs to hold up the 1.9.0 release.

[rat-moonshine]

I updated the test JAR. @piotrzarzycki21 and @rat-moonshine, please try running this with your JRE/JDKs. 2018_02_07__CheckJCE.zip

@JoelProminic , do you have a plan to distribute this version in future, with Moonshine App Store Helper or as an auto-download feature with Moonshine SDK Installer?

[EDIT: It looks like I mis-understood the purpose of the .jar file, I understood now]

Reviewing the F.A.Q. section on moonshine-ide.com, I feel that this is issue would not fit well in this context. It would work better in some sort of knowledgebase. Do you guys think we should set this up with the GitHub wiki, or through something on the moonshine-ide.com website?

I agree. This is not something directly related to Moonshine project, and may not fit in FAQ section of the IDE. One place we surely can have the write-up is in the downloader page of Moonshine App Store Helper - http://moonshine-ide.com/moonshine-app-store-helper/. Since Moonshine also has an in-build Apache installer that works on Windows, we may need the write-up to somewhere else as well, I can only think about some sort of information popup inside Moonshine at this moment, though.

[piotrzarzycki21]

@JoelProminic I think Wiki on GitHub is a good place, but I would add to that one more thing. Let's create wiki document called "Moonshine Known Issues" - or something more suitable. Put your message there along with maybe known issue from CHANGELOG.

Additionally I will create branch gh-pages - where I put that document into knownissues.md file. What's the advantage:

1. In future we can create automatically some web page from that documents
2. Users can potentially make pull requests. GitHub do not allow do pull requests to Wiki pages.

[piotrzarzycki21]

In my tests with your file I got following results:

D:\Work\projects\Prominic.NET\Files>java -jar CheckJCE.jar
Maximum allowed key length:  2147483647
This JRE/JDK uses unlimited strength keys.

I have fresh Windows 10 installed on my PC. It looks like when I install jdk1.8.0_161 those files are in place with installation!

To make sure that I'm correct I have uninstall Java completely and install it again. Files were in place. Maybe we are good on Windows ?

[JoelProminic]

I just downloaded a fresh Java 1.8u161 SDK on Windows, and I also found that it had the unlimited keys by default. My old SDK had a maximum length of 128, but that was still sufficient to download an SDK. Hopefully this means that we will not have the problem again on Windows.

@rat-moonshine, I wrote CheckJCE.jar in a way that it could be adapted for use with Moonshine, but I don't think that will be necessary for now. We can revisit this later if we have users reporting problems with the SDK downloads.

[piotrzarzycki21]

Let's close it if we are not going to add anything more in the final result

[rat-moonshine]

Maybe, with latest JDK versions has the unlimited keys bundled with, and we may not be needed to notify this separately (unless, users further report on this), do you think we should release a new version of MASH as well now, @JoelProminic ?

[JoelProminic]

@rat-moonshine and I revisited this today. I see that neither the wiki nor the installer page were updated.

I have initialized the wiki here. I am not really satisfied with the format, but I don't see how to create child pages: https://github.com/prominic/Moonshine-IDE/wiki/Known-Issues

[rat-moonshine]

I've a long troubled day today, with my Windows system and with my ISP. I'm trying to re-write following details again - thanks to above two factors.

After updating Bamboo build version of Moonshine App Store Helper (MASH) to latest Apache Flex 4.16.1 with AIR 28.0, I come up by different problems at different levels. I'm trying to document them all here.

I tried to download following SDKs (development version) in my tests on macOS:

• Apache Flex 4.16.1, AIR 28.0
• Apache FlexJS 0.8.0, AIR 28.0
• Apache Ant 1.10.1

Local/debug (28.0)

• MASH thrown error accessing http://flex.apache.org/single-mirror-url--xml.cgi (which we also discussed in this post in past), this is probably due to App Transport Security
• Once I changed the Apache Flex URL protocol from 'http' to 'https' (inside the code), I didn't had any problem accessing the above URL anymore
• Application failed to download Flex SDK (no handler error shown)

Local/debug (22.0)

• MASH didn't reported any error against http://flex.apache.org/single-mirror-url--xml.cgi usage
• Application completed downloading Apache Flex SDK but failed during FlexJS SDK download (no handler error shown)

Bamboo (28.0)

• Same as Local/debug (22.0)

While it was clear that http://flex.apache.org/single-mirror-url--xml.cgi is not up-to-date to App Transport Security requirement, I was wondering why downloads were breaking on FlexJS step. Investigating more, I have found that online configuration XML (of Apache) has changed with its URLs and etc. which I was not aware of. In past, we took decision to locally manage the configuration XML in MASH, but we always had this risk - the kind of situation we faced today. Once updated the local configuration copy I able to download FlexJS SDK as well (with AIR 22.0).

So, I have modified the configuration loading ways from local to online. If we still want to use the local configuration copy, let me know. Although, there shall be some risk remain like this.

Even after updating configuration XML to the online one, I continue getting download problems while using AIR SDK 28.0 (IOError). This is very much possible that different download mirrors still not comply to App Transport Security requirement on macOS. I don't think it make sense to investigate more on this when managing those URLs are not in our hand.

I installed Apache Flex 4.16.1 with AIR 26.0 on Bamboo (from 27.0 App Transport Security introduced) and I tested by the build:

• All SDKs downloaded properly

The above configuration URL changes shall also affect Window Moonshine (in-build Apache SDK Installer lib). We shall need to release a version of that build as well in next release.

Current MASH application possibly will break downloading at some point due to the local configuration XML usage. We possibly need to release a new version of MASH with other code updates, soon.

(This may not related to the above - but while testing Windows Moonshine in-build SDK installer, I had download error while downloading Adobe OSMF (can't have the exact details now as my Windows system crashed and all I have this mac system now)).

@JoelProminic , you can download a latest copy from Bamboo (MASH), this is packaged with Apache Flex 4.16.1, AIR 26.0.

Please, check.

[piotrzarzycki21]

@rat-moonshine The problem with OSMF is something which appear on Windows lately. I believe in order to fix that I need to make changes in the installer. It happened probably because sourceforge again changed something and OSMF is stored on that portal.
Summarize I just need to spent some time on fixing installer. - At least this is how I see that.

[JoelProminic]

I ran tests with the latest Bamboo build on both Mac OS Yosemite and macOS High Sierra. Both completed without errors.

An earlier test I ran on High Sierra got a IO Error (#2032). However, my computer locked in the middle of the build, which could have triggered the problem instead. My later test ran without errors.

For http://flex.apache.org/single-mirror-url--xml.cgi, I think we originally moved away from this because we were getting errors for certain mirrors. That is a short sighted solution, though, and if we continue getting errors like this, it might be better allow the users to choose a different mirror (or automatically select a different mirror when there is a major error).

[rat-moonshine]

The error to http://flex.apache.org/single-mirror-url--xml.cgi do not triggers in following scenarios:

• Used by AIR SDK < 27.0
• Changing domain protocol from 'http' to 'https' when AIR SDK > 26

I'm not sure if the configuration file has any alternate option to this mirror URL (so we can give option to the users). In code, there was only one tag reference to the configuration file which supplies the "single-mirror-url--xml.cgi" value from a field, the domain value is mentioned in the codes itself.

Since, we're using AIR SDK 26.0, we shouldn't have any problem with this URL, either. If we want to upgrade to higher SDK, then there shall be risk to get IOError from other different download URLs on App Transport Security.

[rat-moonshine]

AIR SDK download issue is seems to be fixed now and patch also applied to the Windows Moonshine in-build SDK Installer (1.11.0). It seems we overlooked this issue not updated to 1.11.0 CHANGELOG.md, though.

We can follow-up a release of MASH project with these fixes, separately.