New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: Resources directory structure #809

Closed
codenameone opened this Issue Mar 27, 2015 · 14 comments

Comments

Projects
None yet
1 participant
@codenameone
Owner

codenameone commented Mar 27, 2015

Original issue 809 created by codenameone on 2013-07-29T20:37:10.000Z:

It is well known (Shai reminds us often) that you can't embed resources in a directory structure in your app because on iOS (and possibly Windows Phone) all resources are just copied flat in to the top directory of the app bundle.

It doesn't need to be this way. For my offline builds of iOS I have been using full resource path structures for quite some time... I just updated my offline build project to do this too. I do it by not letting XMLVM deal with the resources at all. Instead I add a runscript build phase to the Xcode project that copies the resources into the bundle during the Xcode build step, maintaining the directory structure.

Relevant portions of my offline build script are as follows:

  1. Copy all resources from the class path into a resources directory.
    <copy todir="${xcoderes.path}">
    <fileset dir="src" excludes="/*.java"/>
    <fileset dir="lib/impl/cls" excludes="
    /*.java"/>
    </copy>
  2. Do the XMLVM build normally.
  3. Add the runscript build phase to the xcode project:
    <exec executable="/usr/bin/php" dir="${buildscripts.path}">
    <arg file="${xcodeadd.path}"/>
    <arg file="${xcodeproj.path}/project.pbxproj"/>
    </exec>

The ${xcodeadd.path} variable refers to a PHP script that actually does the processing of the xcode project and adds the runscript. I have attached the script for reference. Ultimately it just adds the following 1-line runscript:

cp -r "${PROJECT_DIR}/../resources/" "${BUILT_PRODUCTS_DIR}"

Justification for this change:

  1. One of the strengths of Java is its namespace ability. Without the ability to also include complementary resources for Java libraries it limits the potential of third-party libraries.
  2. Specific case is for libraries that wrap BrowserComponents and provide HTML-based components. We need to be able to embed stylesheets, images, html files, and Javascript files - or if "need" is too strong a word, it is much nicer to be able to embed these things without having to encode them into a Java class.
  3. Eliminate possibility of conflicts. WIth third party libraries, even if they just use a flat directory structure, there is potential for conflicts if two libraries use resources with the same name.

Possible Pitfalls:

  1. Optimization and Obfuscation. Currently unused java classes and methods are stripped out of the resulting binaries which helps to keep the binary size to a minimum. Resources don't naturally conform to this type of optimization so if the CN1 core gets bloated with lots of resources, it will result in the application binaries growing larger whether or not the additional resources are used.

    Solutions:
    a. Don't use resources in the CN1 core. They are still very useful for external libraries. That way application developers can opt to use the library only if they need it. Thereby not bloating their application with unneeded resources.
    b. Require CN1 classes to declare which resource directory trees they depend on using either an annotation or some other method. Then use s proguard (or similar) as a preprocessing step to strip out resources which are not needed.

     (I think solution a is perfectly fine and it doesn't require any special development).
    
  2. Consistency across platforms. If we make this change to iOS, then what about Windows Phone, Blackberry, J2ME, and Android?
    Solution:
    I would propose that all platforms should retain directory structure of resources. Android already does. I haven't tried Windows Phone but I'm sure a similar solution to the iOS solution would not be difficult to create. I presume since JME and Blackberry are built on Java that they already retain directory structure of resources.

  3. Backward compatibility: What if current apps depend on the flat directory structure?
    Solutions:
    a. Add a built flag to make this optional.
    b. Deprecate the old behavior and ask people to change it. I doubt many people depend on this behavior, and if they do, they are probably placing their resources in the root anyways, which would still function the same way.

This one change will drastically increase the potential for a third-party library eco system.

References:

  1. The offline build tools projects build file. (It's a mess, but you can refer to the "build-for-ios-device-locally" target to see the IOS build rules:
    http://codenameone-incubator.googlecode.com/svn/trunk/shannah/offline-build-tools/templates/build-ios.xml
  2. PHP Script for adding runscript build phase to Xcode Project:
    http://codenameone-incubator.googlecode.com/svn/trunk/shannah/offline-build-tools/scripts/prepare_xcode_project.php

Steve

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #1 originally posted by codenameone on 2013-07-29T20:39:40.000Z:

Whoops. Just noticed a bug in the copy resources snippet I posted here.

The step 1 to copy all resources into a resources directory should be:



Because the lib/impl/cls directory will contain .class files, not .java files.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #2 originally posted by codenameone on 2013-07-31T12:37:48.000Z:

I don't understand how this works. The problem is that xmlvm flattens the hierarchy not us. Where do you update xcode with the files if XMLVM doesn't do that?

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #3 originally posted by codenameone on 2013-07-31T16:37:27.000Z:

XMLVM only needs the .h, .m, and .class files in order to to its work. It outputs an Xcode project.

With the resulting Xcode project, I add a build phase that tells Xcode to copy all of the runtime resources into the project build directory during its compilation.

I did this by:

  1. Convert the .pbxproj file into an XML property list (instead of the old format) using plutil (because XML is easier to work with)
  2. Add a PBXShellScriptBuildPhase entry to the .pbxproj file. It looks like the following in the .pbxproj:
    e4e5a3214a4462a5fd77109d
buildActionMask 2147483647 files inputPaths isa PBXShellScriptBuildPhase outputPaths runOnlyForDeploymentPostprocessing 0 shellPath /bin/sh shellScript cp -r "${PROJECT_DIR}/../resources/" "${BUILT_PRODUCTS_DIR}"

The buildActionMask just needs to be exactly that value (2^32-1). The key just has to be unique among keys in the project file.

So this just tells Xcode to copy all of the files in the resources directory (a directory that I have previously filled with only the runtime resources) to the ${BUILD_PRODUCTS_DIR} which is an Xcode variable for the application bundle.

This results in a bundle that has all of the runtime resources in it with preserved package structure.

  1. Build the xcode project (I guess you guys are using pbxbuild).

Note 1: I had originally thought that I needed to convert the .pbxproj file back to its original form for Xcode to accept it, but it turns out you can just leave it as an XML property list and it will still work fine.

Note 2: The PHP script I attached, handles all of the parsing and converting of the .pbxproj file and insertion of the build phase. The closest thing to a main() method in that script is:

/**

  • A build script that prepares the XMLVM xcode project. Adds the

  • build phase to import resources into the app bundle, and also

  • adds file references for the specified files.

  • @param type $path

  • @param type $filesToAdd
    */
    function prepareXcodeProject($path, $filesToAdd = array()){
    $proj = new XCodeProject();
    $proj->load($path);
    $proj->addShellScriptBuildPhase(array(
    'shellPath' => '/bin/sh',
    'shellScript' => 'cp -r "${PROJECT_DIR}/../resources/" "${BUILT_PRODUCTS_DIR}"'
    ));

    foreach ( $filesToAdd as $file ){
    $proj->addFileReference(array(
    'name' => basename($file),
    'path' => $file
    ));
    }

    $proj->save();
    }

What it does is probably self explanatory.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #4 originally posted by codenameone on 2013-07-31T16:47:09.000Z:

Oh.. one more thing. After adding the build phase to the project file, I needed to add this build phase to each target in the project.

This is the PHP code that does this:
// Now we need to add it to our target
$xpath = new DOMXPath($this->dom);
$keys = $xpath->query('//key');
$arr = null;
foreach ( $keys as $key ){
if ( $key->nodeValue == 'buildPhases'){
$arr = $key->nextSibling;
while ( $arr != null and (!$arr instanceof DOMElement) ){
$arr = $arr->nextSibling;
}
break;
}
}

    if ( $arr === null ){
        throw new Exception("No targets with build phases found");
    }

    $strEl = $this->dom->createElement('string');
    $strEl->nodeValue = $buildPhase['key']->nodeValue;
    $arr->appendChild($strEl);

Essentially it looks for a tag like buildPhases which will always be followed by an tag with ID children.

I just add a xxx tag as a child of this where xxx is the key of my build phase. In the example I posted, it would be e4e5a3214a4462a5fd77109d

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #5 originally posted by codenameone on 2013-07-31T16:51:23.000Z:

We are using xcodebuild and going into those scripts probably wouldn't be something we will do in the near future unless we need to completely overhaul everything. There is so much functionality on top of this its just not practical to convert it.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #6 originally posted by codenameone on 2013-07-31T16:58:17.000Z:

I can understand the reluctance to want to tinker. Hopefully at some point you'll have other use cases that will be more compelling for you to make this change. To me, this is a pretty big deal but it's not a show stopper since I don't need to use the build server. The build server preserves package structure for Android, and I can perform offline builds for iOS.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #7 originally posted by codenameone on 2013-07-31T17:16:28.000Z:

I'd like to have a solution for this but I'm looking for something that is guaranteed to be portable. E.g. character type X on os Y might fail for me. JAR solved these things nicely but you are asking to do this with the native structure which I don't want to rely on.
Furthermore, the advantage of being able to package resources into a RES file will solve one of the more painful aspects of HTML programming since we would be able to use multi images for image resources.
No its not the standard but neither is doing something different for every OS, it could work exactly as it does in the simulator which means consistency.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #8 originally posted by codenameone on 2013-07-31T17:32:29.000Z:

If there is a solution to, say, copy a directory structure with subdirectories, etc.. into a resource file and allow them to be addressed using a namespacing mechanism, and have this compatible with native components like the browser component, that that would be sufficient. To me that sounds like a difficult thing to do.

I'm not sure I understand the issue of portability. Can you elaborate on the "character type X on os Y might fail for me" example and how it might relate maintaining directory structure of resources? Are you talking about characters in file names causing problems for the file system?

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #9 originally posted by codenameone on 2013-07-31T18:24:27.000Z:

Adding the hierarchy to the res file should be pretty easy. The only issue would be in supporting the URL type for browsers, I was hoping you would have some input on that assuming I can get an input stream...

E.g. local UTF-8 characters are very common and people use characters like % in file names. I wouldn't mind if this fails or succeeds just that it would act fail/succeed consistently between the device and the simulator.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #10 originally posted by codenameone on 2013-07-31T23:05:29.000Z:

I will have to look into each platform to see if it supports it. It looks like this may be possible with iOS using an NSURLProtocol implementation (but I've just started reading about this).
http://stackoverflow.com/questions/9301611/using-a-custom-nsurlprotocol-with-uiwebview-and-post-requests

For android I have run through some people asking about this, but haven't seen a definitive solution yet.
E.g. https://groups.google.com/forum/#!topic/android-developers/INijIZ4F5G8

It will take some tinkering to find out one way or the other.

The JavaFX web view supports loading resources to a WebView from a Jar file. It does this by using URL.toExternalForm() on the specific resource. But it is able to correctly load sub-resources in the jar's child directories (e.g. if images are in a subpath of the displayed page in the jar file. This tells us that at least it will be possible to create this behaviour in the simulator.

I haven't even touched a Windows phone yet so I don't know where to begin on there.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #11 originally posted by codenameone on 2013-08-01T05:01:06.000Z:

This seems like a bit too much effort. Maybe a better solution is to just seamlessly unpack the resource the first time it is shown into a temporary folder and just work from there?
This should be reasonably portable and require absolutely no change to our build scripts. The only concern I have here is if a resource changes, I wouldn't want to unpack the data every time the app runs.

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #12 originally posted by codenameone on 2013-08-01T06:24:58.000Z:

Just did a bit of research regarding custom URL handlers. It looks like it isn't that hard on iOS, Android, JavaSE, and WinPhone. Haven't found a way yet on JavaME or BlackBerry.

Here are some links:

Android & JavaSE

Mentions using a Custom URLStreamHandlerFactory for loading images:
http://blog.wu-man.com/2012/10/androidimageloader-loading-images.html

setURLStreamHandlerFactory API docs
http://developer.android.com/reference/java/net/URL.html#setURLStreamHandlerFactory(java.net.URLStreamHandlerFactory)

More Discussion of Loading resources from a resource file using custom URL stream handler
http://stackoverflow.com/questions/8938260/url-seturlstreamhandlerfactory
http://stackoverflow.com/questions/861500/url-to-load-resources-from-the-classpath-in-java

More on custom url stream handlers
http://www.unicon.net/node/776

Windows Phone 7 & 8

WebRequest Creator
http://msdn.microsoft.com/en-us/library/windowsphone/develop/system.net.browser.webrequestcreator(v=vs.105).aspx
(Looks like it can be used to register custom protocols)

iOS

Custom NSURLProtocol
http://stackoverflow.com/questions/9301611/using-a-custom-nsurlprotocol-with-uiwebview-and-post-requests

Java ME

Haven't found a way to make custom protocol handlers yet

Unpacking to a temp directory may work... I haven't tried loading any web pages from any locations other than the main bundle on iOS yet... we'd want to make sure we haven't traded one difficult problem for another one :)

@codenameone

This comment has been minimized.

Owner

codenameone commented Mar 27, 2015

Comment #13 originally posted by codenameone on 2013-08-01T09:57:25.000Z:

Interesting... J2ME doesn't have a native browser so naturally you won't find anything.
HTMLComponent has its own URL code which already supports loading from resources.

@codenameone

This comment has been minimized.

Owner

codenameone commented Apr 10, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment