JDOM2 and Android

This page details five things:

0. Statement of support for Android.
1. How to include JDOM2 in your Android project
2. Standard JDOM functionality that is unavailable or limited on Android
3. Standard JDOM functionality that may have suprising results on Android
4. How the JDOM 2.x code is tested on Android

Statement of JDOM Compatibility with Android

JDOM is fully functional on Android. This is 'tecnhically' true. Android itself is signifcantly limited though in the following areas:

• Android has no StAX API and no StAX implementation. As a result, JDOM cannot read from or write to StAX input and output streams
• The default (ExpatParser) DOM and SAX parsers on Android have limitations in three areas: Validation; suppression of Entity Expansion; handling of the content of DocType declarations.
• It appears to be (currently) impossible to find an Android-compatible SAX/DOM parser that does do XML Validation

Including JDOM in your project

Drop the JDOM 'core' jar in your libs folder in your Android project folder. You may need to create the libs folder if you do not already have it (note, it is the libs folder with an 's', not lib).

The following should be considered:

• If you need to use XPath expressions, you will probably want to include the Jaxen jar as well.
• there are some additional notes in later sections of this page that detail other factors you may need to consider. Please read this entire page.

What's Impossible or Limited

StAX related functionality: Android uses a different type of 'Pull-parsing' - search for 'Android XML Pull Parser'. JDOM is not compatible with this parser, and given that the parser is not implemented in regular Java, it will not happen. Further, Android makes it intentionally difficult to implement new classes in the 'java*' packages, so it is essentially impractical to add the API's necessary to include StAX parsing.

Apart from StAX, everything else in JDOM works! That's the good news, but it comes with one big BUT:

JDOM relies on a third-party parser to load XML documents and to validate the input, and the default Android parser is different to the standard Java parser, and it is also different to Xerces. As a consequence, there are a number of JDOM test cases that fail on Android, but these are not the fault of JDOM, but rather they are limitations of Android. See the next section for a description of these issues.

What JDOM functionality has suprising results on Android

The big issue is the lack of a complete XML parser on Android. The default Android parser has significant limitations, and these limitations are apparent. It is possible to override the default parser, but in the long run that does not help much. Still, it makes sense to do two things: list the issues with the default Android parser; and show how to install a 'third party' parser, even though it will not all work...

The default Android Parser

Here is a full breakdown of the failing JUnit test cases when run on Android.

The following issues are notable on the standard XML Parser in Android:

0. The parser does not 'forward' the DOCTYPE's internal-subset to the SAXHandler when SAX Parsing (does not recognize the optional org.xml.sax.ext.DeclHandler specification). The most significant issue related to this is that JDOM is not able to output the file with the same information as the source.
1. Does not implement the required XMLSchema validation (despite the documentation - SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI) throws IllegalArgumentException!!!

The issues related to the XMLSchema are considerable. Of importance are:

0. XMLReaders.DTDVALIDATING will cause a JDOMException if you try to use it.
1. XMLReaders.XSDVALIDATING will cause a JDOMException if you try to use it.
2. XMLReaderXSDFactory will throw JDOMException from the constructor.

A Third-party Parser

It is logical to consider Xerces as the third-party parser, but this issue is complicated enough to require it's own page... Xerces on Android. Note: Using Xerces Does Not Solve All These Issues - it causes an adroid application crash!

Testing JDOM in Android

Android has a few built-in mechanisms for testing code. The most rigorous mechanism is to actually run the full JDOM test harness on an (emulated) Android device. There are a few challenges with this approach and it is valuable to document how the actual tests are run (these are 'logistics issues' purely related to getting the test infrastructure running):

0. Android uses JUnit3 as a 'foundation' for unit testing, and all the (1800+) JDOM unit tests are written in JUnit4 (this issue has been worked around).
1. A number of JDOM tests read and parse XML 'files' (and DTD's, and XSD's, etc.) and the existing mechanisms in the JDOM tests for accessing these files is 'broken' on Android ClassLoader.getSystemResource() (this issue has been worked around).
2. JDOM has 1800 unit tests, but the eclipse/android test interface cannot process more than a few hunded at a time, before the 'RPC' channel overflows and you get a Failed Binder Transaction (this issue is resolved by using the 'ant' build process for Android).
3. Android has some missing functionality which is/was directly referenced by JDOM Unit tests (ResourceBundles)

A Brief WalkThrough

Android testing is set up with two Android projects. The first project is the 'real' project, and the second project is the 'Test' project. The two projects are linked using Android project references. You build the two projects in to seperate android 'apps', and then install both apps on to the Android device. After they are installed, you then run the tests on the device. The test results are returned to the 'calling' process.

A special note must be made about 'resources'. 'Resource' has a number of meanings in Java and Android. In Android, a resource is a special construct that is available as a special static class called 'R'. JDOM uses a different type of resource accessed through ClassLoader.getSystemResource(...), and the ClassLoader.getSystemResource(...) type methods are completely unreliable on Android. After a lot of back-and-forth, including using custom on-load procedures to move Android 'assets' to the 'cache' filesystem, it was discovered that the this.getClass().getResource(...) method is reliable on both Android and Java... see ClassLoader.getSystemResource().

There are some significant logistical problems with setting up the JDOM tests in such an environment:

0. You need both a 'main' and a 'test' Android project
1. You need to have the jars that the tests use installed in the 'main' app.
2. You need all 'resources' to be in the main app.
3. All the 'resources' need to be made accessible in some form that can be expressed as a URL.
4. You need all the JUnit tests to be in the form of JUnit3 tests
5. The actual JUnit tests need to be installed in the test app.

All these requirements have been met in a way that makes it almost easy.... ;-)

Putting it together - The '13 Steps'

The testing process has been almost entirely automated now. In essence, if you want to run the tests yourself, you need to:

0. Install a Java JDK, and set up your JAVA_HOME and PATH.
1. Install Apache 'ant' and set up your ANT_HOME and PATH.
2. Install the Android SDK and add <AndroidDirectory>/tools to your PATH
3. Create a directory to work in. I recommend something like C:\jdom for windows, or $HOME/jdom for UNIX.
4. Choose a name for your Android project. I use 'JUT' for 'JDOM Unit Testing'
5. Create the 'main' Android project: android create project --target android-15 --name JUT --path ./JUT --activity JUTActivity --package org.jdom2
6. Create the 'test' Android project: android create test-project -m ../JUT -n JUTTest -p JUTTest
7. Get the JDOM source code from GitHub: git clone git://github.com/hunterhacker/jdom.git
8. 'Populate' the projects with JDOM tests, libraries, and resources. This is a little piece of 'magic'.
   1. cd jdom
   2. ant android -Dandroid=../ -Dandroid.project=JUT -Dversion=2.android
9. Build your Android projects with:
   1. cd ../JUTTest
   2. ant debug
10. Start an Android virtual device (see the Android DevKit for instructions).
11. Install the JUT apps: ant installt
12. Run the tests: ant test

The Magic JDOM 'android' ant target

This is the part that interfaces the JDOM project with Android. It does the following:

0. compiles JDOM
1. builds the JDOM jars
2. Copies the JDOM-JUnit test jar (test classes and resources) in to the main Android project. These classes are the original JUnit4 tests.
3. Copies the core and contrib JDOM jars to the main Android project.
4. Copies all the 'support' jars to the main Android project (JUnit4 library, Xalan, Jaxen, etc.)
5. Creates JUnit3 wrapper classes and methods in the 'test' Android project for each JUnit4 test. It first applies a limited amount of 'filtering' for some tests (like it does not test StAX-based code).

Note: a conversion layer is created to translate the JUnit4 tests in JDOM to JUnit3 tests in Android. This layer essentially wraps each JUnit4 test in whatever means necessary to make it behave as if it was called from a JUnit4 TestRunner. You can inspect the generated JUnit3 source code in the test Android project. The program that does this conversion is included in the JDOM contrib section: org.jdom2.contrib.android.TranslateTests

Test Results

As this testing process was built, a number of technical issues in both Android and JDOM became apparent. Each of these issues could be called a 'bug' in JDOM's run-anywhere status. The following JDOM issues have been resolved:

0. JUnit tests were using ResourceBundles which do not work on Android. These were replaced with equivalent, but simpler mechanisms.
1. The XMLReaders Enumeration is a critical, and significant feature in JDOM 2.x. This Enumeration was failing to initialize through because the default javax.xml.validate.SchemaFactory class does not implement an XSD Schema validation mechanism (which is against the specifications for SchemaFactory). Because the enumeration failed to initialize, all references to the Enumeration threw ClassNotFound errors!
2. The LineSeparator Enumeration is also critical. It's construction order appears to be subtly different between Java and Android. This caused the Android implementation to dereference enum members which had not yet been constructed. Changing the way that the LineSeparator DEFAULT member is constructed allows the process to work well in both Android and Java.
3. Some of the JDOM tests are for 'negative' testing of the handling of invalid Unicode 'surrogate' sequences. It is not possible to load/compile these invalid unicode sequences in Android, so these tests have been skipped.

Android also has some issues:

0. AssertionError incorrectly processes the 'initCause()' mechanism causing an IllegalStateException when running JDOM Unit tests. The JDOM tests have been reconfigured to work around this issue, but a bug has been filed against Android too: http://code.google.com/p/android/issues/detail?id=29378

Now that we actually have JUnit tests running on android, we can analyze the results.

The Android XML Parsers

Android ships with its own XML Parser. JDOM typically recommends using the Xerces parsers, and the JDOM tests are all written to pass when using Xerces. Android does not use the Xerces parse, and some differences are a result of that. It is thus useful to document what does not work with the Android built-in XML parser.

StAX support missing

The StAX test code simply cannot run on Android. The core StAX Libraries are not available. The StAX tests cause the Android test project fail to compile (the tests throw XMLStreamException which is not available). As a result, the StAX tests are not even generated.

DocType Internal Subset

Xerces SAX Parser provides any internal subset data from the DocType as part of the SAX Build process. The Android SAX parser does not. This is causing about a dozen tests to fail. This issue is not critical though, I don't think.

ExpandExternalEntities cannot be toggled.

The Android parser ignores the feature to turn off entity expansion (expand-general-entities). As a consequence, the Unit tests that expect to parse a document and get a JDOM EntityRef in the results are going to fail.