Parsing JSON and XML on Android
The Android platform was developed specifically for mobile phones, and therefore, while it uses Java as a programming language, Android has a variety of differences when compared to the JDK/JRE libraries for Java on desktop machines. As a result, certain Java libraries such as the Jackson JSON and XML Processor that were developed for desktops don't always work cleanly on Android without some customization or configuration.
The following sections discuss how we parse JSON and XML Service Interface for Real-time Information (SIRI) responses on Android using Jackson in the SirRestClient mobile app (this Github project).
Why did we choose Jackson? See these references that illustrate the performance advantages of Jackson over other JSON parsers (1) (2). Also, the existing OneBusAway Android app uses Jackson.
If you want to see a simple example of parsing SIRI JSON or XML responses using Jackson and desktop Java, check out our open-source SiriParserJacksonExample project.
We have several choices when deciding how to use Jackson to process JSON or XML (discussed here in greater detail).
For this project we chose [Jackson data binding] (http://wiki.fasterxml.com/JacksonDataBinding). "Data binding" means that we will use Jackson to process SIRI JSON or XML data and turn it into corresponding Siri Plain Old Java Objects (POJOs) in our Android app's memory. POJOs are simple Java container objects (without any advanced code functionality) that let us easily access data in our app via getter methods such as siri.getServiceDelivery().getResponseTimestamp().
Here's an example of a simple data binding to a Siri POJO, which contains references to many children POJOs, using Jackson:
//Declare the Jackson ObjectMapper
ObjectMapper mapper = new ObjectMapper();
...(a few lines of config here)...
//Make the call to the server to retrieve the JSON from the URL and deserialize it into a Siri object
Siri siri = mapper.readValue(url, Siri.class);
These few lines of code fetch SIRI JSON or XML data from a URL and translate it into a Siri Java object in your Android app. Since the developer doesn't have to implement the low-level parsing code, its easier to update to new SIRI versions when they become available - you just switch out the POJOs with a new version based on the updated SIRI spec, and then update your application code in a few places where you're pulling the new data out of the Siri object.
As you can see, data binding has some advantages over other more manual parsing in that it requires only a few lines of code to execute in an application, and therefore when the SIRI API changes, few changes are required in the main client application code. Instead, Jackson does all the heavy lifting in a very optimized way.
However, for Jackson data binding to work on Android, you need a complete set of SIRI POJOs, representing all the SIRI data types that may occur in a SIRI data response, that compile on Android. So, where do we get these?
Well, getting a set of SIRI POJOs that are compatible with Android is the hard part.
In desktop/web Java, we would take the SIRI schema .XSD file and use Java Architecture for XML Binding (JAXB) tools to convert the schema text document into Java objects that are annotated for JAXB. In fact, a SIRI JAXB project already exists. However, JAXB classes are not compatible with Android due to lack of support for XML annotations and certain XML library classes, as well as several other issues.
The simplest path forward to get SIRI POJOs that work on Android is either:
-
Using tools such as [json gen] (http://jsongen.byingtondesign.com/) to generate POJOs based on SIRI JSON data
-
Stripping the XML annotations and other characteristics that are incompatible with Android from the SIRI JAXB classes to create true POJOs that work on Android (either manually, or via or automating via ant scripts)
For this project, we initially tried using Option #1 [json gen] (http://jsongen.byingtondesign.com/), but the tool did not seem to generate correct objects. The automated ant script from Option #2 wasn't available at the time we did the conversion. We feel either of these automated processes are definitely worth further examination in the future, as it could save significant time over #2 Manual Process.
Therefore, for this project, we ended up using Option #2 Manual Process and manually stripping out the XML annotations to create a set of SIRI POJOs. The root Siri.java POJO contains all information from a SIRI JSON or XML response via internal objects accessible via getter methods. A more detailed description of the process to remove the Android incompatibilities is available on the onebusaway-siri-api-v13-pojos wiki page.
After following the above process, we get a group of SIRI POJOs that are compatible with Android:
JSON is a compact format for representing data that has become popular on mobile devices due to its relative simplicity and compactness when compared to XML. As a result, we recommend that you use the JSON API when working with a SIRI API, instead of XML.
Here's a snippet of SIRI data formatted in JSON:
{
Siri: {
ServiceDelivery: {
ResponseTimestamp: "2012-08-21T12:06:21.485-04:00",
VehicleMonitoringDelivery: [
{
VehicleActivity: [
{
MonitoredVehicleJourney: {
LineRef: "MTA NYCT_S40",
DirectionRef: "0",
FramedVehicleJourneyRef: {
DataFrameRef: "2012-08-21",
DatedVehicleJourneyRef: "MTA NYCT_20120701CC_072000_S40_0031_S4090_302"
},
JourneyPatternRef: "MTA NYCT_S400031",
PublishedLineName: "S40",
OperatorRef: "MTA NYCT",
OriginRef: "MTA NYCT_200001"
}
}
]
}
]
}
}
Click here for a full SIRI JSON file.
Using Jackson for parsing JSON SIRI responses on Android is fairly straightforward, although it does require a little configuration. If you're using Maven, including these dependencies in your pom.xml to add Jackson JSON libraries:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.0.6</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.0.6</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.0.6</version>
</dependency>
Assuming you've referenced the SIRI POJOs project in your build, you can retrieve and parse SIRI JSON with the below code:
Siri siri = null;
URL url = null;
ObjectMapper mapper = new ObjectMapper();
try {
String urlString = "http://bustime.mta.info/api/siri/vehicle-monitoring.json?" +
"OperatorRef=MT%20A%20NYCT&DirectionRef=0&LineRef=MTA%20NYCT_S40";
url = new URL(urlString);
//Jackson 2.X configuration settings
mapper.configure(DeserializationFeature.UNWRAP_ROOT_VALUE, true);
mapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
mapper.configure(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT, true);
mapper.configure(DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, true);
mapper.configure(DeserializationFeature.READ_ENUMS_USING_TO_STRING, true);
//Tell Jackson to expect the JSON in PascalCase, instead of camelCase
mapper.setPropertyNamingStrategy(new PascalCaseStrategy());
//Make the HTTP request, and deserialize the JSON response into the Siri object
siri = mapper.readValue(url, Siri.class);
} catch(Exception e){
Log.e(TAG, "Error fetching or parsing JSON: " + e);
}
In Jackson v2.0.6, you need to add your own PascalCaseStrategy class:
public class PascalCaseStrategy extends PropertyNamingStrategyBase {
/**
* Converts camelCase to PascalCase
*
* For example, "responseTimestamp" would be converted to
* "ResponseTimestamp".
*
* This allows Jackson to expect the PascalCase version of the class
* field and method names instead of the camelCase version.
*
* @param input formatted as camelCase string
* @return input converted to PascalCase format
*/
@Override
public String translate(String input) {
// Replace first lower-case letter with upper-case equivalent
return input.substring(0, 1).toUpperCase() + input.substring(1);
}
}
In Jackson v2.1, Jackson includes PascalCaseStrategy, so you won't need to include your own, and the line to set the PascalCaseStrategy can look like this:
mapper.setPropertyNamingStrategy(new PropertyNamingStrategy.PascalCaseStrategy());
XML is an industry-standard representation of data using a set of <tags>.
Here's a snippet of SIRI data formatted in XML:
<Siri xmlns:ns2="http://www.ifopt.org.uk/acsb" xmlns:ns4="http://datex2.eu/schema/1_0/1_0" xmlns:ns3="http://www.ifopt.org.uk/ifopt" xmlns="http://www.siri.org.uk/siri">
<ServiceDelivery>
<ResponseTimestamp>2012-09-12T09:28:17.213-04:00</ResponseTimestamp>
<VehicleMonitoringDelivery>
<VehicleActivity>
<MonitoredVehicleJourney>
<LineRef>MTA NYCT_S40</LineRef>
<DirectionRef>0</DirectionRef>
<FramedVehicleJourneyRef>
<DataFrameRef>2012-09-12</DataFrameRef>
<DatedVehicleJourneyRef>MTA NYCT_20120902EE_054000_S40_0031_MISC_437</DatedVehicleJourneyRef>
</FramedVehicleJourneyRef>
<JourneyPatternRef>MTA NYCT_S400031</JourneyPatternRef>
<PublishedLineName>S40</PublishedLineName>
<OperatorRef>MTA NYCT</OperatorRef>
<OriginRef>MTA NYCT_200001</OriginRef>
</MonitoredVehicleJourney>
</VehicleActivity>
</VehicleMonitoringDelivery>
<ServiceDelivery>
</Siri>
Click here for a full SIRI XML file.
While XML works well in web and desktop applications, it is incredibly verbose when compared to a more compact representation such as JSON. Mobile devices have limited resources, including battery life and cellular data transfer limitations, that can be negatively affected by large amounts of data transmissions.
As a result, we recommend that you use the JSON API when working with SIRI mobile devices, instead of XML.
XML handling on Android is significantly more difficult than JSON, since Android does not include support for some key XML features, as stated above in the POJOs section. This makes Jackson configuration for XML different from JSON, as we'll see shortly.
We need two main types of Jackson libraries to parse XML:
- Core Jackson libraries - Handle some of the core parsing activities (same libraries used in JSON parsing)
- Jackson XML extensions and XML parser implementation - Add XML parsing capabilities to Jackson, and depend on the core Jackson libraries
However, since SIRI uses an "unwrapped" style for lists in the XML, you'll need Jackson v2.1 core libraries to support unwrapped lists, which are still in pre-release as of Sept. 2012 (For JSON parsing, you can just use the v2.0.6 release - no special support required). Fortunately, we can access the Jackson v2.1 core libraries pre-release 2.1.0-SNAPSHOT version from the Sonatype repository.
Include the Jackson core libraries 2.1.0-SNAPSHOT in your project via Maven by adding the below info to your pom.xml:
<dependencies>
<!-- SIRI XML uses unwrapped lists, so we need Jackson v2.1, which is pre-release and only available
from the Sonatype repo as of 9/19/2012 (see <repository> entry below). After Jackson v2.1 is
released and is available from the central Maven repository, we should use that repo -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.1.0-SNAPSHOT</version>
</dependency>
</dependencies>
<!-- For the Jackson SNAPSHOTS -->
<repositories>
<repository>
<id>sonatype-nexus-snapshots</id>
<url>http://oss.sonatype.org/content/repositories/snapshots</url>
</repository>
</repositories>
The above instructions will give you "Dependency #1 - Core Jackson Libraries" in your project.
Now, we need "Dependency #2 - Jackson XML extensions and XML parser implementation".
Here's where things get tricky for Android. Most Java XML parsers are based on the Java Specification Request (JSR) 173 - Streaming API for XML (StAX) Java standard. However, the StAX is not supported on Android. Jackson's XML extension sub-project jackson-dataformat-xml depends on StAX.
We can try to include a library in our Android app for a pure StAX implementation that doesn't relay on the desktop JDK libraries, but then we run into another problem: Android doesn't allow you to include libraries that are in the protected Java namespace, including the javax.xml.* namespace. This creates a problem, since by definition of JSR 173 StAX, StAX implementations include classes in the javax.xml.* namespace.
Our solution is to use the open-source project Jar Jar Links, or JarJar for short, to modify our XML libraries that will be bundled with our app so that they will run on Android.
The libraries modified with JarJar, which are required for the Jackson XML parser jackson-dataformat-xml project, are included in the "libs" directory of the SiriRestClient project instead of being managed through Maven.
Here's a full list of dependencies, in addition to the core Jackson libraries mentioned above, that need to be included in the "libs" directory for parsing XML:
- [StAX API] (http://jcp.org/en/jsr/detail?id=173) - JSR 173 API interface definitions. Also see Wikipedia StAX entry for more info.
- StAX2 API - Experimental expansion of the original StAX API used by Aalto
- Aalto - StAX 1 and 2 XML parser implementation
- jackson-dataformat-xml - integrates the above XML parser APIs and implementations with the core Jackson project
See the Modifying XML Libraries for Android wiki page for example files, scripts, and detailed instructions for generating Android-compatible XML libraries.
Once you've performed all the required steps to get Jackson and the required dependencies in your Android project, you can parse SIRI XML using Jackson with the following code:
Siri siri = null;
URL url = null;
try {
String urlString = "http://bustime.mta.info/api/siri/vehicle-monitoring.xml?" +
"OperatorRef=MT%20A%20NYCT&DirectionRef=0&LineRef=MTA%20NYCT_S40";
url = new URL(urlString);
//Use Aalto StAX implementation explicitly for XML parsing
XmlFactory f = new XmlFactory(new InputFactoryImpl(), new OutputFactoryImpl());
JacksonXmlModule module = new JacksonXmlModule();
/*
* Tell Jackson that Lists are using "unwrapped" style (i.e.,
* there is no wrapper element for list).
* NOTE - This requires Jackson 2.1, which is still pre-release
* as of 9/12/2012
*/
module.setDefaultUseWrapper(false);
XmlMapper xmlMapper = new XmlMapper(f, module);
xmlMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
xmlMapper.configure(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT,true);
xmlMapper.configure(DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, true);
xmlMapper.configure(DeserializationFeature.READ_ENUMS_USING_TO_STRING,true);
//Tell Jackson to expect the XML in PascalCase, instead of camelCase
xmlMapper.setPropertyNamingStrategy(new PropertyNamingStrategy.PascalCaseStrategy());
//Make the HTTP request, and deserialize the XML response into the Siri object
siri = xmlMapper.readValue(url, Siri.class);
} catch(Exception e){
Log.e(TAG, "Error fetching or parsing XML: " + e);
}
Note that since we need Jackson v2.1 for the module.setDefaultUseWrapper(false) method, we can go ahead and use the included PascalCaseStrategy as well, and don't need to include our own.
TODO - Cover Android HTTPURLConnection, ObjectMapper vs. ObjectReader, (maybe XML vs. JSON if we have data) etc.
- Android HttpURLConnection object - http://developer.android.com/reference/java/net/HttpURLConnection.html
- Comparison of Android's HTTP clients - http://android-developers.blogspot.com/2011/09/androids-http-clients.html
- or use Jackson's internal HTTP connection
TODO - Android - cover disableConnectionReuseIfNecessary()
TODO - include test results here
- JAXB problems on Android - http://www.docx4java.org/blog/2012/05/jaxb-can-be-made-to-run-on-android/
- Check out our sample SiriRestClient Android app (this Github project) to see an implementation of Jackson JSON and XML parsing on Android.
- Modifying XML libraries for Android Tutorial - Discusses the challenges of using XML parsing libraries on Android and how to overcome them