Parsing JSON and XML on Android
The Android platform was developed specifically for mobile phones, and therefore, while it uses Java as a programming language, Android has a variety of differences when compared to the JDK/JRE libraries for Java on desktop machines. As a result, certain Java libraries such as the Jackson JSON and XML Processor that were developed for desktops don't always work cleanly on Android without some customization or configuration.
The following sections discuss how we parse JSON and XML Service Interface for Real-time Information (SIRI) responses on Android using Jackson in the SirRestClient mobile app (this Github project).
Why did we choose Jackson? See these references that illustrate the performance advantages of Jackson over other JSON parsers (1) (2). Also, the existing OneBusAway Android app uses Jackson.
If you want to see a simple example of parsing SIRI JSON or XML responses using Jackson and desktop Java, check out our open-source SiriParserJacksonExample project.
We have several choices when deciding how to use Jackson to process JSON or XML (discussed here in greater detail).
For this project we chose [Jackson data binding] (http://wiki.fasterxml.com/JacksonDataBinding). "Data binding" means that we will use Jackson to process SIRI JSON or XML data and turn it into corresponding Siri Plain Old Java Objects (POJOs) in our Android app's memory. POJOs are simple Java container objects (without any advanced code functionality) that let us easily access data in our app via getter methods such as siri.getServiceDelivery().getResponseTimestamp().
Here's an example of a simple data binding to a Siri POJO, which contains references to many children POJOs, using Jackson:
//Declare the Jackson ObjectMapper
ObjectMapper mapper = new ObjectMapper();
...(a few lines of config here)...
//Make the call to the server to retrieve the JSON from the URL and deserialize it into a Siri object
Siri siri = mapper.readValue(url, Siri.class);
These few lines of code fetch SIRI JSON or XML data from a URL and translate it into a Siri Java object in your Android app. Since the developer doesn't have to implement the low-level parsing code, its easier to update to new SIRI versions when they become available - you just switch out the POJOs with a new version based on the updated SIRI spec, and then update your application code in a few places where you're pulling the new data out of the Siri object.
As you can see, data binding has some advantages over other more manual parsing in that it requires only a few lines of code to execute in an application, and therefore when the SIRI API changes, few changes are required in the main client application code. Instead, Jackson does all the heavy lifting in a very optimized way.
However, for Jackson data binding to work on Android, you need a complete set of SIRI POJOs, representing all the SIRI data types that may occur in a SIRI data response, that compile on Android. So, where do we get these?
Well, getting a set of SIRI POJOs that are compatible with Android is the hard part.
In desktop/web Java, we would take the SIRI schema .XSD file and use Java Architecture for XML Binding (JAXB) tools to convert the schema text document into Java objects that are annotated for JAXB. In fact, a SIRI JAXB project already exists. However, JAXB classes are not compatible with Android due to lack of support for XML annotations and certain XML library classes, as well as several other issues.
The simplest path forward to get SIRI POJOs that work on Android is either:
-
Using tools such as [json gen] (http://jsongen.byingtondesign.com/) to generate POJOs based on SIRI JSON data
-
Stripping the XML annotations and other characteristics that are incompatible with Android from the SIRI JAXB classes to create true POJOs that work on Android (either manually, or via automating ant scripts)
For this project, we initially tried using Option #1 [json gen] (http://jsongen.byingtondesign.com/), but the tool did not seem to generate correct objects. The automated ant script from Option #2 wasn't available at the time we did the conversion. We feel either of these automated processes are definitely worth further examination in the future, as it could save significant time over #2 Manual Process.
Therefore, for this project, we ended up using Option #2 Manual Process and manually stripping out the XML annotations to create a set of SIRI POJOs. The root Siri.java POJO contains all information from a SIRI JSON or XML response via internal objects accessible via getter methods. A more detailed description of the process to remove the Android incompatibilities is available on the onebusaway-siri-api-v13-pojos wiki page.
After following the above process, we get a group of SIRI POJOs that are compatible with Android:
JSON is a compact format for representing data that has become popular on mobile devices due to its relative simplicity and compactness when compared to XML. As a result, we recommend that you use the JSON API when working with a SIRI API, instead of XML.
Here's a snippet of SIRI data formatted in JSON:
{
Siri: {
ServiceDelivery: {
ResponseTimestamp: "2012-08-21T12:06:21.485-04:00",
VehicleMonitoringDelivery: [
{
VehicleActivity: [
{
MonitoredVehicleJourney: {
LineRef: "MTA NYCT_S40",
DirectionRef: "0",
FramedVehicleJourneyRef: {
DataFrameRef: "2012-08-21",
DatedVehicleJourneyRef: "MTA NYCT_20120701CC_072000_S40_0031_S4090_302"
},
JourneyPatternRef: "MTA NYCT_S400031",
PublishedLineName: "S40",
OperatorRef: "MTA NYCT",
OriginRef: "MTA NYCT_200001"
}
}
]
}
]
}
}
Click here for a full SIRI JSON file.
Using Jackson for parsing JSON SIRI responses on Android is fairly straightforward, although it does require a little configuration. If you're using Maven, including these dependencies in your pom.xml to add Jackson JSON libraries:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.1.2</version>
</dependency>
Assuming you've referenced the SIRI POJOs project in your build, you can retrieve and parse SIRI JSON with the below code:
Siri siri = null;
URL url = null;
ObjectMapper mapper = new ObjectMapper();
try {
String urlString = "http://bustime.mta.info/api/siri/vehicle-monitoring.json?" +
"OperatorRef=MT%20A%20NYCT&DirectionRef=0&LineRef=MTA%20NYCT_S40";
url = new URL(urlString);
//Jackson 2.X configuration settings
mapper.configure(DeserializationFeature.UNWRAP_ROOT_VALUE, true);
mapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
mapper.configure(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT, true);
mapper.configure(DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, true);
mapper.configure(DeserializationFeature.READ_ENUMS_USING_TO_STRING, true);
//Tell Jackson to expect the JSON in PascalCase, instead of camelCase
mapper.setPropertyNamingStrategy(new PropertyNamingStrategy.PascalCaseStrategy());
//Make the HTTP request, and deserialize the JSON response into the Siri object
siri = mapper.readValue(url, Siri.class);
} catch(Exception e){
Log.e(TAG, "Error fetching or parsing JSON: " + e);
}
XML is an industry-standard representation of data using a set of <tags>.
Here's a snippet of SIRI data formatted in XML:
<Siri xmlns:ns2="http://www.ifopt.org.uk/acsb" xmlns:ns4="http://datex2.eu/schema/1_0/1_0" xmlns:ns3="http://www.ifopt.org.uk/ifopt" xmlns="http://www.siri.org.uk/siri">
<ServiceDelivery>
<ResponseTimestamp>2012-09-12T09:28:17.213-04:00</ResponseTimestamp>
<VehicleMonitoringDelivery>
<VehicleActivity>
<MonitoredVehicleJourney>
<LineRef>MTA NYCT_S40</LineRef>
<DirectionRef>0</DirectionRef>
<FramedVehicleJourneyRef>
<DataFrameRef>2012-09-12</DataFrameRef>
<DatedVehicleJourneyRef>MTA NYCT_20120902EE_054000_S40_0031_MISC_437</DatedVehicleJourneyRef>
</FramedVehicleJourneyRef>
<JourneyPatternRef>MTA NYCT_S400031</JourneyPatternRef>
<PublishedLineName>S40</PublishedLineName>
<OperatorRef>MTA NYCT</OperatorRef>
<OriginRef>MTA NYCT_200001</OriginRef>
</MonitoredVehicleJourney>
</VehicleActivity>
</VehicleMonitoringDelivery>
<ServiceDelivery>
</Siri>
Click here for a full SIRI XML file.
While XML works well in web and desktop applications, it is incredibly verbose when compared to a more compact representation such as JSON. Mobile devices have limited resources, including battery life and cellular data transfer limitations, that can be negatively affected by large amounts of data transmissions.
As a result, we recommend that you use the JSON API when working with SIRI mobile devices, instead of XML.
XML handling on Android is significantly more difficult than JSON, since Android does not include support for some key XML features, as stated above in the POJOs section. This makes Jackson configuration for XML different from JSON, as we'll see shortly.
We need two main types of Jackson libraries to parse XML:
- Core Jackson libraries - Handle some of the core parsing activities (same libraries used in JSON parsing)
- Jackson XML extensions and XML parser implementation - Add XML parsing capabilities to Jackson, and depend on the core Jackson libraries
Since SIRI uses an "unwrapped" style for lists in the XML, you'll need Jackson v2.1 or higher core libraries to support unwrapped lists (For JSON parsing, you can just use v2.0.6 or higher - no special support required).
Include the Jackson core libraries 2.1.2 in your project via Maven by adding the below info to your pom.xml:
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.1.2</version>
</dependency>
</dependencies>
The above instructions will give you "Dependency #1 - Core Jackson Libraries" in your project.
Now, we need "Dependency #2 - Jackson XML extensions and XML parser implementation".
Here's where things get tricky for Android. Most Java XML parsers are based on the Java Specification Request (JSR) 173 - Streaming API for XML (StAX) Java standard. However, the StAX is not supported on Android. Jackson's XML extension sub-project jackson-dataformat-xml depends on StAX.
We can try to include a library in our Android app for a pure StAX implementation that doesn't relay on the desktop JDK libraries, but then we run into another problem: Android doesn't allow you to include libraries that are in the protected Java namespace, including the javax.xml.* namespace. This creates a problem, since by definition of JSR 173 StAX, StAX implementations include classes in the javax.xml.* namespace.
Our solution is to use the open-source project Jar Jar Links, or JarJar for short, to modify our XML libraries that will be bundled with our app so that they will run on Android.
The libraries modified with JarJar, which are required for the Jackson XML parser jackson-dataformat-xml project, are included in the "libs" directory of the SiriRestClient project instead of being managed through Maven.
Here's a full list of dependencies, in addition to the core Jackson libraries mentioned above, that need to be included in the "libs" directory for parsing XML:
- [StAX API] (http://jcp.org/en/jsr/detail?id=173) - JSR 173 API interface definitions. Also see Wikipedia StAX entry for more info.
- StAX2 API - Experimental expansion of the original StAX API used by Aalto
- Aalto - StAX 1 and 2 XML parser implementation
- jackson-dataformat-xml - integrates the above XML parser APIs and implementations with the core Jackson project
If you're using Maven, you can include the following XML libraries, including those that we modified for Android:
<dependencies>
<dependency>
<groupId>edu.usf.cutr.android.xml</groupId>
<artifactId>jackson-dataformat-xml-android</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>edu.usf.cutr.android.xml</groupId>
<artifactId>stax2-api-android</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>edu.usf.cutr.android.xml</groupId>
<artifactId>stax-api-android</artifactId>
<version>1.0-2</version>
</dependency>
<dependency>
<groupId>edu.usf.cutr.android.xml</groupId>
<artifactId>aalto-xml-android</artifactId>
<version>0.9.8</version>
</dependency>
</dependencies>
<repositories>
<!-- CUTR Android XML libraries Releases -->
<repository>
<id>cutr-releases</id>
<url>https://github.com/CUTR-at-USF/cutr-mvn-repo/raw/master/releases</url>
</repository>
</repositories>
See the Modifying XML Libraries for Android wiki page for example files, scripts, and detailed instructions for generating Android-compatible XML libraries.
Once you've performed all the required steps to get Jackson and the required dependencies in your Android project, you can parse SIRI XML using Jackson with the following code:
Siri siri = null;
URL url = null;
try {
String urlString = "http://bustime.mta.info/api/siri/vehicle-monitoring.xml?" +
"OperatorRef=MT%20A%20NYCT&DirectionRef=0&LineRef=MTA%20NYCT_S40";
url = new URL(urlString);
//Use Aalto StAX implementation explicitly for XML parsing
XmlFactory f = new XmlFactory(new InputFactoryImpl(), new OutputFactoryImpl());
JacksonXmlModule module = new JacksonXmlModule();
/*
* Tell Jackson that Lists are using "unwrapped" style (i.e.,
* there is no wrapper element for list).
* NOTE - This requires Jackson 2.1 or higher
*/
module.setDefaultUseWrapper(false);
XmlMapper xmlMapper = new XmlMapper(f, module);
xmlMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
xmlMapper.configure(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT,true);
xmlMapper.configure(DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, true);
xmlMapper.configure(DeserializationFeature.READ_ENUMS_USING_TO_STRING,true);
//Tell Jackson to expect the XML in PascalCase, instead of camelCase
xmlMapper.setPropertyNamingStrategy(new PropertyNamingStrategy.PascalCaseStrategy());
//Make the HTTP request, and deserialize the XML response into the Siri object
siri = xmlMapper.readValue(url, Siri.class);
} catch(Exception e){
Log.e(TAG, "Error fetching or parsing XML: " + e);
}
Performance (i.e., fast parsing of JSON or XML) is important to us, since it has a direct affect on the amount of time a user has to wait before they can see results fetched from the internet - in our case, real-time transit information. Slower parsing also typically results in longer CPU, screen, and radio usage, which all negatively affect battery life.
We've outlined a few steps you can take to enhance the performance of parsing JSON or XML on the Performance Benchmarks page, which includes test results demonstrating the benefits on real devices! These optimizations are also built into the SiriRestClient library so you can start using them in your own app!
- JAXB problems on Android - http://www.docx4java.org/blog/2012/05/jaxb-can-be-made-to-run-on-android/
- Check out our sample SiriRestClient Android app (this Github project) to see an implementation of Jackson JSON and XML parsing on Android.
- Modifying XML libraries for Android Tutorial - Discusses the challenges of using XML parsing libraries on Android and how to overcome them
- Optimizations and Performance Benchmarks - We've outlined a few steps you can take to enhance the performance of parsing JSON or XML, including test results demonstrating the benefits on real devices! These optimizations are also built into the SiriRestClient library so you can start using them in your own app!