Skip to content

Latest commit

 

History

History
165 lines (105 loc) · 19.3 KB

AppscaleAPI.md

File metadata and controls

165 lines (105 loc) · 19.3 KB
Raw

Introduction

This page describes how to use the API and install it in an Appscale deployment, as well as how to modify and extend the API.

Using the API

Note: For the current version of the API, a slightly modified

Installing in Appscale

Note: The API was designed for Appscale-1.11.0, no guarantees are made for compatibiliy with later versions, although as long as the AppServer_Java doesn't change everything should work as expected.

Note: At the time being, the API does not yet have the ability to store history data persistently in the database, so it is recommended to configure Appscale so it does not replicate your App, different Apps on different servers should not be a problem as long as you don't want to share history data.

Note: The CoAP version used in the Gateway and the API is coap-12; as the implementation of the protcol stack, californium-coap-12 is used, should the Gateway get upgraded to a newer version, please see Changing the Californium Version on how to upgrade the API

To build the API, you need:

  • Gradle 1.8 or later

To install the API in Appscale, simply clone the repository and execute the Gradle task :iotsys-appscale:assemble to build a jar file. When the task has executed successfully, you will find the following four files in the iotsys-appscale/build/deploy folder:

  • build.xml - ant file for inserting the API into the existing server jars
  • iotsys-appscale-{version}.jar - API code that will be inserted into the existing Appscale App-Server jars
  • com.google.appengine.spi.FactoryProvider - Will be inserted into the server jars
  • com.google.appengine.tools.development.LocalRpcService - Will be inserted into the server jars

Copy these four files into the AppServer_Java/appengine-java-sdk-repacked folder of your Appscale deployment and execute ant to run the build.xml in the directory. After Ant has finished successfully, every new App that gets deployed to Appscale will be able to make use of the IoTSyS API.

Creating Web Apps

Appscale is designed to run Google App Engine Apps, this means that you can create a Google Appengine project (e.g. in Eclipse, with google plugin) and simply add iotsys-appscale/deploy/iotsys-appscale-{version}.jar as a compile-time library to your project. You will not be able to test the project directly with the project's testserver, see Testing the API for information on how to test your app locally.

If you don't want to or cannot use the Google toolkit, you can create the web-app file structure in your project yourself, all you need for a project to work in Appscale are:

  • war/WEB-INF/appengine-web.xml
  • war/WEB-INF/web.xml

To get all needed libraries, first execute gradle prepareTestserver from the project directory, then execute gradle in the test subdirectory to repack the google testserver with the custom API code, alternatively, you can specify the gradle target updateTestserver to achieve the same functionality. After the taks has finished, you can use the jar files in iotsys-appscale/test/appengine-java-sdk-repacked/lib/user and iotsys-appscale/test/appengine-java-sdk-repacked/lib/shared as your project's compile-time libraries. These same libraries will be copied into your App's war/WEB-INF/lib folder when you deploy it to Appscale, with this approach, you won't need to add iotsys-appscale-{version}.jar as an extra library, as it is already packaged with the other libraries.

Testing the API

For running a web-app (a demo or your own app), you need a version of the google testserver repacked with the API code, to get that, all you need to do is execute gradle in the test subdirectory of the project, this will assemble the API jar, download the testserver, extract it and repack it with the API code.

Using the Demos

Before you can use the demos in Eclipse, please follow the instructions below.

To run a demo, execute the gradle task compileJavaDemos in the iotsys-appscale project, this task sets up the testserver by repacking it with the needed files from the iotsys API and compiles the demos' code using those repacked files.

You can then use iotsys-appscale/run_demo.sh <demo-name> [port] (where demo-name is the directory name of a demo, e.g. ModifyObjects) to start a demo.

After the needed libraries have been copied into the demos' war/WEB-INF/lib directory (after compileJavaDemos, or more specifically after prepareDemos), eclipse metadata for the demos can be generated by executing the gradle task eclipseDemos from the project directory.

Note: If you generate eclipse metadata before the libraries have been copied, eclipse will not recognise the libraries and you will have to import them manually

The testserver can be cleaned by executing the gradle task cleanTestserver, this task will however not delete the original GAE testserver code located in appengine-java-sdk-repacked, to reset the whole directory to the default state, execute cleanAllTestserver.

The demos can be cleaned by executing cleanDemos

Testing your own App

To test your own application, you first need to copy all necessary jar files from iotsys-appscale/test/appengine-java-sdk-repacked/lib/user and iotsys-appscale/test/appengine-java-sdk-repacked/lib/shared into your own app's war/WEB-INF/lib directory. After copying the jar files, you can start the testserver by executing iotsys-appscale/test/appengine-java-sdk-repacked/bin/dev_appserver.sh [--port=xxxx] <path to your app>.

Modifying and Extending the API

General Layout

Appscale uses a modified version of the Google testserver for testing Google App Engine applications as its App-Server. Google APIs have the following general structure (using the classes of the IoTSyS API:

  • IotsysServiceFactoryProvider: a class annotated with the com.google.appengine.spi.ServiceProvider interface which uses as its value the com.google.appengine.spi.FactoryProvider interface, indicating that it can be used to get a factory for the APIs service, in this case an IIotsysServiceFactory instance (IotsysServiceFactoryImpl); this class must be registered in the com.google.appengine.spi.FactoryProvider file located in META-INF/services of the corresponding jar files used by the server.
  • IIotsysServiceFactory: an interface defining methods to get the actual API service (IotsysService)
  • IotsysServiceFactoryImpl: the implementation of the IIotsysServiceFactory interface, returns an implementation of the actual API service (IotsysServiceImpl)
  • IotsysServiceFactory: a class used by the user to get the service he/she wants; uses com.google.appengine.spi.ServiceFactoryFactory#getFactory(Class<?>), a google class with a method for retrieving the ServiceFactory for the specific API and then creating the service by invoking the factory's getIotsysService() method.
  • IotsysService: an interface specifying the capabilities of the API service; used by the user.
  • IotsysServiceImpl: the implementation of the API service interface
  • LocalIotsysService: a class annotated with the ServiceProvider interface which uses as its value the com.google.appengine.tools.development.LocalRpcService interface and extending com.google.appengine.tools.development.AbstractLocalRpcService; the class has to be registered in the com.google.appengine.tools.development.LocalRpcService file located in META-INF/services of the corresponding jar files used by the server. This class is instantiated by google code and used when calls to the same package (getPackage() are made by user-space classes (most calls of the API services are forwarded to corresponding LocalRpcService of that API); all calls to this class are made via the com.google.apphosting.api.ApiProxy and have elevated security rights, which enables those calls to write to files or open sockets.
  • A Protocol Buffer class specifying API specific messages wich are passed between user-space and server-space code (com.google.apphosting.api.IotsysServicePb.

An API contains user-space code and server-space (elevated security rights) code, to be conform with other APIs, the server-space classes are located in com.google.appengine.api.iotsys.dev.*, user-space classes are located in com.google.appengine.api.iotsys.* excluding the dev package. When calls to the API service are made, these calls are forwarded to the ApiProxy which eventually loads the LocalIotsysService identified by the package name returned from LocalIotsysService#getPackage() and calls the method given to the ApiProxy using reflection. The ApiProxy takes as arguments the package name of the needed service, the method name to call and the bytes from a valid Protocol Buffer, it looks at the Protocol Buffer class needed by the loaded service's method and uses this class to generate a Protocol Buffer from the given bytes, this buffer is then given as input argument to the loaded service.

In this API, the ApiProxy is called from com.google.appengine.api.iotsys.IotsysConnectionProxy to make CoAP request to a gateway and from com.google.appengine.api.iotsys.IotsysHistoryProxy to create threads.

Note that an API can only have one instance of LocalRpcService which receives calls from the ApiProxy, in the case of this API, this is com.google.appengine.api.iotsys.dev.LocalIotsysService.

For integrating the API into the server code, the following server jar files have to be updated with the custom API code (paths are based on the {server-root}/lib directory):

  • impl/appengine-api.jar: has to be updated with the user-space classes and the Protocol Buffer class(es). Also, the FactoryProvider (IotsysServiceFactoryProvider) needs to be included in the META-INF/services/com.google.appengine.spi.FactoryProvider file.
  • user/appengine-api-1.0-sdk-1.8.0.jar: needs to be updated with the same files mentioned above.
  • impl/appengine-api-labs.jar: needs to be updated with all server-space classes, this includes all library-code needed by the API.
  • user/appengine-api-labs-1.8.0.jar: needs to be updated with the same files mentioned above.
  • impl/appengine-api-stubs.jar: needs to be updated with the same files mentioned above; also, the LocalIotsysService needs to be included in the META-INF/services/com.google.appengine.tools.development.LocalRpcService file.

Californium

Note: For the current version of the API, californium-coap-12 is used

Changes in Californium

To use Californium in a web-based environment, a few changes had to be made:

  • changed ch.ethz.inf.vs.californium.util.Properties, this class no longer inherits from java.util.Properties, calls to load(String) and store(String) have no effect, and storing and loading properties now happens in a simple Map object. This prevents writing properties to files on the webserver, default properties are used for each request.
  • changed the default property "MAX_RETRANSMIT" in ch.ethz.inf.vs.californium.util.Properties#init() to 2 to reduce the time a HTTP request can take.
  • changed ch.ethz.inf.vs.californium.coap.CommunicatorFactory; getCommunicator() now returns a new Communicator object upon each invocation of the method and declares SocketException and IOException to be thrown instead of just exiting and thereby crashing the server. Other classes that were effected by the new throws declaration were modified by catching the two exceptions at each invocation of the method.

Changing the Californium Version

Californium is only used in com.google.appengine.api.iotsys.dev.comm.coap.SynchronousCoapChannelImpl, when upgrading to a newer version, just update this class, especially SynchronousCoapChannelImpl#sendRequest(Request), accordingly.

Google Protcol Buffers

Google Protocol Buffers (Protobufs for short) are an easy way to design your own protocol. They are language independent (currently supporting Java, Python, C++) and are created by the protobuf-compiler (protoc), a tool provided by google. Protobufs can be easily and efficiently transferred over a network, so they are used in the Google App Engine and also by the APIs in the testserver. Because the testserver is used by Appscale as the App-Server, Protobufs had to be used in the API.

As shown above, protobufs are used as input arguments for the server-side LocalRpcService implementation of the APIs, these protobufs are forwarded to the server-side code by the ApiProxy which takes the bytes of a protobuf as argument. These bytes are created by creating a respective protobuf object, setting the fields and calling its toByteArray() method, the ApiProxy then gets the corresponding protobuf class based on the input protobuf class of the LocalRpcService (LocalIotsysService) and calls its static parseFrom(byte[]) method to create the input argument for the LocalRpcService. The return protobuf object traverses the whole process in the other direction.

The protobufs in the real App Engine are probably used to communicate with remote servers, it is understandable that the same frontend is used for the testserver although it seems a little bit confusing to convert from byte arrays to Protobufs and back so often.

To add new features to the API, the protobuf code probably needs to be updated to be able to carry the information of the new features to the LocalIotsysService. To do this, modify the iotsysservice.proto file in the project root and compile it using compile_proto.sh, to do this, you need to have the protoc installed, see the above link on how to download and install it. The current testserver needs protobufs compiled with protoc version 2.5. If you get strange runtime exceptions occurring in the protobuf file, you should check and update your protoc version and compile the file again.

New Protocols and Formats

New protocols and formats for communicating with a server can be added to the API relatively easy.

To add a new protocol (e.g. HTTP):

  • create an new class extending the com.google.appengine.api.iotsys.dev.comm.TransportProtocol class and implement the missing methods for making requests.
  • assign a new unique integer identifier to the protocol in com.google.appengine.api.iotsys.comm.Protocol so that the user can specify that he wants to use this protocol.
  • add the identifier and the class of the new protocol to the map in com.google.appengine.api.iotsys.dev.comm.ProtocolRegistry so that the class can be instantiated by the server-side code when needed.

Note: The protocol gets instantiated using reflection and an empty constructor, so your new protocol class must have an empty constructor

To add a new format (e.g. JSON):

  • create a new class extending the com.google.appengine.api.iotsys.dev.comm.TransportFormat class and implement the missing methods.
  • assign a new unique integer identifier (for example the media type of the format) to the format in com.google.appengine.api.iotsys.comm.Format so that the user can specify that he wants to use this format.
  • add the identifier and the class of the new format to the map in com.google.appengine.api.iotsys.dev.comm.FormatRegistry so that the class can be instantiated when such a format is requested.

Note: The format gets instantiated using reflection and an empty constructor, so your new format class must have an empty constructor

Note: If you do not want the user to be able to use your new format in requests, do not put the identifier into com.google.appengine.api.iotsys.comm.Format. This can be the case when you only want to be able to receive that type of format but do not want the user to send requests in this format.

Note: A format for the response content of a request is created based on its media type, if you do not use a valid media type as the identifier of the format, the format can not be automatically instantiated based on the response media type. If you only want to send messages in this format and do not expect to receive content which needs the new format to resolve it, this should be no problem.

New Objects

If you plan on adding new objects to the API, you should keep in mind:

  • If you want to add a new object that should be instantiated based on its obix object class, inherit from com.google.appengine.api.iotsys.object.IotObject.
  • If you want to add a new object that should be instantiated based on its explicit contract, inherit from com.google.appengine.api.iotsys.object.IotContractObject.
  • If you need to initialize your own fields from an obix object or a protocol buffer right after instantiation of your object, override the methods IotObject#initialize(Obj) and IotObject#initialize(IotsysServicePb.IotObjectProto), in these methods, you can assign your own fields based on the values of the arguments, but do not forget to call the super method to let the base object initialize.
  • If you need to send your own fields via protobuffer or obix object, override the methods IotObject#writeToObj(Obj) and/or IotObject#writeToProtobuf(IotsysServicePb.IotObjectProto.Builder), there you can add your own fields to the obix object or the protocol buffer, but do not forget to call the super method to write the standard fields into the protobuf/obj.
  • If you have your own fields that need to be updated after a refresh or write to the object based on the object returned from the server/gateway, override IotObject#merge(IotObject), the given object is the object the server returns when invoking IotObject#refresh() or IotObject#write() so it should be of the same type as your object. Do not forget to call the super method to let the base object merge; call the super method last to ensure your postInit() method works the correct way (see below).
  • Lastly, if you want to perform configuration of your object based on its children or fields, override IotObject#postInit(), this method gets called after the object has been fully assembled (fields set, children added). Do not forget to call the super method to configure the base object.

When you have your object class, you need to ensure that it gets instantiated and transferred through the API correctly, to ensure this, you need to:

  • add a new unique value to the protocol buffer to identify you object type after converting it to protocol buffer and back, to do this, add a new value to the IotObjectType enum in {project-root}/iotsysservice.proto and compile the protobuf, see Google Protocol Buffers on how to compile the protobuf.
  • add the new type and the class of your new object to com.google.appengine.api.iotsys.comm.ProtobufConverter, so that it can be converted from object to protobuf and back.
  • If your class should be instantiated based on the class of an obix object, add the obix object's class and your object's class to the iotObjectForObject map in com.google.appengine.api.iotsys.dev.comm.obix.ObixConverter.
  • If your class should be instantiated based on the explicit contract of a received obix object, add the contract and your class to the iotObjectForContract map in com.google.appengine.api.iotsys.dev.comm.obix.ObixConverter.

Quick Start

iotsys-appscale> gradle assemble
iotsys-appscale> gradle prepareTestserver
iotsys-appscale> gradle compileJavaDemos