Troubleshooting

Sean Barbeau edited this page Aug 22, 2014 · 39 revisions

How can I check if OneBusAway is running correctly?

If youve installed and configured OneBusAway, but arent sure if everything is running correctly, here are some things you can check.

For all the following examples, we assume you've deployed OneBusAway at http://localhost:8080/. This is the default for the OneBusAway Quick-Start application.

If you've deployed OneBusAway at a different location, update the paths appropriately. For example, if you're using the Developer Installation Guide with Eclipse, you'll need to add the context path for each web app (e.g., http://localhost:8080/onebusaway-api-webapp/)

Checking the Logs for Exceptions

When you start up any of the webapps, they produce a lot of logging messages. Most of the messages are just info diagnostics that you can safely ignore. Unfortunately, sometimes an exception that will prevent the application from running properly can be buried in these logs as well. It can be a good idea to go back and scan the logs to see if there were any exceptions thrown that might indicate a larger problem.

Checking the Webapp

First, browse to the root of the webapp:

http://localhost:8080/     (for Eclipse, use http://localhost:8080/onebusaway-webapp/)

You should see either the welcome page from the default wiki if you are running the quick-start or perhaps other content if you configured a different content source.

Next, check for agencies:

http://localhost:8080/where/standard/agencies.action       (for Eclipse, use http://localhost:8080/onebusaway-webapp/where/standard/agencies.action)

Assuming that you have properly built and loaded your transit data bundle, you should see your transit agencies listed there, along with the general service area highlighted on the map. If you dont see any agencies listed, then it indicates a problem with your transit data bundle (improperly built? improper path specified to the webapp?).

Next, check the main map view:

http://localhost:8080/where/standard/              (for Eclipse, use http://localhost:8080/onebusaway-webapp/where/standard/)

This should bring up a browseable map. If you get an error about your Google Maps API key, you need to specify your own API key. Check out the defaultWebappConfigurationSource.googleMapsApiKey parameter in the parameter config documentation.

You should be able to search for routes by name and zoom in to find transit stops on the map. If you cant find any routes or stops, you might jump ahead to the API section, which will let you interrogate all data loaded in your bundle.

If you know the specific stop id of a stop, you can see arrivals and departures for that stop by going to the following url:

http://localhost:8080/where/standard/stop.action?id=AGENCYID_STOPID         (for Eclipse, use http://localhost:8080/onebusaway-webapp/where/standard/stop.action?id=AGENCYID_STOPID)

where AGENCYID and STOPID are the agency id and stop id configured for the stop in your GTFS.

Checking the API Webapp

First, request the set of agencies currently loaded in your system (method documentation):

http://localhost:8080/api/where/agencies-with-coverage.xml?key=TEST         (for Eclipse, use http://localhost:8080/onebusaway-api-webapp/api/where/agencies-with-coverage.xml?key=TEST)

You should see a response like:

<response>
  <version>2</version>
  <code>200</code>
  <currentTime>1334485773643</currentTime>
  <text>OK</text>
  <data class="listWithReferences">
    <references>
      <agencies>
        <agency>
          <id>3</id>
          <name>Pierce Transit</name>
          ...
        </agency>
      </agencies>
    </references>
    <list>
      <agencyWithCoverage>
        <agencyId>3</agencyId>
        <lat>47.221315000000004</lat>
        <lon>-122.4051325</lon>
        ...
      </agencyWithCoverage>
    </list>
  </data>
</response>

If you dont see any entries, it probably means your transit data bundle wasnt configured properly.

Next, for a specific agency, you can query the set of routes and stops for the agency.

For routes, use the following URL, replace AGENCYID with the agency id from the element in the agencies response.

http://localhost:8080/api/where/routes-for-agency/AGENCYID.xml?key=TEST         (for Eclipse, use http://localhost:8080/onebusaway-api-webapp/api/where/routes-for-agency/AGENCYID.xml?key=TEST)

You should see a number of elements for the routes in your system.

<response>
  <version>2</version>
  <code>200</code>
  <currentTime>1334486660558</currentTime>
  <text>OK</text>
  <data class="listWithReferences">
    <references>
      <agencies>...</agencies>
    </references>
    <list>
      <route>
        <id>1_1</id>
        <shortName>1</shortName>
        ...
      </route>
      ...

If you dont see any elements, it could indicate a problem with your transit data bundle.

For stops, use the following URL, again updating AGENCYID appropriately:

http://localhost:8080/api/where/stop-ids-for-agency/AGENCYID.xml?key=TEST           (for Eclipse, use http://localhost:8080/onebusaway-api-webapp/api/where/stop-ids-for-agency/AGENCYID.xml?key=TEST)

Here, you should see a list of stop ids for all the stops in your system.

<response>
  <version>2</version>
  <code>200</code>
  <currentTime>1334486349294</currentTime>
  <text>OK</text>
  <data class="listWithReferences">
    <references/>
    <list>
      <string>1_46660</string>
      <string>1_69670</string>
      <string>1_75580</string>
      ...

If you dont see any elements, it could indicate a problem with your transit data bundle.

Known Issues

Below are known issues people have had setting up OneBusAway and solutions that have worked.

InvalidStopToShapeMappingException when trying to build transit bundle

Symptom - You get a "java.lang.IllegalStateException: Multiple instances of InvalidStopToShapeMappingException thrown: count=7." error when trying to build a transit bundle.

Solution - This is likely due to problems with your GTFS data. Check out the "Stop to Shape Matching" page for more info on how to resolve this issue.

Web app projects not deployable to Tomcat in Eclipse

Symptom - The onebusaway-api-webapp, onebusaway-transit-data-federation-webapp, and onebusaway-webapp aren't recognized by Eclipse as web projects, and can't be deployed to Tomcat or other servers from within Eclipse. When clicking "Run As -> Run on Server" you might get the error "The selection did not contain any resources that can run on a server".

Solution - Install the Eclipse "Maven Integration for WTP" plugin using "Help->Install New Software" from this download site. If you're not using Eclipse for Java EE, try it, since the Java EE version has many web application plugins pre-installed.

Webapps fail to load on Jetty or WebSphere using Java 7.

Symptom - Errors are thrown when starting up the webapps on Jetty or WebSphere when using Java 7.

Solution - Use Tomcat with Java 7, or GlassFish 4 (with implicit CDI disabled). You can use Jetty or WebSphere with Java 6, but Java 6 is unsupported and therefore this solution isn't recommended.

Blank page (/onebusaway-webapp/where/standard/) on the main OneBusAway map interface

Symptom - When browsing to the main OBA map page (http://localhost:8080/onebusaway-webapp/where/standard/), you see a blank page with only the header. When viewing the output of Firebug in Firefox, it may have an error like "NetworkError: 404 Not Found - http://localhost:8080/onebusaway-webapp/where/standard/05786C619C689A398ECA56880E40A801.cache.html".

Solution - You need to generate the JavaScript for the main onebusaway-webapp using the Google Web Toolkit (GWT). If you're setting up a first OBA instance in Eclipse, make sure you've followed the steps in the OBA Developer Installation Guide to generate the GWT output. If you're editing the GWT content in OBA, check out the OBA Webapp Configuration Guide to see additional pointers when using GWT with OBA.

Tomcat times out when starting up with the OneBusAway webapps

Symptom - When starting Tomcat in Eclipse with the OBA webapps, you get an error "Server Tomcat v5.5 Server at localhost was unable to start within X seconds. If the server requires more time, try increasing the timeout in the server editor."

Solution - Increase the startup timeout for Tomcat by double-clicking the Tomcat entry in the Server window, which should open up the Tomcat config window. Expand the "Timeouts" section, and change the start value to a larger number (e.g., 300 seconds).

Database lock not released when running OneBusAway webapps or bundle builder

Symptom - When starting Tomcat in Eclipse with the OBA webapps, you get an error like "java.sql.SQLException: Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@dfeedeec[file =C:\OneBusAway\bundle\org_onebusaway_users.lck, exists=true, locked=false, valid=false, ] method: checkHeartbeat read: 2012-07-25 15:37:58 heartbeat - read: -1031 ms."

Solution - This happens when OBA webapps or bundle builder don't cleanly shut down (e.g., if Tomcat times out on startup with a large bundle) and it leaves the database locked. Stop the Tomcat server and/or the bundle builder, and wait for it to shut down. Then, go to your bundle directory and delete the *.lck files (org_onebusaway_database.lck, org_onebusaway_users.lck). Tomcat should start cleanly now. If not, it may be due to you have multiple web apps pointing to the same bundle directory when using the HSQLDB embedded database. When using HSQLDB, each webapp should have a unique bundle directory. Please see the "Choosing a Database" section on the Developer Installation Guide for more details.

Exceptions related to memory when trying to start up Tomcat with the webapps

Symptom - When starting up Tomcat, you get memory-related exceptions such as "Caused by: java.lang.OutOfMemoryError: PermGen space".

Solution - Tweak your Tomcat VM arguments:

  • In Eclipse, do this by double-clicking on the Tomcat entry in the Servers window (Window → Show View → Servers). This should open up the Tomcat config window. Click on the "Open launch configuration" link under the General Info tab, then click on the "Arguments" tab. In the VM arguments window, add the following after the existing arguments: -Xmx2000m -XX:MaxPermSize=128m. You can up the memory sizes if you get the same error again.
  • In a stand-alone Tomcat instance, do this by changing the CATALINA_OPTS environment variable in bin/setenv.sh or bin\setenv.bat (if neither of the files exists, simply create the file under bin and populate with the OS-appropriate command to set the environment variable). Refer to the chapter 'Configuring Tomcat' for your version of Tomcat for more information.

Problem starting the QuickStart bundle with "annotations only supported at Java 5 compliance level or above"

Symptom - When starting up OBA QuickStart, you get an exception "Initialization of bean failed; nested exception is java.lang.IllegalArgumentException: error the @annotation pointcut expression is only supported at Java 5 compliance level or above at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:452)"

Solution - This can be caused by running OBA with a version of Java under Java 5 (i.e., 1.5) OR by running OBA with Java 7 (i.e., 1.7)! As of mid-2012, it is recommended that you run OBA in Java JDK 6 (i.e., 1.6), since Java 7 seems to have problems with the annotations used in OBA classes. Check what version of Java you're running at the command line with java -version.

  • In Windows, you can change what version of Java you're using by default by setting your JAVA_HOME environment variable. If you're having problems with Java still using a different version by default after changing your JAVA_HOME variable, make sure your JAVA_HOME variable is set with a path without spaces, or use the abbreviation of the path (e.g., C:\Progra~1\Java\jdk1.6.0_27). Also make sure you're pointing to the root Java directory, not the "bin" directory. If you're still having problems, make sure that %JAVA_HOME%\bin is the FIRST entry in your "Path" environmental variable.

On webapp startup, errors with 'ehCacheMBeanRegistration' or CacheManager

Symptom - When starting up the web apps, you get the errors:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehCacheMBeanRegistration' defined in class path resource [org/onebusaway/container/application-context.xml]: Invocation of init method failed; nested exception is net.sf.ehcache.CacheException: javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=org.onebusaway.cacheManager
...
Caused by: javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=org.onebusaway.cacheManager

Solution - This issue can be caused by trying to load multiple bundles in the same JVM without the proper configuration in the data-sources.xml file, or a corrupted cache. It is suggested that you:

  • Make sure you have the cacheManager.cacheManagerName bean in your data-sources.xml file for all the webapps with a unique key value for each webapp. See the Developer Installation Guide page for example data-sources.xml files that contain this bean.
  • Delete the current contents of your bundle directory, and start with a freshly built bundle

In OBA main website webapp, error that Google has disabled your Maps API key for your application

Symptom - When browsing to the map pages in the onebusaway-webapp, the map tiles disappear and you get the popup "Error: Google has disabled use of the Maps API for this application. The provided key is not a valid Google API Key, or it is not authorized for the Google Maps Javascript API v2 on this site. If you are the owner of this application, you can learn about obtaining a valid key here: http://code.google.com/apis/maps/documentation/javascript/v2/introduction.html#Obtaining_key"

Solution - Make sure your Maps key is active for v2 of the Maps API, not v3. Google is now managing the creation and maintenance of all Google API keys using the centralized Google APIs Console. Go to your Google API console, and click on the "Services" link in the left hand column to display all the Google services. Make sure that the "Google Maps API v2" is turned ON, and the "Google Maps API v3" is turned OFF. You'll get the above error if the v3 API is turned on and the v2 API is not.

On webapp startup, errors with 'ehCacheMBeanRegistration' or CacheManager

Symptom - When starting up the web apps, you get the errors:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehCacheMBeanRegistration' defined in class path resource [org/onebusaway/container/application-context.xml]: Invocation of init method failed; nested exception is net.sf.ehcache.CacheException: javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=org.onebusaway.cacheManager
...
Caused by: javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=org.onebusaway.cacheManager

Solution - This issue can be caused by trying to load multiple bundles in the same JVM without the proper configuration in the data-sources.xml file, or a corrupted cache. It is suggested that you:

  • Make sure you have the cacheManager.cacheManagerName bean in your data-sources.xml file for all the webapps with a unique key value for each webapp. See the Developer Installation Guide page for example data-sources.xml files that contain this bean.
  • Delete the current contents of your bundle directory, and start with a freshly built bundle

When trying to connect to Tomcat instance deployed from Eclipse using web browser, you get an HTTP 404 error and Tomcat Server Config server locations is grayed out

Symptom - You start up the Tomcat server with the onebusaway-transit-data-federation-webapp and onebusaway-webapp in Eclipse, everything looks fine from the Console log but when you browse to http://localhost:8080/onebusaway-webapp/ you get an HTTP 404 Not found error. When trying to change the Tomcat Server Config page "Server Locations" section, its grayed out.

Solution - Eclipse occasionally loses touch with the Tomcat instance, especially if you try to rebuild the app while Tomcat is still started. First, you need to clear the current Tomcat instance. Do this by stopping the server, removing all webapps, and then right-click on Server node and do "Publish" without any webapps installed. This will clean Tomcat (more here on this). Then, in the Tomcat Server Config file change the Server Location to "Use Tomcat installation", instead of "Use workspace metadata" (more here).

Can't add new REST API keys using the admin console in the main OBA webapp

Symptom - You add a new API key via the admin console in the webapp and test it with the api-webapp REST API, but you get a "permission denied" response from the API.

Solution - This can be a symptom resulting from having multiple HSQLDB databases, one for onebusaway-transit-data-federation-webapp, onebusaway-api-webapp, and onebusaway-webapp, which is required for HSQLDB databases according to limitations discussed here. The onebusaway-webappis inserting the keys into its HSQLDB instance, which is separate from the HSQLDB instance that onebusaway-api-webapp sees. To support adding REST API keys via the webapp admin console, you'll need to use a database provider that supports simultaneous access by multiple webapps (i.e., onebusaway-webapp and onebusaway-api-webapp). See the Database Configuration Guide page for more information on using a different database provider such as MySQL with OneBusAway.

On bundle/GTFS update for a OBA server, "Complete timetable" calendar in onebusaway-webapp is blank or has old dates

Symptom - After updating OBA with new bundle info from new GTFS data, the "Complete timetable" calendar doesn't show any information on scheduled stops, or shows old information from the previous GTFS dataset.

Solution - This can happen if you copy new bundle information into the server directory while the Tomcat instance for OBA is still running. Tomcat retains file locks on some data that needs to be updated with a new bundle, and these files aren't updated if you copy in the new bundle info while Tomcat is running. If this happens, wipe the directory you’re using to build the transit bundle in, re-build the bundle, shut down Tomcat, and copy and overwrite the old server bundle folder with the new bundle info, and then startup Tomcat again.

Hibernate complains that a duplicate key value violates unique constraints

Symptom - After upgrading from a previous version, hibernate didnt update my DB schema, and now its complaining that duplicate key value violates unique constraint oba_nyc_raw_location_pkey.

Solution - The best way to fix this is to either add the changed columns yourself (not recommended) or, your better bet, drop the database and let hibernate regenerate it all over again. Wed had bad luck dropping individual tables and waiting for those to be rebuilt. If you need to migrate data from a previous schema version, we found this little script handy:

-- create function to prevent optimizer from removing the select nextval
CREATE FUNCTION volatile_nextval(text) RETURNS bigint VOLATILE AS 'BEGIN RETURN nextval($1); END;' LANGUAGE 'plpgsql';

-- move old data into temp table
select bearing, destinationsigncode, deviceid, gga, latitude, longitude, rmc, time, -1 AS timereceived, vehicle_agencyid, vehicle_id, (select volatile_nextval('hibernate_sequence') AS id) AS id into temporary table test from oba_nyc_raw_location_old;

-- update id value to sequence value (can't do this in a select...)
update test set id=volatile_nextval('hibernate_sequence');

-- move back to real table
insert into oba_nyc_raw_location(bearing, destinationsigncode, deviceid, gga, latitude, longitude, rmc, time, timereceived, vehicle_agencyid, vehicle_id, id) select bearing, destinationsigncode, deviceid, gga, latitude, longitude, rmc, time, -1, vehicle_agencyid, vehicle_id, id from test;

Im running OBA in a virtualized environment, and seeing OBAs memory consumption keep going up. What can I do?

On certain virtualized environments, the JDK can have trouble detecting how much memory to allocate to each webapp. The solution is to set a maximum memory limit by setting the JAVA_OPTS environment variable to: -Xmx256M -server. See this page for more information on the specific interaction with OpenVZ and the JDK.