Skip to content
Permalink
Browse files

Fix #294 - Move validation rule implementation into library module (#295

)

Project is now broken up into two submodules:
* gtfs-realtime-validator-lib - All core validation classes and the batch processor
* gtfs-realtime-validator-webapp - Web server and web client.  Uses the gtfs-realtime-validator-lib as a dependency.
  • Loading branch information...
barbeau committed Nov 1, 2017
1 parent 05c0f7c commit ee8fbc79e1cff6d748a098eb04792f0aeb223b8f
Showing with 1,545 additions and 1,098 deletions.
  1. +36 −0 CONFIG.md
  2. +19 −107 README.md
  3. +18 −0 gtfs-realtime-validator-lib/.gitignore
  4. 0 { → gtfs-realtime-validator-lib}/.mvn/jvm.config
  5. +15 −0 gtfs-realtime-validator-lib/LICENSE
  6. +125 −0 gtfs-realtime-validator-lib/README.md
  7. +197 −0 gtfs-realtime-validator-lib/pom.xml
  8. +30 −138 ...validator → gtfs-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/Main.java
  9. +12 −12 ...-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/batch/BatchProcessor.java
  10. +1 −1 ...altime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ErrorMessageModel.java
  11. +1 −1 ...e-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/GtfsFeedIterationModel.java
  12. +1 −1 ...s-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/GtfsFeedModel.java
  13. +1 −1 ...validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/GtfsRtFeedIterationModel.java
  14. +1 −1 ...alidator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/GtfsRtFeedIterationString.java
  15. +1 −1 ...realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/GtfsRtFeedModel.java
  16. +1 −1 ...realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/MessageLogModel.java
  17. +1 −1 ...realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/OccurrenceModel.java
  18. +1 −1 ...fs-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/SessionModel.java
  19. +1 −1 ...-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ValidationRule.java
  20. +1 −1 ...altime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewErrorLogModel.java
  21. +1 −1 ...me-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewErrorSummaryModel.java
  22. +1 −1 ...-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewFeedIterationsCount.java
  23. +1 −1 ...ime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewFeedMessageModel.java
  24. +1 −1 ...idator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewFeedUniqueResponseCount.java
  25. +2 −2 ...-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewGtfsErrorCountModel.java
  26. +1 −1 ...ator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewGtfsRtFeedErrorCountModel.java
  27. +1 −1 ...validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewIterationErrorsModel.java
  28. +4 −4 ...-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/ViewMessageDetailsModel.java
  29. +2 −2 ...src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/combined/CombinedIterationMessageModel.java
  30. +3 −3 ...rc/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/model/combined/CombinedMessageOccurrenceModel.java
  31. +3 −3 ...idator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib/model}/helper/ErrorListHelperModel.java
  32. +2 −2 ...b/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib/model}/helper/IterationErrorListHelperModel.java
  33. +4 −4 ...-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib/model}/helper/MergeMonitorData.java
  34. +1 −1 ...→ gtfs-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/util/GtfsUtils.java
  35. +3 −3 ...→ gtfs-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/util/RuleUtils.java
  36. +1 −1 ...→ gtfs-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/util/SortUtils.java
  37. +1 −1 ...s-realtime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/util/TimestampUtils.java
  38. +5 −5 ...altime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib/validation}/GtfsMetadata.java
  39. +1 −1 ...validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/IterationStatistics.java
  40. +1 −1 ...time-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/RuleStatistics.java
  41. +3 −2 ...ime-validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/ValidationRules.java
  42. +7 −7 ...ib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/gtfs/StopLocationTypeValidator.java
  43. +3 −3 ...ib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/interfaces/FeedEntityValidator.java
  44. +2 −2 ...-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/interfaces/GtfsFeedValidator.java
  45. +14 −14 ...rc/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/CrossFeedDescriptorValidator.java
  46. +9 −9 ...b/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/FrequencyTypeOneValidator.java
  47. +8 −8 .../src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/FrequencyTypeZeroValidator.java
  48. +9 −9 ...lidator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/HeaderValidator.java
  49. +44 −45 ...lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/StopTimeUpdateValidator.java
  50. +9 −9 ...validator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/StopValidator.java
  51. +13 −14 ...ator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/TimestampValidator.java
  52. +26 −26 ...lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/TripDescriptorValidator.java
  53. +15 −15 ...idator-lib/src/main/java/edu/usf/cutr/gtfsrtvalidator/lib}/validation/rules/VehicleValidator.java
  54. 0 { → gtfs-realtime-validator-lib}/src/main/resources/simplelogger.properties
  55. +3 −3 ...→ gtfs-realtime-validator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/BatchTest.java
  56. +3 −3 ...-realtime-validator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/FeedMessageTest.java
  57. +15 −14 ... → gtfs-realtime-validator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/UtilTest.java
  58. +33 −35 .../src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/CrossFeedDescriptorValidatorTest.java
  59. +7 −7 ...lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/FrequencyTypeOneValidatorTest.java
  60. +33 −34 ...ib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/FrequencyTypeZeroValidatorTest.java
  61. +26 −27 ...validator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/HeaderValidatorTest.java
  62. +7 −7 ...lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/StopLocationTypeValidatorTest.java
  63. +105 −105 ...r-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/StopTimeUpdateValidatorTest.java
  64. +16 −16 ...e-validator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/StopValidatorTest.java
  65. +8 −8 ...idator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/TimestampValidatorTest.java
  66. +7 −7 ...r-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/TripDescriptorValidatorTest.java
  67. +8 −8 ...alidator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/rules/VehicleValidatorTest.java
  68. +3 −3 ...s-realtime-validator-lib/src/test/java/edu/usf/cutr/gtfsrtvalidator/lib}/test/util/TestUtils.java
  69. +1 −0 gtfs-realtime-validator-lib/src/test/resources/.gitignore
  70. 0 { → gtfs-realtime-validator-lib}/src/test/resources/TripUpdates-2017-02-18T20-00-08Z.txt
  71. 0 { → gtfs-realtime-validator-lib}/src/test/resources/TripUpdates-2017-02-18T20-00-23Z.txt
  72. 0 { → gtfs-realtime-validator-lib}/src/test/resources/TripUpdates-2017-02-18T20-01-08Z.txt
  73. 0 { → gtfs-realtime-validator-lib}/src/test/resources/VehiclePositions-2017-02-18T20-01-08Z.txt
  74. BIN { → gtfs-realtime-validator-lib}/src/test/resources/badgtfs.zip
  75. BIN { → gtfs-realtime-validator-lib}/src/test/resources/bullrunner-gtfs-no-shapes.zip
  76. BIN ...altime-validator-lib}/src/test/resources/bullrunner-gtfs-timepoints-only-legacy-exact-times-1.zip
  77. BIN { → gtfs-realtime-validator-lib}/src/test/resources/bullrunner-gtfs.zip
  78. BIN { → gtfs-realtime-validator-lib}/src/test/resources/bullrunner-vehicle-positions
  79. 0 { → gtfs-realtime-validator-lib}/src/test/resources/testSQLScript.sql
  80. BIN { → gtfs-realtime-validator-lib}/src/test/resources/testagency.zip
  81. BIN { → gtfs-realtime-validator-lib}/src/test/resources/testagency2.zip
  82. +18 −0 gtfs-realtime-validator-webapp/.gitignore
  83. +1 −0 gtfs-realtime-validator-webapp/.mvn/jvm.config
  84. +15 −0 gtfs-realtime-validator-webapp/LICENSE
  85. +9 −0 gtfs-realtime-validator-webapp/README.md
  86. +327 −0 gtfs-realtime-validator-webapp/pom.xml
  87. +124 −0 gtfs-realtime-validator-webapp/src/main/java/edu/usf/cutr/gtfsrtvalidator/Main.java
  88. +1 −1 ...-realtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/api/resource/GtfsFeed.java
  89. +5 −5 ...ealtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/api/resource/GtfsRtFeed.java
  90. +10 −9 ...ltime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/background/BackgroundTask.java
  91. +2 −2 { → gtfs-realtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/db/GTFSDB.java
  92. +2 −1 { → gtfs-realtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/helper/DBHelper.java
  93. 0 { → gtfs-realtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/helper/GetFile.java
  94. +1 −1 ...altime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/helper/HttpMessageHelper.java
  95. 0 ...tfs-realtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/helper/QueryHelper.java
  96. 0 ...ealtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/helper/ServiceScheduler.java
  97. 0 ...ealtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/hibernate/HibernateUtil.java
  98. +2 −4 ...s-realtime-validator-webapp}/src/main/java/edu/usf/cutr/gtfsrtvalidator/servlets/GetFeedJSON.java
  99. +8 −10 ...gtfs-realtime-validator-webapp/src/main/java/edu/usf/cutr/gtfsrtvalidator/util/ProtoBufUtils.java
  100. +17 −17 { → gtfs-realtime-validator-webapp}/src/main/resources/hibernate.cfg.xml
  101. +8 −0 gtfs-realtime-validator-webapp/src/main/resources/simplelogger.properties
  102. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap-table.css
  103. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap-theme.css
  104. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap-theme.css.map
  105. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap-theme.min.css
  106. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap-toggle.min.css
  107. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap.css
  108. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap.css.map
  109. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/bootstrap.min.css
  110. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/jquery.bs_pagination.min.css
  111. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/css/style.css
  112. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/custom-js/error.js
  113. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/custom-js/index.js
  114. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/custom-js/iteration.js
  115. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/custom-js/loading.js
  116. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/custom-js/monitoring.js
  117. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/custom-js/session-monitor.js
  118. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/detailed.html
  119. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/error.html
  120. BIN ...gtfs-realtime-validator-webapp}/src/main/resources/webroot/fonts/glyphicons-halflings-regular.eot
  121. 0 ...gtfs-realtime-validator-webapp}/src/main/resources/webroot/fonts/glyphicons-halflings-regular.svg
  122. BIN ...gtfs-realtime-validator-webapp}/src/main/resources/webroot/fonts/glyphicons-halflings-regular.ttf
  123. BIN ...tfs-realtime-validator-webapp}/src/main/resources/webroot/fonts/glyphicons-halflings-regular.woff
  124. BIN ...fs-realtime-validator-webapp}/src/main/resources/webroot/fonts/glyphicons-halflings-regular.woff2
  125. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/index.html
  126. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/iteration.html
  127. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/bootstrap-table.js
  128. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/bootstrap-toggle.min.js
  129. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/bootstrap.min.js
  130. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/clipboard.min.js
  131. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/detail_table.js
  132. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/detail_table_sample_data.js
  133. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/en.min.js
  134. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/filesaver.min.js
  135. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/handlebars.js
  136. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/jquery-2.1.4.min.js
  137. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/jquery-highlight1.js
  138. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/jquery.bs_pagination.min.js
  139. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/jquery.flot.js
  140. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/jquery.highlight.js
  141. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/jsonpath-0.8.0.js
  142. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/js/tableexport.js
  143. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/loading.html
  144. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/monitoring.html
  145. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/session-monitor.html
  146. 0 { → gtfs-realtime-validator-webapp}/src/main/resources/webroot/swagger.yaml
  147. 0 ...ltime-validator-webapp}/src/test/java/edu/usf/cutr/gtfsrtvalidator/api/resource/GtfsFeedTest.java
  148. +3 −3 ...realtime-validator-webapp}/src/test/java/edu/usf/cutr/gtfsrtvalidator/test/queries/QueryTest.java
  149. 0 gtfs-realtime-validator-webapp/src/test/resources/TripUpdates-2017-02-18T20-00-08Z.txt
  150. 0 gtfs-realtime-validator-webapp/src/test/resources/TripUpdates-2017-02-18T20-00-23Z.txt
  151. 0 gtfs-realtime-validator-webapp/src/test/resources/TripUpdates-2017-02-18T20-01-08Z.txt
  152. 0 gtfs-realtime-validator-webapp/src/test/resources/VehiclePositions-2017-02-18T20-01-08Z.txt
  153. BIN gtfs-realtime-validator-webapp/src/test/resources/badgtfs.zip
  154. BIN gtfs-realtime-validator-webapp/src/test/resources/bullrunner-gtfs-no-shapes.zip
  155. BIN ...time-validator-webapp/src/test/resources/bullrunner-gtfs-timepoints-only-legacy-exact-times-1.zip
  156. BIN gtfs-realtime-validator-webapp/src/test/resources/bullrunner-gtfs.zip
  157. BIN gtfs-realtime-validator-webapp/src/test/resources/bullrunner-vehicle-positions
  158. +48 −0 gtfs-realtime-validator-webapp/src/test/resources/testSQLScript.sql
  159. BIN gtfs-realtime-validator-webapp/src/test/resources/testagency.zip
  160. BIN gtfs-realtime-validator-webapp/src/test/resources/testagency2.zip
  161. +5 −298 pom.xml
@@ -0,0 +1,36 @@
## Configuration options

#### Port number

In server mode, port `8080` is used by default. If you'd like to change the port number (e.g., port `80`), you can use the command line parameter `-port 80`:

`java -jar target/gtfs-realtime-validator-webapp-1.0.0-SNAPSHOT.jar -port 80`

#### Database

We use [Hibernate](http://hibernate.org/) to manage data persistence to a database. To allow you to get the tool up and running quickly, we use the embedded [HSQLDB](http://hsqldb.org/) by default. This is not recommended for a production deployment.

Hibernate configuration can be changed in [`src/main/resources/hibernate.cfg.xml`](https://github.com/CUTR-at-USF/gtfs-realtime-validator/blob/master/src/main/resources/hibernate.cfg.xml) to store data in any relational database. You might want to check out the following resources for getting started:

* [MySQL](https://docs.jboss.org/hibernate/orm/3.3/reference/en-US/html/session-configuration.html#configuration-xmlconfig)
* [PostgreSQL](http://stackoverflow.com/a/16572156/937715)
* [Microsoft SQL Server](http://stackoverflow.com/a/3588652/937715)
* [Oracle](https://docs.oracle.com/cd/E11035_01/workshop102/ormworkbench/hibernate-tutorial/tutHibernate9.html)

A list of all the dialect properties for specific database versions is shown [here](http://www.tutorialspoint.com/hibernate/hibernate_configuration.htm).

#### Logging

If you'd like to change the logging level, for example to see all debug statements, in `src/main/resources/simplelogger.properties` change the following line to say `DEBUG`:

~~~
org.slf4j.simpleLogger.defaultLogLevel=DEBUG
~~~

`DEBUG` level will show the output for all rule validation in the log.

`WARN` will show a smaller number of informational messages.

#### Batch processing

We support a command-line batch processing mode for archived GTFS-realtime files. See the [**gtfs-realtime-validator-lib** README](gtfs-realtime-validator-lib/README.md) page for details, including command-line configuration options for the batch processing mode.
126 README.md
@@ -17,15 +17,21 @@ We have a bleeding-edge alpha version running here as a demo:

*Please note that this project is still under active development and is in an early alpha state.*

## Prerequisites

1. Install [Java Development Kit (JDK) 1.8 or higher](http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html)
2. Download the latest alpha build:
* [gtfs-realtime-validator-1.0.0-SNAPSHOT.jar](https://s3.amazonaws.com/gtfs-rt-validator/travis_builds/gtfs-realtime-validator-1.0.0-SNAPSHOT.jar)

**To run the validator in default server mode, which provides a web user interface:**
There are two components to this project:
* **gtfs-realtime-validator-lib** - The core library that implements GTFS-realtime [validation rules](RULES.md) as well as [batch processing mode](BATCH.md).
* **gtfs-realtime-validator-webapp** - A server and website that allows multiple users to validate GTFS-relatime feeds by simply entering URLs into the website.

## Run the webapp

3. From the command line run `java -Djsse.enableSNIExtension=false -jar gtfs-realtime-validator-1.0.0-SNAPSHOT.jar`
4. When prompted, in your browser go to `http://localhost:8080`
5. Enter your [General Transit Feed Specification (GTFS)-realtime](https://developers.google.com/transit/gtfs-realtime/) and [GTFS](https://developers.google.com/transit/gtfs/) feed URLs and click "Start". Example feeds:
1. Download the latest webapp alpha build:
* [gtfs-realtime-validator-webapp-1.0.0-SNAPSHOT.jar](https://s3.amazonaws.com/gtfs-rt-validator/travis_builds/gtfs-realtime-validator-webapp-1.0.0-SNAPSHOT.jar)
1. From the command line run `java -Djsse.enableSNIExtension=false -jar gtfs-realtime-validator-webapp-1.0.0-SNAPSHOT.jar`
1. When prompted, in your browser go to `http://localhost:8080`
1. Enter your [General Transit Feed Specification (GTFS)-realtime](https://developers.google.com/transit/gtfs-realtime/) and [GTFS](https://developers.google.com/transit/gtfs/) feed URLs and click "Start". Example feeds:
* HART (Tampa, FL)
* GTFS-realtime - http://api.tampa.onebusaway.org:8088/trip-updates
* GTFS - http://gohart.org/google/google_transit.zip
@@ -36,13 +42,9 @@ We have a bleeding-edge alpha version running here as a demo:

Please note that if you're using `https` URLS, you'll need to use the `-Djsse.enableSNIExtension=false` command-line parameter or install the [Java Cryptography Extension (JCE)](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html) - see the [Prerequisites](https://github.com/CUTR-at-USF/gtfs-realtime-validator#prerequisites) section for details.

**To run the validator in batch processing mode, to validate a large number of archived feed files:**
## Run batch validation

3. From the command line run `java -jar target/gtfs-realtime-validator-1.0.0-SNAPSHOT.jar -batch yes -gtfs "D:\HART\google_transit.zip" -gtfsrealtimepath "D:\HART\gtfs-rt"`
* `-gtfs` should point to the GTFS zip file
* `-gtfsrealtimepath` should point to the directory holding the GTFS-realtime files

See ["Configuration Options -> Batch Processing"](https://github.com/CUTR-at-USF/gtfs-realtime-validator#batch-processing) for more documentation.
See the [batch processing](gtfs-realtime-validator-lib/README.md#batch-processing) section of the [**gtfs-realtime-validator-lib** README](gtfs-realtime-validator-lib/README.md).

## Rules

@@ -53,7 +55,7 @@ Have a suggestion for a new rule? Open an issue with the ["new rule" label](htt

## Building the project

The main GTFS-Realtime Validator user interface is implemented as a web application, with the backend code written in Java. An instance of the [Jetty embedded server](http://www.eclipse.org/jetty/) is used to run the application, with [Hibernate](http://hibernate.org/) used for data persistence.
The main **gtfs-realtime-validator-webapp** user interface is implemented as a web application, with the backend code written in Java. An instance of the [Jetty embedded server](http://www.eclipse.org/jetty/) is used to run the application, with [Hibernate](http://hibernate.org/) used for data persistence.

#### Prerequisites

@@ -72,7 +74,7 @@ From the command-line, run:

`mvn package`

This will generate an executable file in the `target/` directory with all the dependencies needed to run the application.
This will generate an executable file in the `gtfs-realtime-validator-webapp/target/` directory with all the dependencies needed to run the web application.

Note that this might take a while - this project also builds and packages the [gtfs-validator](https://github.com/conveyal/gtfs-validator) so a static GTFS validation report can be seen within the GTFS-rt validator tool.

@@ -82,111 +84,21 @@ If you're going to be rebuilding the project frequently (e.g., editing source co

To start up the server so you can view the web interface, from the command-line, run:

`java -Djsse.enableSNIExtension=false -jar target/gtfs-realtime-validator-1.0.0-SNAPSHOT.jar`
`java -Djsse.enableSNIExtension=false -jar gtfs-realtime-validator-webapp/target/gtfs-realtime-validator-webapp-1.0.0-SNAPSHOT.jar`

You should see some output, and a message saying `Go to http://localhost:8080 in your browser`.

Note that there is also an option to run the validator as a batch process on archived feeds - see ["Configuration Options -> Batch Processing"](https://github.com/CUTR-at-USF/gtfs-realtime-validator#batch-processing).

#### 3. View the application

Once the application has been started, you can enter URLs for the feeds you'd like to have validated at:

http://localhost:8080

Note that there is also an option for [Batch Processing](https://github.com/CUTR-at-USF/gtfs-realtime-validator#batch-processing) feeds from the command line.

## Configuration options

#### Logging

If you'd like to change the logging level, for example to see all debug statements, in `src/main/resources/simplelogger.properties` change the following line to say `DEBUG`:

~~~
org.slf4j.simpleLogger.defaultLogLevel=DEBUG
~~~

`DEBUG` level will show the output for all rule validation in the log.

`WARN` will show a smaller number of informational messages.

#### Port number

Port `8080` is used by default. If you'd like to change the port number (e.g., port `80`), you can use the command line parameter `-port 80`:

`java -jar target/gtfs-realtime-validator-1.0.0-SNAPSHOT.jar -port 80`

#### Database

We use [Hibernate](http://hibernate.org/) to manage data persistence to a database. To allow you to get the tool up and running quickly, we use the embedded [HSQLDB](http://hsqldb.org/) by default. This is not recommended for a production deployment.

Hibernate configuration can be changed in [`src/main/resources/hibernate.cfg.xml`](https://github.com/CUTR-at-USF/gtfs-realtime-validator/blob/master/src/main/resources/hibernate.cfg.xml) to store data in any relational database. You might want to check out the following resources for getting started:

* [MySQL](https://docs.jboss.org/hibernate/orm/3.3/reference/en-US/html/session-configuration.html#configuration-xmlconfig)
* [PostgreSQL](http://stackoverflow.com/a/16572156/937715)
* [Microsoft SQL Server](http://stackoverflow.com/a/3588652/937715)
* [Oracle](https://docs.oracle.com/cd/E11035_01/workshop102/ormworkbench/hibernate-tutorial/tutHibernate9.html)

A list of all the dialect properties for specific database versions is shown [here](http://www.tutorialspoint.com/hibernate/hibernate_configuration.htm).

#### Batch processing

We support a command-line batch processing mode for archived GTFS-realtime files.

Here's an example of a command to batch process a set of GTFS-realtime files:

`java -jar target/gtfs-realtime-validator-1.0.0-SNAPSHOT.jar -batch yes -gtfs "D:\HART\google_transit.zip" -gtfsrealtimepath "D:\HART\gtfs-rt" -sort date`

Parameters:

* `-batch` - Must be provided for the validator to start in batch processing mode. If this parameter isn't provided, the server will start in the normal mode.
* `-gtfs` - The path and file name of the GTFS zip file. GTFS zip file must cover the time period for the GTFS-rt archived files. You can combine GTFS zip files if needed using the [Google transitfeed tool's](https://github.com/google/transitfeed/wiki/Merge).
* `-gtfsrealtimepath` - The path to the folder that contains the individual GTFS-realtime protocol buffer files
* `-sort` *(Optional)* - `date` if the GTFS-realtime files should be processed chronologically by the "last modified" date of the file (default), or `name` if the files should be ordered by the name of the file. If you use the name of the file to order the files, then the validator will try to parse the date/time from each individual file name and use that date/time as the "current" time. Date/times in file names must be in the [ISO_DATE_TIME](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ISO_DATE_TIME) format and must be the last 20 characters prior to the file extension - for example, `TripUpdates-2017-02-18T20-00-08Z.pb`. If a date/time can't be parsed from the file name, then the last modified date is used as the "current" time. GTFS-realtime file order is important for rules such as E012, E018, and W007, which compare the previous feed iteration against the current one.
* `-plaintext` *(Optional)* - If this argument is supplied, the validator will output a plain text version of each of the protocol buffer files with the provided file extension. For example, if the protocol buffer file has the name `trip-update.pb`, and the text `-plaintext txt` is provided as the argument, then the plain text version of this file will be `trip-update.pb.txt`.
* `-stats` *(Optional)* - If this argument is supplied (e.g., `-stats yes`), the validator will save statistics to memory for each of the validation files that are processed, and will return a list of `ValidationStatistics` objects from `BatchProcessor.processFeeds()` that can be examined to see individual iteration and rule processing times (in decimal seconds).
* `-ignoreshapes` *(Optional)* - If this argument is supplied (e.g., `-ignoreshapes yes`), the validator will ignore the shapes.txt file for the GTFS feed. If you are getting OutOfMemoryErrors when processing very large feeds, you should try setting this to true. Note that setting this to true will prevent the validator from checking rules like E029 that require spatial data. See [this issue](https://github.com/CUTR-at-USF/gtfs-realtime-validator/issues/284) for details.

After execution finishes, the results for each GTFS-realtime protocol buffer file will be output in JSON format with the same file name, but with "results.json" appended to the end. For example, if one GTFS-realtime procotol buffer file name was `TripUpdates-2017-02-18T20-00-08Z.pb`, the validation results for that file name will be output as `TripUpdates-2017-02-18T20-00-08Z.pb.results.json`.

It will look something like:

~~~
[ {
"errorMessage" : {
"messageId" : 0,
"gtfsRtFeedIterationModel" : null,
"validationRule" : {
"errorId" : "W001",
"severity" : "WARNING",
"title" : "timestamp not populated",
"errorDescription" : "Timestamps should be populated for all elements",
"occurrenceSuffix" : "does not have a timestamp"
},
"errorDetails" : null
},
"occurrenceList" : [ {
"occurrenceId" : 0,
"messageLogModel" : null,
"prefix" : "trip_id 277716"
}, {
"occurrenceId" : 0,
"messageLogModel" : null,
"prefix" : "trip_id 277767"
}, {
"occurrenceId" : 0,
"messageLogModel" : null,
"prefix" : "trip_id 277768"
},
...
~~~

In the above example, three `trip_updates` have been validated, and each was missing a timestamp (warning `W001`). To put together the full message for each occurrence of the warning or error, you add the occurrence `prefix` to the validationRule `occurrenceSuffix`.
See our [Configuration Guide](CONFIG.md) for various configuration options, including changing the port number that the server runs on, what database it connects to, and more.

For example, in log format the above would look like:
* `trip_id 277716 does not have a timestamp`
* `trip_id 277767 does not have a timestamp`
* `trip_id 277768 does not have a timestamp`
Note that the validator also has a [batch processing mode](gtfs-realtime-validator-lib/README.md#batch-processing) - see the [**gtfs-realtime-validator-lib** README](gtfs-realtime-validator-lib/README.md).

#### Docker

@@ -0,0 +1,18 @@
*.iml
.idea/

target/
**/target/
pom.xml.tag
pom.xml.releaseBackup
pom.xml.versionsBackup
pom.xml.next
release.properties
dependency-reduced-pom.xml
buildNumber.properties
.mvn/timing.properties
*.swp
*.log
*~

*.db
File renamed without changes.
@@ -0,0 +1,15 @@
Copyright (C) 2015-2017 Nipuna Gunathilake, University of South Florida
All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

0 comments on commit ee8fbc7

Please sign in to comment.
You can’t perform that action at this time.