feat: Packaged Executable (#1125)

Introduces functionality to package the GTFS Validator as an installable application, including a bundled JRE, to make it easier for users to run the validator. As discussed in issue #1112, https://bit.ly/gtfs-validator-packaged-exe, and #1124.

This is the initial entry for a minimal packaged validator app, to be run as a native application
on Windows and Mac OS.  Right now, it's just a simple wrapper around the CLI app.  It includes the following changes:

* Switch to ClassGraph for package class reflection, moving away from Guava ClassPath.

Guava's ClassPath scanning has issues when run against Java Modules and its own source
code advises you to use ClassGraph instead:
https://github.com/google/guava/blob/master/guava/src/com/google/common/reflect/ClassPath.java

This change will better support running the validator as a Java Module in a packaged runtime.

	modified:   core/build.gradle
	modified:   core/src/main/java/org/mobilitydata/gtfsvalidator/notice/NoticeSchemaGenerator.java
	modified:   core/src/main/java/org/mobilitydata/gtfsvalidator/table/GtfsFeedLoader.java
	modified:   core/src/main/java/org/mobilitydata/gtfsvalidator/validator/ValidatorLoader.java


* Add some documentation comments to app.gui.Main

* Write flogger statments to ~/gtfs-validator.log in addition to the console, for easier
debugging when running the app in non command-line mode.

* Update copyright headers on new files.

* Fix issue with javadoc aggregation.

Specifically, switch the project to use the io.freefair.aggregate-javadoc-jar plugin and disable Javadoc generation for the :app:pkg sub-project.

Javadoc generation appears to get tripped up for the :app:pkg sub-project because of the previously discussed tricks I used to get Java Modules working (shadow jar wrapped as a single module). Specifically, Javadoc appears to be adding the the shadow jar its classpath along with all the other project dependencies, which is causing a bunch of duplicate class warnings.

There seem to be two options for fixing this:

1)   We try to fully modularize the entire project. I'm wary of going down this road for now.
2)   We try to exclude the :app:pkg project from Javadoc generation. It only has a single dummy class anyway, so we're not missing much there.

The trick with option # 2 is that we are using the nebula-aggregate-javadocs plugin from Netflix for project-wide javadoc aggregation. This plugin doesn't appear to have a way of excluding a project (I've looked at the source). This plugin also hasn't been updated in five years?

As an alternative, there are newer Gradle plugins that support javadoc aggregation. Specifically, I just tried the io.freefair.aggregate-javadoc-jar, which is currently being actively maintained. Per the source, it's possible to exclude particular projects with a "javadoc { enabled = false }" clause in a sub-project. I've verified this behavior on own project. It seems to generate aggregated javadoc in build/docs/javadoc/ like the old plugin, though the target name has changed:

aggregateJavadocs => aggregateJavadoc (the s is dropped)

* Add details about app:gui and app:pkg sub-projects to ARCHITECTURE.md.

I took the liberty of replacing the existing static architecture diagram with a Mermaid diagram that can be edited directly in Markdown.

* Add and update TODOs to reference github issues.

* Update app to use a default output directory of `~/GtfsValidator`.

* Add build instructions for the installable application in BUILD.md.

* Remove $ from command line in BUILD.md.

* Note requirement for Wix when building package on Windows.

* Add note about how app is under active development.

Co-authored-by: Brian Ferris <bdferris@google.com>

.github/workflows/test_pack_doc.yml

@@ -136,7 +136,7 @@ jobs:
         env:
           versionTag: ${{ needs.prepare-version-name.outputs.versionTag }}
         with:
-          arguments: aggregateJavadocs
+          arguments: aggregateJavadoc
       - name: Persist javadoc
         uses: actions/upload-artifact@v2
         with:

app/gui/build.gradle

@@ -0,0 +1,59 @@
+/*
+ * Copyright 2020-2022 Google LLC, MobilityData IO
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+    id 'application'
+    id 'java'
+    id 'com.github.johnrengelman.shadow' version '5.2.0'
+}
+
+sourceCompatibility = JavaVersion.VERSION_11
+mainClassName = 'org.mobilitydata.gtfsvalidator.app.gui.Main'
+compileJava.options.encoding = "UTF-8"
+
+repositories {
+    mavenCentral()
+}
+
+dependencies {
+    implementation project(':main')
+    implementation 'com.google.flogger:flogger:0.6'
+    implementation 'com.google.flogger:flogger-system-backend:0.6'
+}
+
+jar {
+    manifest {
+        attributes('Implementation-Title': rootProject.name,
+                'Implementation-Version': project.version,
+                'Main-Class': 'org.mobilitydata.gtfsvalidator.app.gui.Main')
+    }
+}
+
+shadowJar {
+    minimize {
+        // We don't want to minimize the :main project, as it will drop the validators
+        // loaded via reflection
+        exclude(project(':main'))
+
+        exclude(dependency('org.apache.httpcomponents:httpclient'))
+    }
+
+    // Some of our dependencies include their own module-info declarations.  We drop
+    // all of them, as we'll be wrapping the entire uber jar as a module in :app:pkg
+    // and we don't want these existing module-info declarations to conflict.
+    exclude 'module-info.class'
+    exclude 'META-INF/versions/*/module-info.class'
+}

app/gui/src/main/java/org/mobilitydata/gtfsvalidator/app/gui/Main.java

@@ -0,0 +1,128 @@
+/*
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.mobilitydata.gtfsvalidator.app.gui;
+
+import com.google.common.flogger.FluentLogger;
+import java.awt.Desktop;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.logging.LogManager;
+import java.util.logging.Logger;
+
+/**
+ * The main entry point for the GUI application.
+ *
+ * <p>Compared to the CLI jar, this entry point is designed to be packaged as a native application
+ * to be run directly by the user.
+ *
+ * <p>TODO(#1134): Follow up work will add a minimal UI for selecting the input GTFS and potentially
+ * the output directory.
+ */
+public class Main {
+  // We use `~/GtfsValidator` as the default output directory for reports and
+  // logs if the user has not specified an explicit directory.  If this name
+  // is updated, make sure to update `logging.properties` as well.
+  private static final String DEFAULT_OUTPUT_DIRECTORY_NAME = "GtfsValidator";
+
+  static {
+    // Attempt to create the default output directory, as we'll use it for
+    // the output directory of the file-based logger below.
+    File outputDirectory = getDefaultOutputDirectory().toFile();
+    if (!outputDirectory.exists()) {
+      outputDirectory.mkdir();
+    }
+
+    try (InputStream inputStream = Main.class.getResourceAsStream("/logging.properties")) {
+      LogManager.getLogManager().readConfiguration(inputStream);
+    } catch (IOException e) {
+      Logger.getAnonymousLogger().severe("Could not load default logging.properties file");
+      Logger.getAnonymousLogger().severe(e.getMessage());
+    }
+  }
+
+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
+  public static void main(String[] args) {
+    logger.atInfo().log("gtfs-validator: start");
+
+    // On Windows, if you drag a file onto the application shortcut, it will
+    // execute the app with the file as the first command-line argument.  This
+    // doesn't appear to work on Mac OS.
+    if (args.length != 1) {
+      logger.atSevere().log("No GTFS input specified - args=%d", args.length);
+      System.exit(-1);
+    } else {
+      run(args[0]);
+    }
+
+    logger.atInfo().log("gtfs-validator: exit");
+  }
+
+  private static void run(String path) {
+    // TODO(#1135): Refactor this code to call GTFS validation code directly
+    // instead of constructing artifical command-line args and calling cli.Main.
+    Path workingDirectory = getDefaultOutputDirectory();
+
+    List<String> cliArgs = new ArrayList<>();
+    cliArgs.add("-i");
+    cliArgs.add(path);
+    cliArgs.add("-o");
+    cliArgs.add(workingDirectory.toString());
+    cliArgs.add("--pretty");
+
+    org.mobilitydata.gtfsvalidator.cli.Main.main(cliArgs.toArray(new String[] {}));
+
+    Path reportPath = workingDirectory.resolve("report.json");
+
+    try {
+      Desktop.getDesktop().browse(reportPath.toUri());
+    } catch (IOException ex) {
+      logger.atSevere().withCause(ex).log("Error opening browser");
+      System.exit(-1);
+    }
+  }
+
+  /**
+   * Try to return `~/GtfsValidator` as the default output directory for reports and logs if we are
+   * able to resolve the user's home directory. We do not attempt to create this directory if it
+   * does not exist.
+   *
+   * <p>If we are not able to resolve the user's home directory, we fall back to a temporary
+   * directory.
+   */
+  private static Path getDefaultOutputDirectory() {
+    String path = System.getProperty("user.home");
+    if (path != null) {
+      return Path.of(path, DEFAULT_OUTPUT_DIRECTORY_NAME);
+    }
+
+    // If for some reason the user home directory cannot be resolved, we fall
+    // back to a temporary directory.
+    Path workingDirectory = null;
+    try {
+      workingDirectory = Files.createTempDirectory("GtfsValidator");
+    } catch (IOException ex) {
+      logger.atSevere().withCause(ex).log("Error creating working directory");
+      System.exit(-1);
+    }
+    return workingDirectory;
+  }
+}

app/gui/src/main/resources/logging.properties

@@ -0,0 +1,5 @@
+handlers=java.util.logging.FileHandler,java.util.logging.ConsoleHandler
+
+java.util.logging.FileHandler.pattern=%h/GtfsValidator/run.log
+java.util.logging.FileHandler.count=1
+java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

app/pkg/build.gradle

@@ -0,0 +1,110 @@
+/*
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+    id 'java'
+    id "de.jjohannes.extra-java-module-info" version "0.11"
+    id "org.beryx.jlink" version "2.24.1"
+}
+
+repositories {
+    mavenCentral()
+}
+
+dependencies {
+    // Note that we depend on the :app:gui shadow jar, which bundles the gui
+    // application and all its dependencies into a single uber jar.
+    implementation project(path: ':app:gui', configuration: 'shadow')
+}
+
+extraJavaModuleInfo {
+    // JPackage (below) requires that we package our application as a Java
+    // Module.  Instead of attempting to modularize the entire gtfs-validator
+    // project, we instead make this project a module and moduarlize our single
+    // :app:gui uber-jar dependency by injecting a module-info.class entry into
+    // the jar with its native dependencies.
+    //
+    // See additional discussion in https://bit.ly/gtfs-validator-packaged-exe and
+    // https://docs.gradle.org/current/samples/sample_java_modules_with_transform.html
+    module('gui-all.jar', 'org.mobilitydata.gtfsvalidator.app.gui', project.version) {
+        exports('org.mobilitydata.gtfsvalidator.app.gui')
+
+        // This is the set of core Java modules that our application depends on.
+        // This list was hand-curated by looking at the output of `jdeps -s` on
+        // the full application jar, which produces some false positives given
+        // that not all code-paths in our dependencies are actually used.  I
+        // have also run the app with no module dependencies to see what kind
+        // of exceptions we get.
+        requires('java.compiler')
+        requires('java.desktop')
+        requires('java.logging')
+        requires('java.naming')
+        requires('java.security.jgss')
+        requires('java.sql')
+  }
+}
+
+sourceCompatibility = JavaVersion.VERSION_11
+compileJava.options.encoding = "UTF-8"
+
+java {
+    modularity.inferModulePath = true
+}
+
+application {
+    mainClass = 'org.mobilitydata.gtfsvalidator.app.pkg.Main'
+    mainModule = 'org.mobilitydata.gtfsvalidator.app.pkg'
+}
+
+jar {
+    // Add the manifest within the JAR, using gtfs-validator as the title
+    manifest {
+        attributes('Implementation-Title': rootProject.name,
+                'Implementation-Version': project.version)
+    }
+}
+
+// Debugging tips:
+// If you ever get an error when running the `jpackage` task like:
+//   Error reading module: app/pkg/build/jlinkbase/jlinkjars/pkg.jar
+// (and not much else), rerun the Gradle task with --debug specified and look
+// for the full command-line executed for `jlink` or `jpackage`.  Copy the
+// command-line and run it directly with a `-J-Djlink.debug=true` flag added
+// and you will get more useful information about what went wrong.
+jlink {
+    moduleName = 'org.mobilitydata.gtfsvalidator.app.pkg'
+    launcher {
+        name = 'GTFS Validator'
+    }
+    // Passed to jlink to create an even smaller JRE.
+    options = ['--strip-debug', '--compress', '2', '--no-header-files', '--no-man-pages']
+    jpackage {
+        if (org.gradle.internal.os.OperatingSystem.current().windows) {
+            installerOptions += ['--win-per-user-install', '--win-dir-chooser', '--win-menu', '--win-shortcut']
+            imageOptions += ['--win-console']
+        }
+    }
+}
+
+javadoc {
+    // Our complex use of a shadow jar dependency bundled as a Java Module
+    // (see above) causes problems for the Javadoc compiler, especially when
+    // run in aggregate mode over the entire project.  As such, we disable
+    // Javadoc for this sub-project to avoid issues.  Since there isn't much
+    // real source-code in the :pkg project, it's no big loss.
+    enabled = false
+}
+

app/pkg/src/main/java/module-info.java

@@ -0,0 +1,3 @@
+module org.mobilitydata.gtfsvalidator.app.pkg {
+  requires org.mobilitydata.gtfsvalidator.app.gui;
+}

app/pkg/src/main/java/org/mobilitydata/gtfsvalidator/app/pkg/Main.java

@@ -0,0 +1,25 @@
+/*
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.mobilitydata.gtfsvalidator.app.pkg;
+
+/**
+ * This is just a simple stub around app.gui.Main to assist with Java modularization and packaging.
+ */
+public class Main {
+  public static void main(String[] args) {
+    org.mobilitydata.gtfsvalidator.app.gui.Main.main(args);
+  }
+}

build.gradle

@@ -14,27 +14,17 @@
  * limitations under the License.
  */
 
-// so we can aggregate all Javadocs in one
-// see: https://github.com/nebula-plugins/gradle-aggregate-javadocs-plugin
-buildscript {
-    repositories { mavenCentral() }
-
-    dependencies {
-        classpath 'com.netflix.nebula:gradle-aggregate-javadocs-plugin:3.0.1'
-    }
-}
-
 plugins {
     id 'java'
     id 'com.github.sherter.google-java-format' version '0.9'
+    id "io.freefair.aggregate-javadoc-jar" version "6.4.3"
 }
 
 repositories {
     mavenCentral()
 }
 
 // Setup and configure Javadoc plugin
-apply plugin: 'nebula-aggregate-javadocs'
 allprojects {
     tasks.withType(Javadoc) { options.encoding = 'UTF-8' }
 }

core/build.gradle

@@ -43,6 +43,7 @@ dependencies {
     implementation 'commons-validator:commons-validator:1.6'
     implementation 'com.googlecode.libphonenumber:libphonenumber:8.12.13'
     implementation 'com.google.flogger:flogger:0.6'
+    implementation 'io.github.classgraph:classgraph:4.8.146'
     testImplementation 'com.google.flogger:flogger-system-backend:0.6'
     testImplementation group: 'junit', name: 'junit', version: '4.13'
     testImplementation "com.google.truth:truth:1.0.1"