Skip to content

Commit

Permalink
Merge pull request #2859 from broadinstitute/ks_generate_rest_api_docs
Browse files Browse the repository at this point in the history
Added REST API docs generator.
  • Loading branch information
kshakir committed Nov 14, 2017
2 parents d17f08a + f6f5606 commit 78047f8
Show file tree
Hide file tree
Showing 7 changed files with 257 additions and 19 deletions.
2 changes: 1 addition & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ lazy val womtool = project
.dependsOn(wom % "test->test")

lazy val root = (project in file("."))
.withExecutableSettings("cromwell", rootDependencies)
.withExecutableSettings("cromwell", rootDependencies, rootSettings)
// Next level of projects to include in the fat jar (their dependsOn will be transitively included)
.dependsOn(engine)
.dependsOn(jesBackend)
Expand Down
32 changes: 19 additions & 13 deletions docs/api/RESTAPI.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
<!--
This file was generated by `sbt generateRestApiDocs` on Mon, 13 Nov 2017 18:39:55 -0500
!!! DO NOT CHANGE THIS FILE DIRECTLY !!!
If you wish to change something in this file, either change cromwell.yaml or GenerateRestApiDocs.scala then
regenerate.
-->
# Cromwell Server REST API


Expand All @@ -17,9 +25,13 @@ Describes the REST API provided by a Cromwell server


**Produces**

* `application/json`





<a name="api-workflows-version-post"></a>
## Submit a workflow for execution
```
Expand All @@ -38,11 +50,11 @@ Submits a workflow to Cromwell. Note that this endpoint can accept an unlimited
|**Path**|**version** <br>*required*|Cromwell API Version|string|`"v1"`|
|**FormData**|**customLabels** <br>*optional*|JSON file containing a set of collection of key/value pairs for labels to apply to this workflow.|file||
|**FormData**|**workflowDependencies** <br>*optional*|ZIP file containing workflow source files that are used to resolve local imports. This zip bundle will be unpacked in a sandbox accessible to this workflow.|file||
|**FormData**|**workflowInputs** <br>*optional*|JSON file containing the inputs as an object. For WDL workflows a skeleton file can be generated from wdltool using the "inputs" subcommand. When multiple files are specified, in case of key conflicts between multiple input JSON files, higher values of x in workflowInputs_x override lower values. For example, an input specified in workflowInputs_3 will override an input with the same name in workflowInputs or workflowInputs_2. Similarly, an input key specified in workflowInputs_5 will override an identical input key in any other input file.|file||
|**FormData**|**workflowInputs_2** <br>*optional*|A second JSON file containing inputs.|file||
|**FormData**|**workflowInputs_3** <br>*optional*|A third JSON file containing inputs.|file||
|**FormData**|**workflowInputs_4** <br>*optional*|A fourth JSON file containing inputs.|file||
|**FormData**|**workflowInputs_5** <br>*optional*|A fifth JSON file containing inputs.|file||
|**FormData**|**workflowInputs** <br>*optional*|JSON or YAML file containing the inputs as an object. For WDL workflows a skeleton file can be generated from wdltool using the "inputs" subcommand. When multiple files are specified, in case of key conflicts between multiple input JSON files, higher values of x in workflowInputs_x override lower values. For example, an input specified in workflowInputs_3 will override an input with the same name in workflowInputs or workflowInputs_2. Similarly, an input key specified in workflowInputs_5 will override an identical input key in any other input file.|file||
|**FormData**|**workflowInputs_2** <br>*optional*|A second JSON or YAML file containing inputs.|file||
|**FormData**|**workflowInputs_3** <br>*optional*|A third JSON or YAML file containing inputs.|file||
|**FormData**|**workflowInputs_4** <br>*optional*|A fourth JSON or YAML file containing inputs.|file||
|**FormData**|**workflowInputs_5** <br>*optional*|A fifth JSON or YAML file containing inputs.|file||
|**FormData**|**workflowOptions** <br>*optional*|JSON file containing configuration options for the execution of this workflow.|file||
|**FormData**|**workflowSource** <br>*required*|The workflow source file to submit for execution.|file||
|**FormData**|**workflowType** <br>*optional*|The workflow language for the file you submitted. Cromwell currently supports WDL.|enum (wdl, cwl)|`"wdl"`|
Expand All @@ -54,7 +66,7 @@ Submits a workflow to Cromwell. Note that this endpoint can accept an unlimited
|HTTP Code|Description|Schema|
|---|---|---|
|**201**|Successful Request|[WorkflowIdAndStatus](#workflowidandstatus)|
|**400**|Malformed Workflow ID|No Content|
|**400**|Invalid submission request|No Content|
|**500**|Internal Error|No Content|


Expand Down Expand Up @@ -634,7 +646,7 @@ GET /engine/{version}/status


<a name="engine-version-version-get"></a>
### Return the version of this Cromwell server
## Return the version of this Cromwell server
```
GET /engine/{version}/version
```
Expand All @@ -659,8 +671,6 @@ GET /engine/{version}/version
* Engine




<a name="definitions"></a>
## Definitions

Expand Down Expand Up @@ -827,8 +837,6 @@ Result for an individual workflow returned by a workflow query
|**status** <br>*required*|The status of the workflow <br>**Example** : `"Submitted"`|string|




<a name="securityscheme"></a>
## Security

Expand All @@ -843,5 +851,3 @@ Result for an individual workflow returned by a workflow query
|---|---|
|openid|open id authorization|



2 changes: 1 addition & 1 deletion engine/src/main/resources/swagger/cromwell.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
swagger: '2.0'
info:
title: Cromwell Server REST API
description: Describes the rest API provided by a Cromwell server
description: Describes the REST API provided by a Cromwell server
license:
name: BSD
url: 'https://github.com/broadinstitute/cromwell/blob/develop/LICENSE.txt'
Expand Down
53 changes: 49 additions & 4 deletions project/Dependencies.scala
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,37 @@ object Dependencies {
val akkaV = "2.5.4"
val ammoniteOpsV = "1.0.1"
val apacheHttpClientV = "4.5.3"
val apacheHttpCoreV = "4.4.6"
val betterFilesV = "2.17.1"
val catsEffectV = "0.4"
val catsV = "1.0.0-MF"
val circeV = "0.9.0-M1"
val circeYamlV = "0.7.0-M1"
val commonsCodecV = "1.10"
val commonsIoV = "2.5"
val commonsLang3V = "3.5"
val commonsLoggingV = "1.2"
val commonsTextV = "1.1"
val configsV = "0.4.4"
val errorProneAnnotationsV = "2.0.19"
val ficusV = "1.4.1"
val fs2V = "0.9.7"
val googleClientApiV = "1.22.0"
val gaxV = "1.9.0"
val googleApiClientV = "1.22.0"
val googleCloudComputeV = "0.26.0-alpha"
val googleCloudCoreV = "1.8.0"
val googleCloudNioV = "0.20.1-alpha"
val googleCredentialsV = "0.8.0"
val googleGenomicsServicesApiV = "v1alpha2-rev64-1.22.0"
val googleHttpClientV = googleApiClientV
val googleOauth2V = "0.8.0"
val googleOauthClientV = googleApiClientV
val grpcV = "1.5.0"
val guavaV = "22.0"
val hsqldbV = "2.3.4"
val jacksonV = "2.8.9"
val janinoV = "3.0.7"
val jodaTimeV = "2.9.4"
val jsr305V = "3.0.0"
val kittensV = "1.0.0-RC0"
val liquibaseSlf4jV = "2.0.0"
Expand All @@ -41,10 +50,13 @@ object Dependencies {
val pegdownV = "1.6.0"
val protoGoogleCommonProtosV = "0.1.21"
val protoGoogleIamV1V = "0.1.21"
val protobufJavaV = "3.3.1"
val ravenLogbackV = "8.0.3"
val reactiveStreamsV = "1.0.1"
val refinedV = "0.8.3"
val scalaGraphV = "1.12.0"
val scalaLoggingV = "3.6.0"
val scalaXmlV = "1.0.6"
val scalacheckV = "1.13.4"
val scalacticV = "3.0.1"
val scalameterV = "0.8.2"
Expand All @@ -55,6 +67,7 @@ object Dependencies {
val slickV = "3.2.0"
val snakeyamlV = "1.17"
val specs2MockV = "3.8.9" // 3.9.X doesn't enjoy the spark backend or refined
val sprayJsonV = "1.3.3"
val sttpV = "0.0.16"
val swaggerParserV = "1.0.22"
val swaggerUiV = "3.2.2"
Expand All @@ -64,17 +77,49 @@ object Dependencies {
If you see warnings from SBT about evictions, insert a specific dependency version into this list.
*/
val cromwellDependencyOverrides = List(
"ch.qos.logback" % "logback-classic" % logbackV,
"ch.qos.logback" % "logback-core" % logbackV,
"com.fasterxml.jackson.core" % "jackson-annotations" % jacksonV,
"com.fasterxml.jackson.core" % "jackson-core" % jacksonV,
"com.fasterxml.jackson.module" %% "jackson-module-scala" % jacksonV,
"com.google.api" % "gax" % gaxV,
"com.google.api-client" % "google-api-client" % googleApiClientV,
"com.google.api.grpc" % "proto-google-common-protos" % protoGoogleCommonProtosV,
"com.google.api.grpc" % "proto-google-iam-v1" % protoGoogleIamV1V,
"com.google.auth" % "google-auth-library-credentials" % googleCredentialsV,
"com.google.auth" % "google-auth-library-oauth2-http" % googleOauth2V,
"com.google.cloud" % "google-cloud-core" % googleCloudCoreV,
"com.google.cloud" % "google-cloud-core-http" % googleCloudCoreV,
"com.google.code.findbugs" % "jsr305" % jsr305V,
"com.google.errorprone" % "error_prone_annotations" % errorProneAnnotationsV,
"com.google.guava" % "guava" % guavaV,
"com.google.http-client" % "google-http-client" % googleHttpClientV,
"com.google.http-client" % "google-http-client-appengine" % googleHttpClientV,
"com.google.http-client" % "google-http-client-jackson" % googleHttpClientV,
"com.google.http-client" % "google-http-client-jackson2" % googleHttpClientV,
"com.google.oauth-client" % "google-oauth-client" % googleOauthClientV,
"com.google.protobuf" % "protobuf-java" % protobufJavaV,
"com.google.protobuf" % "protobuf-java-util" % protobufJavaV,
"com.typesafe" % "config" % typesafeConfigV,
"com.typesafe.akka" %% "akka-actor" % akkaV,
"com.typesafe.akka" %% "akka-stream" % akkaV,
"commons-codec" % "commons-codec" % commonsCodecV,
"commons-io" % "commons-io" % commonsIoV,
"commons-logging" % "commons-logging" % commonsLoggingV,
"eu.timepit" %% "refined" % refinedV,
"io.grpc" % "grpc-context" % grpcV,
"io.netty" % "netty-handler" % nettyHandlerV,
"io.spray" %% "spray-json" % sprayJsonV,
"joda-time" % "joda-time" % jodaTimeV,
"org.apache.commons" % "commons-lang3" % commonsLang3V,
"org.apache.httpcomponents" % "httpclient" % apacheHttpClientV,
"org.apache.httpcomponents" % "httpcore" % apacheHttpCoreV,
"org.reactivestreams" % "reactive-streams" % reactiveStreamsV,
"org.scala-lang.modules" %% "scala-xml" % scalaXmlV,
"org.slf4j" % "slf4j-api" % slf4jV,
"org.typelevel" %% "cats-core" % catsV,
"org.typelevel" %% "cats-kernel" % catsV
"org.typelevel" %% "cats-kernel" % catsV,
"org.yaml" % "snakeyaml" % snakeyamlV
)

// Internal collections of dependencies
Expand Down Expand Up @@ -127,9 +172,9 @@ object Dependencies {
"com.fasterxml.jackson.core" % "jackson-core" % jacksonV,
// The exclusions prevent guava from colliding at assembly time.
"com.google.guava" % "guava" % guavaV,
"com.google.api-client" % "google-api-client-java6" % googleClientApiV
"com.google.api-client" % "google-api-client-java6" % googleApiClientV
exclude("com.google.guava", "guava-jdk5"),
"com.google.api-client" % "google-api-client-jackson2" % googleClientApiV
"com.google.api-client" % "google-api-client-jackson2" % googleApiClientV
exclude("com.google.guava", "guava-jdk5")
)

Expand Down
146 changes: 146 additions & 0 deletions project/GenerateRestApiDocs.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
import java.time.OffsetDateTime
import java.time.format.DateTimeFormatter

import io.github.swagger2markup.Swagger2MarkupConverter
import io.github.swagger2markup.builder.Swagger2MarkupConfigBuilder
import io.github.swagger2markup.markup.builder.MarkupLanguage
import org.apache.commons.lang3.ClassUtils
import sbt._

/**
* Provides a task to generate the REST API markdown from the Swagger YAML.
*
* Uses imports provided by swagger2markup.sbt.
*
* Based partially on https://github.com/21re/sbt-swagger-plugin
*/
object GenerateRestApiDocs {
private lazy val generateRestApiDocs = taskKey[Unit]("Generates the docs/api/RESTAPI.md")

/** Returns a timestamped preamble for the modified markdown output. */
private def preamble(): String = {
val now = OffsetDateTime.now()
s"""|<!--
|This file was generated by `sbt generateRestApiDocs` on ${now.format(DateTimeFormatter.RFC_1123_DATE_TIME)}
|
|!!! DO NOT CHANGE THIS FILE DIRECTLY !!!
|
|If you wish to change something in this file, either change cromwell.yaml or GenerateRestApiDocs.scala then
|regenerate.
|-->
|""".stripMargin
}

// Path to the input swagger yaml
private val SwaggerYamlFile = new File("engine/src/main/resources/swagger/cromwell.yaml")

// Path to the generated markdown
private val RestApiMarkdownFile = new File("docs/api/RESTAPI.md")

// A regex to locate the collection of swagger paths.
private val PathsRegex = "(?s)(.*## Paths)(.*)(## Definitions.*)".r
// A regex to locate an individual swagger path within the above collection. Each is turned into "## <original text>"
private val PathRegex = "(?s)### (.*)".r

private val GenericReplacements = List(
// Change a few section headers into bolded text.
"## Overview" -> "**Overview** ",
"### Version information" -> "**Version information** ",
"### License information" -> "**License information** ",
"### Produces" -> "**Produces** ",
// Since individual paths are moved from "###" to "##" we don't need "Paths" anymore.
"<a name=\"paths\"></a>" -> "",
"## Paths" -> ""
)

/**
* Move contents of the "## Paths" section up a level.
*
* Each "### example path" will be come "## example path".
*
* Should be run before `replaceGenerics` as it looks for the string "## Paths".
*
* @param content The original contents of the RESTAPI.md.
* @return The contents with updated paths.
*/
private def replacePaths(content: String): String = {
content match {
case PathsRegex(start, paths, end) =>
val replacedPaths = paths.linesWithSeparators map {
case PathRegex(description) => s"## $description"
case other => other
}
replacedPaths.mkString(start, "", end)
case _ => throw new IllegalArgumentException(
"Content did not match expected regex. " +
"Did the swagger2markdown format change significantly? " +
"If so, a new regex may be required.")
}
}

/**
* Replaces generic strings in the generated RESTAPI.md.
*
* @param content The contents of the RESTAPI.md.
* @return The contents with generic replacements.
*/
private def replaceGenerics(content: String): String = {
GenericReplacements.foldRight(content)(replaceGeneric)
}

/**
* Replaces a single generic string in the generated RESTAPI.md.
*
* @param tokens The original and replacement strings.
* @param content The contents of the RESTAPI.md.
* @return The contents with generic replacements.
*/
private def replaceGeneric(tokens: (String, String), content: String): String = {
val (original, replacement) = tokens
content.replace(original, replacement)
}

/**
* Apache commons tries to dynamically load classes from the current thread classloader. Unfortunately the
* classloader in the current thread from SBT does NOT have all of the libraries loaded.
*
* So-- hack in ClassUtils' classloader instead, that has access to all the required libraries including
* org.apache.commons.configuration2.PropertiesConfiguration
*
* Otherwise, a ClassNotFoundException will occur.
*/
private def withPatchedClassLoader[A](block: => A): A = {
val classUtilsClassLoader = classOf[ClassUtils].getClassLoader
val currentThread = Thread.currentThread
val originalThreadClassLoader = currentThread.getContextClassLoader
try {
currentThread.setContextClassLoader(classUtilsClassLoader)
block
} finally {
currentThread.setContextClassLoader(originalThreadClassLoader)
}
}

/**
* Generates the markdown from the swagger YAML, with some Cromwell customizations.
*/
private def writeModifiedMarkdown(): Unit = {
withPatchedClassLoader {
val config = new Swagger2MarkupConfigBuilder()
.withMarkupLanguage(MarkupLanguage.MARKDOWN)
.build()
val converter = Swagger2MarkupConverter
.from(SwaggerYamlFile.toPath)
.withConfig(config)
.build()
val contents = converter.toString
val replacedContents = preamble() + replaceGenerics(replacePaths(contents))
IO.write(RestApiMarkdownFile, replacedContents)
}
}

// Returns a settings including the `generateRestApiDocs` task.
val generateRestApiDocsSettings = List[Setting[_]](
generateRestApiDocs := writeModifiedMarkdown()
)
}
1 change: 1 addition & 0 deletions project/Settings.scala
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ object Settings {
)

val engineSettings = List(resourceGenerators in Compile += writeSwaggerUiVersionConf)
val rootSettings = GenerateRestApiDocs.generateRestApiDocsSettings

private def buildProject(project: Project,
projectName: String,
Expand Down

0 comments on commit 78047f8

Please sign in to comment.