diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 697c167e..0bf91c04 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -12,7 +12,7 @@ jobs:
- uses: olafurpg/setup-scala@v10
- uses: actions/setup-go@v2
with:
- go-version: '^1.13.1'
+ go-version: "^1.13.1"
- run: go get github.com/sourcegraph/lsif-semanticdb/cmd/lsif-semanticdb
- run: sbt test
check:
diff --git a/.github/workflows/mdoc.yml b/.github/workflows/mdoc.yml
new file mode 100644
index 00000000..7d4ee799
--- /dev/null
+++ b/.github/workflows/mdoc.yml
@@ -0,0 +1,14 @@
+name: Website
+on:
+ push:
+ branches: [master, documentation-website]
+ tags: ["*"]
+jobs:
+ publish:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v2
+ - uses: olafurpg/setup-scala@v10
+ - run: sbt docs/docusaurusPublishGhpages
+ env:
+ GIT_DEPLOY_KEY: ${{ secrets.GIT_DEPLOY_KEY }}
diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index d9186ead..14c7de58 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -30,7 +30,7 @@ jobs:
- name: sbt nativeImage
shell: bash
run: |
- sbt cli/nativeImage "cli/nativeImageRun --cwd tests/gradle-example"
+ sbt cli/nativeImage "cli/nativeImageRun --cwd tests/gradle-example index"
- uses: actions/upload-artifact@master
with:
path: ${{ matrix.local_path }}
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 00000000..fd947f01
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,3 @@
+# Contributing guide
+
+See https://sourcegraph.github.io/lsif-java/docs/contributing.html
diff --git a/README.md b/README.md
index 97336cfe..027a58f6 100644
--- a/README.md
+++ b/README.md
@@ -1,175 +1,9 @@
-# Java LSIF indexer 
-
-Visit https://lsif.dev/ to learn about LSIF.
-
-## Usage
-
-⚠️ This project is under development so there is nothing to try out at the
-moment.
-
-### Supported tools and versions
-
-Currently, only Java 8 with the build tool sbt is supported. We hope to increase
-compatibility with more Java language versions and build tools as the project
-evolves.
-
-| Language version | Support |
-| ---------------- | --------------------------------- |
-| Java 7 | ❌ |
-| Java 8 | ✅ |
-| Java 11 | ✅ |
-| Java 12 | Not tested in CI, but should work |
-| Java 13 | Not tested in CI, but should work |
-| Java 14 | Not tested in CI, but should work |
-| Java 15 | ✅ |
-| Java 16 | Not tested in CI, but should work |
-| Java 17 | Not tested in CI, but should work |
-
-| Build tool | Support |
-| ---------- | ------- |
-| Gradle | ✅ |
-| Maven | ✅ |
-| Bazel | ❌ |
-| Buck | ❌ |
-| sbt | ✅ |
-
-## Overview
-
-This project is implemented as a
-[Java compiler plugin](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.compiler/com/sun/source/util/Plugin.html)
-that generates one
-[SemanticDB](https://scalameta.org/docs/semanticdb/specification.html) file for
-every `*.java` source file. After compilation completes, the SemanticDB files
-are processed to produce LSIF.
-
-
-
-### Why Java compiler plugin?
-
-There are several benefits to implementing lsif-java as a compiler plugin:
-
-- **Simple installation**: compiler plugins are enabled with the `-Xplugin`
- compiler option. All Java build tools support a way to customize compiler
- options, simplifying installation.
-- **Language fidelity**: by using the Java compiler to produce semantic
- information, we ensure that the produced LSIF data is accurate even as new
- Java language versions with new language features are released.
-- **Environment fidelity**: by hooking into the compilation process of the build
- tool, we minimize the risk of diverging from the CI build environment such as
- installed system dependencies, custom compiler options and custom annotation
- processors.
-
-### Why SemanticDB?
-
-SemanticDB is Protobuf schema for information about symbols and types in Java
-programs, Scala programs and other languages. There are several benefits to
-using SemanticDB as an intermediary representation for LSIF:
-
-- **Simplicity**: It's easy to translate a single Java source file into a single
- SemanticDB file inside a compiler plugin. It's more complicated to produce
- LSIF because compiler plugins does not have access to a project-wide context,
- which is necessary to produce accurate definitions and hovers in multi-module
- projects with external library dependencies.
-- **Performance**: SemanticDB is fast to write and read. Each compilation unit
- can be processed independently to keep memory usage low. The final conversion
- from SemanticDB to LSIF can be safely parallelized.
-- **Cross-language**: SemanticDB has a
- [spec](https://scalameta.org/docs/semanticdb/specification.html) for Java and
- Scala enabling cross-language navigation in hybrid Java/Scala codebases.
-- **Cross-repository**: Compiler plugins have access to both source code and the
- classpath (compiled bytecode of upstream dependencies). SemanticDB has been
- designed so that it's also possible to generate spec-compliant symbols from
- the classpath alone (no source code) and from the syntax tree of an individual
- source file (no classpath). This flexibility allows the
- [Metals](https://scalameta.org/metals/) language server to index codebases
- from a variety of different inputs, and will be helpful for lsif-java in the
- future to unblock cross-repository navigation.
-
-## Contributing
-
-The following sections provide tips on how to contribute to this codebase.
-
-### System dependencies
-
-- `java`: any version should work
-- `git`: any version should work
-- `lsif-semanticdb`:
- `go get github.com/sourcegraph/lsif-semanticdb/cmd/lsif-semanticdb`
-- `gradle`: `brew install gradle`, or see
- [general installation guide](https://gradle.org/install/).
-- `mvn`: `brew install maven`, or see
- [general installation guide](https://www.baeldung.com/install-maven-on-windows-linux-mac).
-
-### Project structure
-
-These are the main components of the project.
-
-- `semanticdb-javac/src/main/java`: the Java compiler plugin that creates
- SemanticDB files.
-- `tests/minimized`: minimized Java source files that reproduce interesting test
- cases.
-- `tests/unit`: fast running unit tests that are helpful for local edit-and-test
- workflows.
-- `tests/snapshots`: slow running
- ["snapshot tests"](https://jestjs.io/docs/en/snapshot-testing) that index a
- corpus of published Java libraries.
-- `build.sbt`: the sbt build definition.
-- `project/plugins.sbt`: plugins for the sbt build.
-
-### Helpful commands
-
-| Command | Where | Description |
-| ------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------- |
-| `./sbt` | terminal | Start interactive sbt shell with Java 11. Takes a while to load on the first run. |
-| `unit/test` | sbt | Run fast unit tests. |
-| `~unit/test` | sbt | Start watch mode to run tests on file save, good for local edit-and-test workflows. |
-| `buildTools/test` | sbt | Run slow build tool tests (Gradle, Maven). |
-| `snapshots/testOnly tests.MinimizedSnapshotSuite` | sbt | Runs fast snapshot tests. Indexes a small set of files under `tests/minimized`. |
-| `snapshots/testOnly tests.MinimizedSnapshotSuite -- *InnerClasses*` | sbt | Runs only individual tests cases matching the name "InnerClasses". |
-| `snapshots/testOnly tests.LibrarySnapshotSuite` | sbt | Runs slow snapshot tests. Indexes a corpus of external Java libraries. |
-| `snapshots/test` | sbt | Runs all snapshot tests. |
-| `snapshots/run` | sbt | Update snapshot tests. Use this command after you have fixed a bug. |
-| `cli/run --cwd DIRECTORY` | sbt | Run `lsif-java` command-line tool against a given Gradle/Maven build. |
-| `fixAll` | sbt | Run Scalafmt, Scalafix and Javafmt on all sources. Run this before opening a PR. |
-
-### Import the project into IntelliJ
-
-It's recommended to use IntelliJ when editing code in this codebase.
-
-First, install the
-[IntelliJ Community Edition](https://www.jetbrains.com/idea/download/). The
-community edition is
-[open source](https://github.com/JetBrains/intellij-community) and free to use.
-
-Next, install the IntelliJ Scala plugin.
-
-Finally, run "File > Project From Existing Sources" to import the sbt build into
-IntelliJ. Select the "sbt" option if it asks you to choose between
-sbt/BSP/Bloop.
-
-It's best to run tests from the sbt shell, not from the IntelliJ UI.
-
-### Don't use VS Code/Vim/Sublime Text/Emacs
-
-If you want to use completions and precise code navigation, it's not recommended
-to use other editors than IntelliJ. IntelliJ is the only IDE that properly
-supports hybrid Java/Scala codebases at the moment, although that may change
-soon thanks to lsif-java :)
-
-### Tests are written in Scala
-
-This codebases uses the Scala library [MUnit](https://scalameta.org/munit/) to
-write tests because:
-
-- MUnit has built-in assertions that print readable multiline diffs in color.
-- MUnit makes it easy to implement
- [snapshot testing](https://jestjs.io/docs/en/snapshot-testing), which is a
- testing technique that's heavily used in this codebase.
-- Multiline literal strings in Scala make it easy to write unit tests for source
- code (which is always multiline). Modern versions of Java support multiline
- string literals, but they're not supported in Java 8, which is supported by
- lsif-java.
-
-## Benchmarks
-
-See [docs/benchmarks.md] for benchmark results.
+# Java indexer for the Language Server Index Format (LSIF) 
+
+| Documentation | Link |
+| -------------------- | ---------------------------------------------------------------------- |
+| Landing page | https://sourcegraph.github.io/lsif-java |
+| Getting started | https://sourcegraph.github.io/lsif-java/docs/getting-started.html |
+| Manual configuration | https://sourcegraph.github.io/lsif-java/docs/manual-configuration.html |
+| Contributing | https://sourcegraph.github.io/lsif-java/docs/contributing.html |
+| Design | https://sourcegraph.github.io/lsif-java/docs/design.html |
diff --git a/build.sbt b/build.sbt
index a1a8dc72..f65b4f36 100644
--- a/build.sbt
+++ b/build.sbt
@@ -1,3 +1,5 @@
+import scala.xml.{Node => XmlNode, NodeSeq => XmlNodeSeq, _}
+import scala.xml.transform.{RewriteRule, RuleTransformer}
import java.io.File
import java.util.Properties
import scala.collection.mutable.ListBuffer
@@ -59,7 +61,7 @@ commands +=
commands +=
Command.command("checkAll") { s =>
"scalafmtCheckAll" :: "scalafmtSbtCheck" :: "scalafixAll --check" ::
- "javafmtCheckAll" :: "publishLocal" :: s
+ "javafmtCheckAll" :: "publishLocal" :: "docs/docusaurusCreateSite" :: s
}
lazy val agent = project
@@ -97,6 +99,16 @@ lazy val plugin = project
old.withEnabled(false)
},
fatjarPackageSettings,
+ assemblyShadeRules.in(assembly) :=
+ Seq(
+ ShadeRule
+ .rename(
+ "com.google.**" -> "com.sourcegraph.shaded.com.google.@1",
+ "google.**" -> "com.sourcegraph.shaded.google.@1",
+ "org.relaxng.**" -> "com.sourcegraph.shaded.relaxng.@1"
+ )
+ .inAll
+ ),
crossPaths := false,
PB.targets.in(Compile) :=
Seq(PB.gens.java -> (Compile / sourceManaged).value)
@@ -280,7 +292,7 @@ lazy val fatjarPackageSettings = List[Def.Setting[_]](
case PathList("sun", _ @_*) =>
MergeStrategy.discard
case PathList("META-INF", "versions", "9", "module-info.class") =>
- MergeStrategy.first
+ MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
@@ -291,5 +303,48 @@ lazy val fatjarPackageSettings = List[Def.Setting[_]](
val _ = assembly.value
IO.copyFile(fatJar, slimJar, CopyOptions().withOverwrite(true))
slimJar
+ },
+ packagedArtifact.in(Compile).in(packageBin) := {
+ val (art, slimJar) = packagedArtifact.in(Compile).in(packageBin).value
+ val fatJar =
+ new File(crossTarget.value + "/" + assemblyJarName.in(assembly).value)
+ val _ = assembly.value
+ IO.copy(List(fatJar -> slimJar), CopyOptions().withOverwrite(true))
+ (art, slimJar)
+ },
+ pomPostProcess := { node =>
+ new RuleTransformer(
+ new RewriteRule {
+ private def isAbsorbedDependency(node: XmlNode): Boolean = {
+ node.label == "dependency" &&
+ node.child.exists(child => child.label == "artifactId")
+ }
+ override def transform(node: XmlNode): XmlNodeSeq =
+ node match {
+ case e: Elem if isAbsorbedDependency(node) =>
+ Comment(
+ "the dependency that was here has been absorbed via sbt-assembly"
+ )
+ case _ =>
+ node
+ }
+ }
+ ).transform(node).head
}
)
+
+lazy val docs = project
+ .in(file("lsif-java-docs"))
+ .settings(
+ mdocOut :=
+ baseDirectory.in(ThisBuild).value / "website" / "target" / "docs",
+ fork := false,
+ mdocVariables :=
+ Map[String, String](
+ "VERSION" -> version.value,
+ "SCALA_VERSION" -> scalaVersion.value,
+ "STABLE_VERSION" -> version.value.replaceFirst("\\-.*", "")
+ )
+ )
+ .dependsOn(unit)
+ .enablePlugins(DocusaurusPlugin)
diff --git a/cli/src/main/scala/com/sourcegraph/lsif_java/IndexCommand.scala b/cli/src/main/scala/com/sourcegraph/lsif_java/IndexCommand.scala
index cbc7d7ff..4e8e4cab 100644
--- a/cli/src/main/scala/com/sourcegraph/lsif_java/IndexCommand.scala
+++ b/cli/src/main/scala/com/sourcegraph/lsif_java/IndexCommand.scala
@@ -6,9 +6,11 @@ import java.nio.file.Paths
import com.sourcegraph.lsif_java.buildtools.BuildTool
import moped.annotations.Description
+import moped.annotations.ExampleUsage
import moped.annotations.ExampleValue
import moped.annotations.Inline
import moped.annotations.TrailingArguments
+import moped.annotations.Usage
import moped.cli.Application
import moped.cli.Command
import moped.cli.CommandParser
@@ -18,7 +20,12 @@ import os.Inherit
import os.Shellable
@Description(
- "Generates an LSIF index for the Java build of a provided workspace directory."
+ "Automatically generate an LSIF index in the current working directory."
+)
+@Usage("lsif-java index [OPTIONS ...] -- [TRAILING_ARGUMENTS ...]")
+@ExampleUsage(
+ """|# Running the `index` command with no flags should work most of the time.
+ |$ lsif-java index""".stripMargin
)
case class IndexCommand(
@Description("The path where to generate the LSIF index.") output: Path =
@@ -41,12 +48,12 @@ case class IndexCommand(
)
@ExampleValue("Gradle") buildTool: Option[String] = None,
@Description(
- "Whether to enable remove generated temporary files on exit."
+ "Whether to remove generated temporary files on exit."
) cleanup: Boolean = true,
@Description(
- "The build command to use to compile all sources. " +
+ "Optional. The build command to use to compile all sources. " +
"Defaults to a build-specific command. For example, the default command for Maven command is 'clean verify -DskipTests'." +
- "To override the default, pass in the build command after a double dash: 'lsif-java -- compile'"
+ "To override the default, pass in the build command after a double dash: 'lsif-java index -- compile test:compile'"
)
@TrailingArguments() buildCommand: List[String] = Nil,
@Inline
@@ -79,6 +86,7 @@ case class IndexCommand(
def workingDirectory: Path = AbsolutePath.of(app.env.workingDirectory)
def finalTargetroot(default: Path): Path =
AbsolutePath.of(targetroot.getOrElse(default), workingDirectory)
+ def finalOutput: Path = AbsolutePath.of(output, workingDirectory)
def finalBuildCommand(default: List[String]): List[String] =
if (buildCommand.isEmpty)
default
@@ -142,8 +150,14 @@ case class IndexCommand(
} else {
val generateLsifResult = process(
"lsif-semanticdb",
+ s"--out=${finalOutput}",
s"--semanticdbDir=${tool.targetroot}"
)
+ if (
+ generateLsifResult.exitCode == 0 && Files.isRegularFile(finalOutput)
+ ) {
+ app.info(finalOutput.toAbsolutePath().toString())
+ }
generateSemanticdbResult.exitCode + generateLsifResult.exitCode
}
case many =>
diff --git a/cli/src/main/scala/com/sourcegraph/lsif_java/IndexSemanticdbCommand.scala b/cli/src/main/scala/com/sourcegraph/lsif_java/IndexSemanticdbCommand.scala
new file mode 100644
index 00000000..55449b99
--- /dev/null
+++ b/cli/src/main/scala/com/sourcegraph/lsif_java/IndexSemanticdbCommand.scala
@@ -0,0 +1,55 @@
+package com.sourcegraph.lsif_java
+
+import java.nio.file.Path
+
+import scala.collection.mutable.ListBuffer
+
+import moped.annotations.CommandName
+import moped.annotations.Description
+import moped.annotations.ExampleUsage
+import moped.annotations.Inline
+import moped.annotations.PositionalArguments
+import moped.annotations.Usage
+import moped.cli.Application
+import moped.cli.Command
+import moped.cli.CommandParser
+import os.Inherit
+import os.Shellable
+
+@Description("Converts SemanticDB files into a single LSIF index file.")
+@Usage("lsif-java index-semanticdb [OPTIONS ...] [POSITIONAL ARGUMENTS ...]")
+@ExampleUsage(
+ "lsif-java index-semanticdb --out=myindex.lsif my/targetroot1 my/targetroot2"
+)
+@CommandName("index-semanticdb")
+final case class IndexSemanticdbCommand(
+ @Description(
+ "The name of the output file. Defaults to 'dump.lsif'"
+ ) out: Option[Path] = None,
+ @Description(
+ "SemanticDB file paths or directories that contain SemanticDB files."
+ )
+ @PositionalArguments() directories: List[Path] = Nil,
+ @Inline() app: Application = Application.default
+) extends Command {
+ def run(): Int = {
+ val arguments = ListBuffer.empty[String]
+ arguments += "lsif-semanticdb"
+ out.foreach { dir =>
+ arguments += s"--out=$dir"
+ }
+ directories.foreach { dir =>
+ arguments += s"--semanticdbDir=$dir"
+ }
+ app.info(arguments.mkString(" "))
+ app
+ .process(Shellable(arguments.toList))
+ .call(check = false, stderr = Inherit, stdout = Inherit)
+ .exitCode
+ }
+}
+
+object IndexSemanticdbCommand {
+ val default = IndexSemanticdbCommand()
+ implicit val parser = CommandParser.derive(default)
+}
diff --git a/cli/src/main/scala/com/sourcegraph/lsif_java/LsifJava.scala b/cli/src/main/scala/com/sourcegraph/lsif_java/LsifJava.scala
index c9d6e9cc..d294755b 100644
--- a/cli/src/main/scala/com/sourcegraph/lsif_java/LsifJava.scala
+++ b/cli/src/main/scala/com/sourcegraph/lsif_java/LsifJava.scala
@@ -1,23 +1,35 @@
package com.sourcegraph.lsif_java
+import java.io.PrintStream
+
import moped.cli.Application
import moped.cli.CommandParser
import moped.commands.HelpCommand
import moped.commands.VersionCommand
+import moped.reporters.Tput
object LsifJava {
- val app: Application = Application
- .fromName(
- binaryName = "lsif-java",
- BuildInfo.version,
- List(
- CommandParser[HelpCommand],
- CommandParser[VersionCommand],
- CommandParser[IndexCommand]
- )
+ val app: Application = Application.fromName(
+ binaryName = "lsif-java",
+ BuildInfo.version,
+ List(
+ CommandParser[HelpCommand],
+ CommandParser[VersionCommand],
+ CommandParser[IndexCommand],
+ CommandParser[IndexSemanticdbCommand]
)
- .withIsSingleCommand(true)
+ )
def main(args: Array[String]): Unit = {
app.runAndExitIfNonZero(args.toList)
}
+
+ def printHelp(out: PrintStream): Unit = {
+ out.println("```text")
+ out.println("$ lsif-java index --help")
+ val newApp = app
+ .withTput(Tput.constant(100))
+ .withEnv(app.env.withStandardOutput(out))
+ newApp.run(List("index", "--help"))
+ out.println("```")
+ }
}
diff --git a/docs/img/semanticdb-javac-pipeline.svg b/docs/assets/semanticdb-javac-pipeline.svg
similarity index 100%
rename from docs/img/semanticdb-javac-pipeline.svg
rename to docs/assets/semanticdb-javac-pipeline.svg
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index 87640660..6911614d 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -1,7 +1,15 @@
-# Benchmarks results
+---
+id: benchmarks
+title: Benchmarks
+---
+
+The repository contains benchmarks to measure the overhead of the SemanticDB
+compiler plugin.
```
-sbt:root> bench/jmh:run -i 3 -wi 3 -f1 -t1
+$ sbt
+...
+sbt:root> bench/jmh:run -i 10 -wi 10 -f1 -t1
...
[info] Benchmark (lib) Mode Cnt Score Error Units
[info] CompileBench.compile guava ss 10 2291.036 ± 243.428 ms/op 1x
diff --git a/docs/contributing.md b/docs/contributing.md
new file mode 100644
index 00000000..e7789c0a
--- /dev/null
+++ b/docs/contributing.md
@@ -0,0 +1,93 @@
+---
+id: contributing
+sidebar_label: Guide
+title: Contributing guide
+---
+
+This page documents tips and tricks for contributing to the
+[sourcegraph/lsif-java](https://github.com/sourcegraph/lsif-java) codebase.
+
+## System dependencies
+
+- `java`: any version should work
+- `git`: any version should work
+- `lsif-semanticdb`:
+ `go get github.com/sourcegraph/lsif-semanticdb/cmd/lsif-semanticdb`
+- `gradle`: `brew install gradle`, or see
+ [general installation guide](https://gradle.org/install/).
+- `mvn`: `brew install maven`, or see
+ [general installation guide](https://www.baeldung.com/install-maven-on-windows-linux-mac).
+
+## Project structure
+
+These are the main components of the project.
+
+- `semanticdb-javac/src/main/java`: the Java compiler plugin that creates
+ SemanticDB files.
+- `tests/minimized`: minimized Java source files that reproduce interesting test
+ cases.
+- `tests/unit`: fast running unit tests that are helpful for local edit-and-test
+ workflows.
+- `tests/snapshots`: slow running
+ ["snapshot tests"](https://jestjs.io/docs/en/snapshot-testing) that index a
+ corpus of published Java libraries.
+- `cli/src/main/scala`: implementation of the `lsif-java` command-line
+ interface.
+- `build.sbt`: the sbt build definition.
+- `project/plugins.sbt`: plugins for the sbt build.
+
+## Helpful commands
+
+| Command | Where | Description |
+| ------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------- |
+| `./sbt` | terminal | Start interactive sbt shell with Java 11. Takes a while to load on the first run. |
+| `unit/test` | sbt | Run fast unit tests. |
+| `~unit/test` | sbt | Start watch mode to run tests on file save, good for local edit-and-test workflows. |
+| `buildTools/test` | sbt | Run slow build tool tests (Gradle, Maven). |
+| `snapshots/testOnly tests.MinimizedSnapshotSuite` | sbt | Runs fast snapshot tests. Indexes a small set of files under `tests/minimized`. |
+| `snapshots/testOnly tests.MinimizedSnapshotSuite -- *InnerClasses*` | sbt | Runs only individual tests cases matching the name "InnerClasses". |
+| `snapshots/testOnly tests.LibrarySnapshotSuite` | sbt | Runs slow snapshot tests. Indexes a corpus of external Java libraries. |
+| `snapshots/test` | sbt | Runs all snapshot tests. |
+| `snapshots/run` | sbt | Update snapshot tests. Use this command after you have fixed a bug. |
+| `cli/run --cwd DIRECTORY` | sbt | Run `lsif-java` command-line tool against a given Gradle/Maven build. |
+| `cd website && yarn install && yarn start` | terminal | Start live-reload preview of the website at http://localhost:3000/lsif-java. |
+| `docs/mdoc --watch` | sbt | Re-generate markdown files in the `docs/` directory. |
+| `fixAll` | sbt | Run Scalafmt, Scalafix and Javafmt on all sources. Run this before opening a PR. |
+
+## Import the project into IntelliJ
+
+It's recommended to use IntelliJ when editing code in this codebase.
+
+First, install the
+[IntelliJ Community Edition](https://www.jetbrains.com/idea/download/). The
+community edition is
+[open source](https://github.com/JetBrains/intellij-community) and free to use.
+
+Next, install the IntelliJ Scala plugin.
+
+Finally, run "File > Project From Existing Sources" to import the sbt build into
+IntelliJ. Select the "sbt" option if it asks you to choose between
+sbt/BSP/Bloop.
+
+It's best to run tests from the sbt shell, not from the IntelliJ UI.
+
+## Don't use VS Code/Vim/Sublime Text/Emacs
+
+If you want to use completions and precise code navigation, it's not recommended
+to use other editors than IntelliJ. IntelliJ is the only IDE that properly
+supports hybrid Java/Scala codebases at the moment, although that may change
+soon thanks to lsif-java :)
+
+## Tests are written in Scala
+
+This codebases uses the Scala library [MUnit](https://scalameta.org/munit/) to
+write tests because:
+
+- MUnit has built-in assertions that print readable multiline diffs in color.
+- MUnit makes it easy to implement
+ [snapshot testing](https://jestjs.io/docs/en/snapshot-testing), which is a
+ testing technique that's heavily used in this codebase.
+- Multiline literal strings in Scala make it easy to write unit tests for source
+ code (which is always multiline). Modern versions of Java support multiline
+ string literals, but they're not supported in Java 8, which is supported by
+ lsif-java.
diff --git a/docs/design.md b/docs/design.md
new file mode 100644
index 00000000..d908e78d
--- /dev/null
+++ b/docs/design.md
@@ -0,0 +1,54 @@
+---
+id: design
+title: Design
+---
+
+This project is implemented as a
+[Java compiler plugin](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.compiler/com/sun/source/util/Plugin.html)
+that generates one
+[SemanticDB](https://scalameta.org/docs/semanticdb/specification.html) file for
+every `*.java` source file. After compilation completes, the SemanticDB files
+are processed to produce LSIF.
+
+
+
+### Why Java compiler plugin?
+
+There are several benefits to implementing lsif-java as a compiler plugin:
+
+- **Simple installation**: compiler plugins are enabled with the `-Xplugin`
+ compiler option. All Java build tools support a way to customize compiler
+ options, simplifying installation.
+- **Language fidelity**: by using the Java compiler to produce semantic
+ information, we ensure that the produced LSIF data is accurate even as new
+ Java language versions with new language features are released.
+- **Environment fidelity**: by hooking into the compilation process of the build
+ tool, we minimize the risk of diverging from the CI build environment such as
+ installed system dependencies, custom compiler options and custom annotation
+ processors.
+
+### Why SemanticDB?
+
+SemanticDB is Protobuf schema for information about symbols and types in Java
+programs, Scala programs and other languages. There are several benefits to
+using SemanticDB as an intermediary representation for LSIF:
+
+- **Simplicity**: It's easy to translate a single Java source file into a single
+ SemanticDB file inside a compiler plugin. It's more complicated to produce
+ LSIF because compiler plugins does not have access to a project-wide context,
+ which is necessary to produce accurate definitions and hovers in multi-module
+ projects with external library dependencies.
+- **Performance**: SemanticDB is fast to write and read. Each compilation unit
+ can be processed independently to keep memory usage low. The final conversion
+ from SemanticDB to LSIF can be safely parallelized.
+- **Cross-language**: SemanticDB has a
+ [spec](https://scalameta.org/docs/semanticdb/specification.html) for Java and
+ Scala enabling cross-language navigation in hybrid Java/Scala codebases.
+- **Cross-repository**: Compiler plugins have access to both source code and the
+ classpath (compiled bytecode of upstream dependencies). SemanticDB has been
+ designed so that it's also possible to generate spec-compliant symbols from
+ the classpath alone (no source code) and from the syntax tree of an individual
+ source file (no classpath). This flexibility allows the
+ [Metals](https://scalameta.org/metals/) language server to index codebases
+ from a variety of different inputs, and will be helpful for lsif-java in the
+ future to unblock cross-repository navigation.
diff --git a/docs/getting-started.md b/docs/getting-started.md
new file mode 100644
index 00000000..852e8664
--- /dev/null
+++ b/docs/getting-started.md
@@ -0,0 +1,165 @@
+---
+id: getting-started
+title: Getting started
+---
+
+By following the instructions on this page, you should be able to generate an
+[LSIF](https://microsoft.github.io/language-server-protocol/specifications/lsif/0.5.0/specification/)
+index of your Java codebase using Gradle or Maven. See
+[Supported build tools](#supported-build-tools) for an overview of other build
+tools that we're planning to support in the future.
+
+## Install `lsif-java`
+
+Most users only install `lsif-java` on a CI machine to upload LSIF indexes to a
+remote service like [Sourcegraph](https://sourcegraph.com/). The easiest way to
+install `lsif-java` is to download the native binary. However, you can also
+install `lsif-java` as a Java binary if that's easier to integrate with your
+setup.
+
+### Native binary
+
+The native binary is only available for Linux and macOS. The native binary
+includes all dependencies and does not need further access to the internet after
+it's been downloaded.
+
+```sh
+# macOS
+curl -Lo lsif-java https://github.com/sourcegraph/lsif-java/releases/download/@STABLE_VERSION@/lsif-java-x86_64-apple-darwin
+chmod +x lsif-java
+./lsif-java --help
+
+# Linux
+curl -Lo lsif-java https://github.com/sourcegraph/lsif-java/releases/download/@STABLE_VERSION@/lsif-java-x86_64-pc-linux
+chmod +x lsif-java
+./lsif-java --help
+```
+
+### Java launcher
+
+Use [Coursier](https://get-coursier.io/docs/cli-installation.html) to launch the
+Java binary. The jar files of `lsif-java` are downloaded the first time you run
+the `launch` command, and they're cached for subsequent runs.
+
+```sh
+# Homebrew
+brew install coursier/formulas/coursier
+coursier launch com.sourcegraph:lsif-java_2.13:@STABLE_VERSION@ -- --help
+
+# macOS/Linux
+curl -fLo coursier https://git.io/coursier-cli
+chmod +x coursier
+./coursier launch com.sourcegraph:lsif-java_2.13:@STABLE_VERSION@ -- --help
+
+# Windows
+bitsadmin /transfer downloadCoursierCli https://git.io/coursier-cli "%cd%\coursier"
+bitsadmin /transfer downloadCoursierBat https://git.io/coursier-bat "%cd%\coursier.bat"
+./coursier launch com.sourcegraph:lsif-java_2.13:@STABLE_VERSION@ -- --help
+```
+
+### Java fat jar
+
+Use the Coursier `bootstrap` command to generate an executable Java binary,
+which includes all dependencies and does not require further access to the
+internet after installation.
+
+```sh
+# macOS/Linux/Windows
+cs bootstrap --standalone -o lsif-java com.sourcegraph:lsif-java_2.13:@STABLE_VERSION@
+./lsif-java --help
+```
+
+### Java library
+
+The `lsif-java` command-line interface is published to Maven Central. You can
+run the command-line interface as a library by directly invoking the `main()`
+method on the `com.sourcegraph.lsif_java.LsifJava` class.
+
+[](https://repo1.maven.org/maven2/com/sourcegraph/lsif-java_2.13/)
+
+If you're using Gradle.
+
+```groovy
+implementation group: 'com.sourcegraph', name: 'lsif-java_2.13', version: '@STABLE_VERSION@'
+```
+
+If you're using Maven.
+
+```xml
+
+ com.sourcegraph
+ lsif-java_2.13
+ @STABLE_VERSION@
+
+```
+
+If you're using sbt.
+
+```scala
+scalaVersion := "@SCALA_VERSION@" // Only Scala 2.13 is supported.
+libraryDependencies += "com.sourcegraph" %% "lsif-java" % "@STABLE_VERSION@"
+```
+
+## Run `lsif-java index`
+
+Run the `lsif-java index` command to generate an LSIF index for your codebase.
+This command should automatically infer the structure of your codebase and
+configure your build tool to generate LSIF.
+
+```sh
+$ lsif-java index
+...
+info: /path/to/dump.lsif
+```
+
+The `dump.lsif` file contains the LSIF index and is ready to be used.
+
+> Running `lsif-java index` may perform side-effects on your build like cleaning
+> the compile cache before indexing. This is done to make sure that all source
+> files in the codebase get indexed.
+
+Run `lsif-java index --help` to learn more about the available command-line
+options.
+
+```scala mdoc:passthrough
+com.sourcegraph.lsif_java.LsifJava.printHelp(Console.out)
+```
+
+## Supported Java versions
+
+The `lsif-java` indexer is implemented as a Java compiler plugin that runs as
+part of your regular compilation in the build tool. By using Java compiler APIs,
+`lsif-java` is able to generate accurate indexing information for a broad range
+of Java versions.
+
+| Language version | Support |
+| ---------------- | ------------------------------ |
+| Java 7 | ❌ |
+| Java 8 | ✅ |
+| Java 11 | ✅ |
+| Java 12 | Not tested in CI, but may work |
+| Java 13 | Not tested in CI, but may work |
+| Java 14 | Not tested in CI, but may work |
+| Java 15 | ✅ |
+| Java 16 | Not tested in CI, but may work |
+| Java 17 | Not tested in CI, but may work |
+
+## Supported build tools
+
+Please open an issue if your build tool is not listed in the table below. Feel
+free to subscribe to the tracking issues to receive updates on your build tool.
+
+| Build tool | Support | Tracking issue |
+| ---------- | ------- | -------------------------------------------------------------------------------- |
+| Gradle | ✅ | |
+| Maven | ✅ | |
+| Bazel | ❌ | [sourcegraph/lsif-java#88](https://github.com/sourcegraph/lsif-java/issues/88) |
+| Buck | ❌ | [sourcegraph/lsif-java#99](https://github.com/sourcegraph/lsif-java/issues/99) |
+| sbt | ❌ | [sourcegraph/lsif-java#110](https://github.com/sourcegraph/lsif-java/issues/110) |
+
+**✅**: automatic indexing is fully supported. Please report a bug if the
+`lsif-java index` command does not work on your codebase.
+
+**❌**: automatic inference is not supported but (!) you may still be able to
+use `lsif-java` by configuring it manually using the instructions
+[here](manual-configuration.md).
diff --git a/docs/manual-configuration.md b/docs/manual-configuration.md
new file mode 100644
index 00000000..a19ad74c
--- /dev/null
+++ b/docs/manual-configuration.md
@@ -0,0 +1,139 @@
+---
+id: manual-configuration
+title: Manual configuration
+---
+
+The `lsif-java index` command does a best-effort to automatically index a given
+codebase. The automatic step may not always work. The purpose of this page is to
+document the steps to manually configure lsif-java when automatic indexing
+fails.
+
+## Overview
+
+Indexing a codebase consists of two independent phases:
+
+- Compile the codebase with the SemanticDB compiler plugin.
+- Generate LSIF index from SemanticDB files.
+
+
+
+The first phase can be complicated to configure and it can take a while to run.
+The second phase is quite simple to configure and it usually runs very fast.
+
+## Step 1: Add SemanticDB compiler plugin to the classpath
+
+The SemanticDB compiler plugin is published to Maven Central as a
+zero-dependency Java library that you can install like any normal dependency.
+
+[](https://repo1.maven.org/maven2/com/sourcegraph/semanticdb-javac/)
+
+If you're using Gradle.
+
+```groovy
+implementation group: 'com.sourcegraph', name: 'semanticdb-javac', version: '@STABLE_VERSION@'
+```
+
+If you're using Maven.
+
+```xml
+
+ com.sourcegraph
+ semanticdb-javac
+ @STABLE_VERSION@
+
+```
+
+If you're using sbt.
+
+```scala
+libraryDependencies += "com.sourcegraph" % "semanticdb-javac" % "@STABLE_VERSION@"
+```
+
+## Step 2: Enable `-Xplugin:semanticdb` compiler option
+
+Add `-Xplugin:semanticdb` to your compiler options to enable the SemanticDB
+compiler plugin. To do this you need to explicitly configure two directories:
+
+- `-sourceroot:PATH`: the absolute path to the root directory of your codebase.
+ It's important that all of the source files that should be index live under
+ this directory.
+- `-targetroot:PATH`: the absolute path to the directory where to generate
+ SemanticDB file. This directory can be anywhere on your file system.
+
+If you're using Gradle.
+
+```groovy
+tasks.withType(JavaCompile) {
+ def sourceroot = rootProject.projectDir
+ def targetroot = new File(rootProject.buildDir, "semanticdb-targetroot")
+ options.compilerArgs << "-Xplugin:semanticdb -sourceroot:$sourceroot -targetroot:$targetroot"
+}
+```
+
+If you're using Maven.
+
+```diff
+
+ [...]
+
+ [...]
+
+
+ org.apache.maven.plugins
+ maven-compiler-plugin
+ 3.8.1
+
+
++ -Xplugin:semanticdb -sourceroot:${session.executionRootDirectory} -targetroot:${session.executionRootDirectory}/target/semanticdb-targetroot
+
+
+
+
+ [...]
+
+ [...]
+
+```
+
+If you're using sbt.
+
+```scala
+javaHome := Some(...) // Must be configured to fork the compiler.
+Compile / javacOptions += {
+ val sourceroot = (ThisBuild / baseDirectory).value
+ val targetroot = sourceroot / "target" / "semanticdb-targetroot"
+ s"-Xplugin:semanticdb -sourceroot:$sourceroot -targetroot:$targetroot"
+}
+```
+
+## Step 3: Compile the codebase
+
+Compile all source files in the codebase once the compiler setting has been
+configured. The exact command depends on your build tool. Below are some
+examples:
+
+- Gradle: `./gradlew clean compileJava compileTestJava`
+- Maven: `mvn clean verify -DskipTests`
+- sbt: `sbt clean test:compile`
+- Bazel: `bazel build //...`
+
+If everything went well, you should have a lot of `*.semanticdb` files in the
+targetroot directory.
+
+```
+❯ find $TARGETROOT -type f
+build/semanticdb-targetroot/META-INF/semanticdb/j8/src/test/java/example/ExampleTest.java.semanticdb
+build/semanticdb-targetroot/META-INF/semanticdb/j8/src/main/java/example/Example.java.semanticdb
+...
+```
+
+## Step 4: Generate LSIF index from SemanticDB files
+
+Run the `lsif-java index-semanticdb` command to convert SemanticDB files into
+LSIF.
+
+```sh
+❯ lsif-java index-semanticdb $TARGETROOT
+❯ file dump.lsif
+dump.lsif: JSON data
+```
diff --git a/project/plugins.sbt b/project/plugins.sbt
index ef3b25f0..6349e1b4 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -2,6 +2,7 @@ addSbtPlugin("org.scalameta" % "sbt-native-image" % "0.3.0")
addSbtPlugin("com.geirsson" % "sbt-ci-release" % "1.5.5")
addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.10.0")
addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.3.4")
+addSbtPlugin("org.scalameta" % "sbt-mdoc" % "2.2.18")
addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.9.25")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.0")
addSbtPlugin("ch.epfl.scala" % "sbt-bloop" % "1.4.6-21-464e4ec4")
diff --git a/tests/buildTools/src/test/scala/tests/BaseBuildToolSuite.scala b/tests/buildTools/src/test/scala/tests/BaseBuildToolSuite.scala
index 9827770c..6171e0ae 100644
--- a/tests/buildTools/src/test/scala/tests/BaseBuildToolSuite.scala
+++ b/tests/buildTools/src/test/scala/tests/BaseBuildToolSuite.scala
@@ -40,7 +40,7 @@ abstract class BaseBuildToolSuite extends MopedSuite(LsifJava.app) {
FileLayout.fromString(original, root = workingDirectory)
val targetroot = workingDirectory.resolve("targetroot")
val arguments =
- List("--targetroot", targetroot.toString) ++ extraArguments
+ List("index", "--targetroot", targetroot.toString) ++ extraArguments
val exit = app().run(arguments)
assertEquals(exit, 0, clues(app.capturedOutput))
val semanticdbFiles = FileIO
diff --git a/tests/buildTools/src/test/scala/tests/MissingBuildToolSuite.scala b/tests/buildTools/src/test/scala/tests/MissingBuildToolSuite.scala
index 1e25ab2f..5f4d3322 100644
--- a/tests/buildTools/src/test/scala/tests/MissingBuildToolSuite.scala
+++ b/tests/buildTools/src/test/scala/tests/MissingBuildToolSuite.scala
@@ -3,7 +3,7 @@ package tests
class MissingBuildToolSuite extends BaseBuildToolSuite {
checkErrorOutput(
"basic",
- List(),
+ List("index"),
expectedOutput =
"""|error: No build tool detected in workspace '/workingDirectory'. At the moment, the only supported build tools are: Gradle, Maven.
|""".stripMargin,
@@ -12,7 +12,7 @@ class MissingBuildToolSuite extends BaseBuildToolSuite {
checkErrorOutput(
"ambiguous",
- List(),
+ List("index"),
expectedOutput =
"""|error: Multiple build tools detected: Gradle, Maven. To fix this problem, use the '--build-tools=BUILD_TOOL_NAME' flag to specify which build tool to run.
|""".stripMargin,
@@ -26,7 +26,7 @@ class MissingBuildToolSuite extends BaseBuildToolSuite {
checkErrorOutput(
"no-matching-explicit",
- List("--build-tool", "Gradle"),
+ List("index", "--build-tool", "Gradle"),
expectedOutput =
"""|error: Automatically detected the build tool(s) Maven but none of them match the explicitly provided flag '--build-tool=Gradle'. To fix this problem, run again with the --build-tool flag set to one of the detected build tools.
|""".stripMargin,
@@ -38,7 +38,7 @@ class MissingBuildToolSuite extends BaseBuildToolSuite {
checkErrorOutput(
"levenshtein",
- List("--build-tool", "Mave"),
+ List("index", "--build-tool", "Mave"),
expectedOutput =
"""|error: Automatically detected the build tool(s) Maven but none of them match the explicitly provided flag '--build-tool=Mave'. Did you mean --build-tool=Maven?
|""".stripMargin,
diff --git a/website/.gitignore b/website/.gitignore
new file mode 100644
index 00000000..4ddfdc49
--- /dev/null
+++ b/website/.gitignore
@@ -0,0 +1 @@
+blog/
diff --git a/website/core/Footer.js b/website/core/Footer.js
new file mode 100644
index 00000000..0e46e90e
--- /dev/null
+++ b/website/core/Footer.js
@@ -0,0 +1,55 @@
+const React = require("react");
+
+const siteConfig = require(process.cwd() + "/siteConfig.js");
+
+class Footer extends React.Component {
+ render() {
+ const {
+ baseUrl,
+ copyright,
+ colors: { secondaryColor }
+ } = this.props.config;
+ const docsUrl = `${baseUrl}docs/`;
+ return (
+
+ );
+ }
+}
+
+module.exports = Footer;
diff --git a/website/package.json b/website/package.json
new file mode 100644
index 00000000..44915009
--- /dev/null
+++ b/website/package.json
@@ -0,0 +1,14 @@
+{
+ "scripts": {
+ "examples": "docusaurus-examples",
+ "start": "docusaurus-start",
+ "build": "docusaurus-build",
+ "publish-gh-pages": "docusaurus-publish",
+ "write-translations": "docusaurus-write-translations",
+ "version": "docusaurus-version",
+ "rename-version": "docusaurus-rename-version"
+ },
+ "devDependencies": {
+ "docusaurus": "1.14.4"
+ }
+}
diff --git a/website/pages/en/index.js b/website/pages/en/index.js
new file mode 100755
index 00000000..d69f63a6
--- /dev/null
+++ b/website/pages/en/index.js
@@ -0,0 +1,142 @@
+const React = require("react");
+
+const CompLibrary = require("../../core/CompLibrary.js");
+const Container = CompLibrary.Container;
+const GridBlock = CompLibrary.GridBlock;
+
+const siteConfig = require(process.cwd() + "/siteConfig.js");
+
+function imgUrl(img) {
+ return siteConfig.baseUrl + "img/" + img;
+}
+
+function docUrl(doc, language) {
+ return siteConfig.baseUrl + "docs/" + (language ? language + "/" : "") + doc;
+}
+
+class Button extends React.Component {
+ render() {
+ return (
+
+
+ );
+ }
+}
+
+const Block = props => (
+
+
+
+);
+
+const Features = props => {
+ const features = [
+ {
+ title: "Multiple language versions",
+ content: "Works with Java 8, Java 11 and Java 15. The latest bleeding edge Java version probably works as well.",
+ image:
+ "https://user-images.githubusercontent.com/1408093/109818700-93378e00-7c33-11eb-8385-7ea88533cb87.png",
+ imageAlign: "left"
+ },
+ {
+ title: "Multiple build tools",
+ content:
+ "The build tools Gradle and Maven are supported. We plan to add support for more build tools in the future.",
+ image:
+ "https://user-images.githubusercontent.com/1408093/109819002-e4478200-7c33-11eb-99d3-c42d84acda47.png",
+ imageAlign: "right"
+ },
+ {
+ title: "Simple installation",
+ content:
+ "Run the `lsif-java` command-line tool at the root of your Gradle/Maven build to generate an LSIF index. No manual configuration needed.",
+ image:
+ "https://user-images.githubusercontent.com/1408093/109820615-769c5580-7c35-11eb-9063-98e932c5d6c5.png",
+ imageAlign: "left"
+ },
+ {
+ title: "Low overhead",
+ content:
+ "Indexing a repository should be roughly as fast as compiling all of the sources in the repository.",
+ image:
+ "https://user-images.githubusercontent.com/1408093/109820240-1dccbd00-7c35-11eb-8a87-6b1850687e28.png",
+ imageAlign: "right"
+ }
+ ];
+ return (
+