v2.1.0

Summary

• Lifts, from libuast library, format types:
  • BblfshClient.UastBinary: our custom binary format to store a tree. We provide the guarantee that encode ∘ decode in this format is the identity.
  • BblfshClient.UastYaml: directly readable from standard output. We do not provide the gurantee that encode ∘ decode in this format is the identity. Should only be used for debugging purposes.
• Lifts, from libuast library, tree orders (for iteration): BblfshClient.AnyOrder, BblfshClient.PreOrder, BblfshClient.PostOrder, BblfshClient.LevelOrder, BblfshClient.ChildrenOrder and BblfshClient.PositionOrder. AnyOrder only provides the guarantee that all nodes will be iterated.
• Adds more tests for iterator and encode ∘ decode.
• Solves sbt dying from incorrect bytes passed to decode method #130.
• Adds 60 seconds default timeout to get a (U)AST and makes timeout parameter work in BblfshClient corresponding parsing methods: parseWithOptions and parseWithTimeout.
• Removes terminal CLI org.bblfsh.client.cli.ScalaClientCLI. This client did not work with an XPath query from several releases ago. If you need a CLI, check out our go-client.
• Improves OSX support (macOS 10.7 onward should work correctly) and adds Windows support. Note since bblfshd does not include support for Windows, you could use this client on a Windows, but bblfshd should be mounted on a virtual machine or be accesible through a URL.
• Adds cache for methods and classes in JNI layer. This should slightly speed up native part of the library.
• Adds filter flavours based on return value: BblfshClient.filterBool, BblfshClient.filterString, BblfshClient.filterInt, BblfshClient.filterUint and BblfshClient.filterFloat. These methods work similarly to filter, but they only output nodes of Bool, String, etc type (respectively).
• Bumps sdk and libuast versions.
• Reduces size of the library: native shared libraries stripped from symbols.
• Cleans compilation of JNI from warnings: this could produce an overflow in older versions due to accumulation of local references if the file being parsed was big.
• Compilation of the project locally now has to be done with build.sh script.

Examples

UastFormat and TreeOrder

import scala.io.Source
import org.bblfsh.client.v2.BblfshClient, BblfshClient._
import org.bblfsh.client.v2.Context, Context._
import gopkg.in.bblfsh.sdk.v2.protocol.driver.Mode

val client = BblfshClient("localhost", 9432)

val filename = "/path/to/file.py" // client responsible for encoding it to utf-8
val fileContent = Source.fromFile(filename).getLines.mkString("\n")
val resp = client.parse(filename, fileContent, Mode.SEMANTIC)
val root = resp.get()
val ctx = Context()

// Get encoding in binary format
val buffer = ctx.encode(root, fmt = UastBinary)
// Old API passing an integer also works
val buffer = ctx.encode(root, fmt = 0)
// Binary by default
val buffer = ctx.encode(root)

// Yaml format
val buffer = ctx.encode(root, fmt = UastYaml)
// Same result: yaml format
val buffer = ctx.encode(fmt = 1)

// Iterator with AnyOrder traversing
val it = iterator(root, AnyOrder)
// Also uses AnyOrder
val it = iterator(root, 0)

Timeout to get the (U)AST

import scala.io.Source
import org.bblfsh.client.v2.BblfshClient, BblfshClient._
import gopkg.in.bblfsh.sdk.v2.protocol.driver.Mode

val client = BblfshClient("localhost", 9432)

val filename = "/path/to/file.py" // client responsible for encoding it to utf-8
val fileContent = Source.fromFile(filename).getLines.mkString("\n")
// 30 seconds timeout to get the UAST
val resp = client.parseWithOptions(filename, fileContent, "python", 30, Mode.SEMANTIC)
// Same as
val resp = client.parseWithOptions(filename, fileContent, lang = "", 30, Mode.SEMANTIC)
// and
val resp = client.parseWithTimeout(filename, fileContent, 30)
// Get annotated AST with 30 seconds timeout to get a response
val resp = client.parseWithOptions(filename, fileContent, "python", 30, Mode.ANNOTATED)
val ctx = resp.get()

// Get default timeout (in seconds)
println(client.DEFAULT_TIMEOUT_SEC)

Compilation of the project locally

chmod +x ./build.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
./build.sh --clean --get-dependencies --all

After that, the following command can be used to execute the interpreter in interactive mode:

./sbt console

v2.0.2

Summary

• Solves several memory leaks and SIGSEVs in the client.
• Bumps libuast version. Changes include new supported orders for traversing a tree: AnyOrder (we do not give assurances on this order, only that it traverses all the nodes) and ChildrenOrder (to traverse only the immediate children for a node from left to right) and solve memory leaks.
• Adds plugins to ease development: sbt-git to take the version to release automatically from the git history and sbt-jni to ease development with JNI.
• v2.0.1 did not make it to Maven due to difficulties with build process for macOs and this release supersedes it (#123). This version includes the fix for a linking error with CGO libraries (#119)

Fixed memory leaks

• Leaks included in former versions of libuast.
• Improves objects management in the native code, taking better care of global references (some of them were not deleted) and improving the caching strategy which could generate memory leaks (#113, #116).
• ContextExt did not free native memory.
• BblfshClient.decode method used a lot of native memory (due to a nio direct buffer used inside, which uses normal heap space) which was not freed until JVM heap was full and GC (Garbage Collector) was triggered.
• If we used a ctx : ContextExt and got a NodeExt or UastIterExt from it and the GC ctx, we could provoke SIGSEVs.

Known issues

• There are some linking problems with older macOS versions (probably 10.11 backwards) due to the need to adjust backward compatibility when compiling libuast (libuast#111)
• As noted in previous versions, pending improvements for v2.1.0.

v2.0.1

Summary

Fixes a linking issue when the library is used with others that depend on a Go runtime (see #119 for context). Also updates examples from the README, and adds some useful syntactic sugar to get this client closer to the syntax of python-client.

New syntactic sugar

We can now call filter and iterator over a client: BblfshClient instance:

import scala.io.Source
import org.bblfsh.client.v2.BblfshClient, BblfshClient._
import gopkg.in.bblfsh.sdk.v2.protocol.driver.Mode

val client = BblfshClient("localhost", 9432)

val filename = "/path/to/file.py" // client responsible for encoding it to utf-8
val fileContent = Source.fromFile(filename).getLines.mkString("\n")
val resp = client.parse(filename, fileContent, Mode.SEMANTIC)

val itFilter = client.filter(resp.get, "//uast:Identifier")
val it = client.iterator(resp.get, PreOrder)

Known issues

• Very unusual SIGSEV we are still investigating the source of.
• Potential memory leak #113, #116 (now moved to v2.0.2 milestone).
• Several minor improvements under v2.1.0 scope (includes iterators for primitive types, customizable gRCP timeouts, new iteration orders and output formats).

v2.0.0

First major release supporting UASTv2 and bblfhsh protocol.v2.

Summary

The biggest change is that protocol.v2 only expose a binary UAST representation in the wire format, as part of the regular gRPC response.

Decoding UAST from that binary format is implemented in libuast in Go and means building in-memory object representation of UAST on the Go side.

Loading UAST from Go side to the JVM memory is done with libuast though CGo with is exposed to JVM by JNI.

API changes

In all examples below this preamble is left out for brievety

V1

import org.bblfsh.client._
val client = BblfshClient("localhost", 9432)
val resp = client.parse(fileName, fileContent)

V2

import org.bblfsh.client.v2._ 
import gopkg.in.bblfsh.sdk.v2.protocol.driver.Mode
val client = BblfshClient("localhost", 9432)
val resp = client.parse(fileName, fileContent, Mode.SEMANTIC)

Supported languages

Before

val resp = client.supportedLanguages()
val supportedLangs = resp.languages.map(_.language)

After

val resp = client.supportedLanguages()
val supportedLangs = resp.languages.flatMap(x => Seq(x.language) ++ x.aliases)

Parsing + decoding + loading UAST object in JVM

Before

val node: Node = resp.get

After

import BblfshClient._ // enables helpers
val node: JNode = resp.get

Parsing + get the binary

Before

resp.get.toByteArray

After

resp.uast

Decoding UAST from the binary + loading to JVM memory

Before

import gopkg.in.bblfsh.sdk.v1.uast.generated.Node

var bytes: Array[Byte]
val node: Node = Node.parseFrom(bytes)

After

var bytes: Array[Byte]
val node: JNode = JNode.parseFrom(bytes)

Encode UAST to a binary

val node: Node
val bytes: Array[Byte] = node.toByteArray

val node: JNode
val bytes: Array[Byte] = node.toByteArray // .toByteBuffer is also available

Iterators

The same as before:

    val it = BblfshClient.iterator(rootNode, BblfshClient.PreOrder)
    it.close()

But as iterator consumes native resources and need to be closed at the end.

XPath queries

Before

val list: List[Node] = BblfshClient.filter(rootNode, "//QualifiedName[@roleExpression]")

After

    val it = BblfshClient.filter(rootNode, "//uast:QualifiedIdentifer")
    val list = it.toList
    it.close()

For both JNode and NodeExt input nodes filter now returns Iterator.

Known issues

• potential memory leak #113 (WIP, planned for v2.0.1)
• several minor improvements under v2.1.0 scope
  (includes iterators for primitive types, customizable gRCP timeouts, new iteration orders and output formats)

v2.0.0-SNAPSHOT RC3

Third SNAPSHOT release of work-in-progress for the UASTv2 migration that is tracked under #83.

It has the same scope and API as RC2 and is a hotfix release.

Bugs fixed

• encoding bug, where Uint is sometimes mistakenly encoded as Bool #104, #103

Known issues

• UAST node iterators and XPath queries are not supported yet (WIP under #83)

v2.0.0-SNAPSHOT RC2

Second SNAPSHOT release of work-in-progress for the UASTv2 migration that is tracked under #83.

It iterates on RC1, has the same scope but also includes a high-level API for a new UASTv2 protocol&wire format. We'll be happy to add a convenience higher-level API, covering common use-cases as soon as possible.

Summary

The biggest change is that protocol.v2 only expose a binary UAST representation in the wire format, as part of the regular gRPC response.

Decoding UAST from that binary format is implemented in libuast in Go and means building in-memory object representation of UAST on the Go side.

Loading UAST from Go side to the JVM memory is done with libuast though CGo with is exposed to JVM by JNI.

API changes

In all examples below this preamble is left out for brievety

V1

import org.bblfsh.client._
val client = BblfshClient("localhost", 9432)
val resp = client.parse(fileName, fileContent)

V2

import org.bblfsh.client.v2._ 
val client = BblfshClient("localhost", 9432)
val resp = client.parse(fileName, fileContent)

Supported languages

Before

val resp = client.supportedLanguages()
val supportedLangs = resp.languages.map(_.language)

After

val resp = client.supportedLanguages()
val supportedLangs = resp.languages.flatMap(x => Seq(x.language) ++ x.aliases)

Parsing + decoding + loading UAST object in JVM

Before

val node: Node = resp.get

After

import BblfshClient._ // enables helpers
val node: JNode = resp.get

Parsing + get the binary

Before

resp.get.toByteArray

After

resp.uast

Decoding UAST from the binary + loading to JVM memory

Before

import gopkg.in.bblfsh.sdk.v1.uast.generated.Node

var bytes: Array[Byte]
val node: Node = Node.parseFrom(bytes)

After

var bytes: Array[Byte]
val node: JNode = JNode.parseFrom(bytes)

Encode UAST to a binary

val node: Node
val bytes: Array[Byte] = node.toByteArray

val node: JNode
val bytes: Array[Byte] = node.toByteArray // .toByteBuffer is also available

Known issues

• UAST node iterators and XPath queries are not supported yet (WIP under #83)
• UAST encoding bug: Uint encoded as Bool is #104

v2.0.0-SNAPSHOT RC1

A SNAPSHOT release of work-in-progress for the UASTv2 migration that is tracked under #83.

This is a preview, showcasing the new, subject of change, low-level API for a new UASTv2 protocol&wire format. We'll be happy to add a convenience higher-level API, covering common use-cases as soon as possible.

Summary

The biggest change is that protocol.v2 only expose a binary UAST representation in the wire format, as part of the regular gRPC response.

Decoding UAST from that binary format is implemented in libuast in Go and means building in-memory object representation of UAST on the Go side.

Loading UAST from Go side to the JVM memory is done with libuast though CGo with is exposed to JVM by JNI.

API changes

In all examples below this preamble is left out for brievety

V1

import org.bblfsh.client._
val client = BblfshClient("localhost", 9432)
val resp = client.parse(fileName, fileContent)

V2

import org.bblfsh.client.v2._ 
val client = BblfshClient("localhost", 9432)
val resp = client.parse(fileName, fileContent)

Supported languages

Before

val resp = client.supportedLanguages()
val supportedLangs = resp.languages.map(_.language)

After

val resp = client.supportedLanguages()
val supportedLangs = resp.languages.flatMap(x => Seq(x.language) ++ x.aliases)

Parsing + UAST object in JVM

Before

val node: Node = resp.get

After

import BblfshClient._ // enables uast.* methods
val node: JNode = resp.uast.decode.root.load()

Parsing + get the binary

Before

resp.get.toByteArray

After

resp.uast

Decoding UAST from the binary => JVM memory

Before

import gopkg.in.bblfsh.sdk.v1.uast.generated.Node

var bytes: Array[Byte]
val node: Node = Node.parseFrom(bytes)

After

var bytes: ByteBuffer // in Direct mode, to avoid copy
val node: JNode = BblfshClient.decode(bytes).root().load()

Encode UAST to a binary

val node: Node
val bytes: Array[Byte] = node.toByteArray

val node: JNode

val ctx = Context()
val bytes: ByteBuffer = ctx.encode(rootTree)
ctx.dispose()

v1.11

New Features

• Expose bblfshd timeout parameter at Parse API:
  Add new BblfshClient.parseWithTimeout()

• ScalaBP used for gRPC and Protobuf is updated from 0.7.1 to latest 0.8.3

Known issues

• #77 a rare NPE in the native JNI libscalaclient

v1.10.1

• Maven fix.

v1.10.0

• Fix macOS build.
• Add close() method.

Thanks @smacker!