Babelfish driver SDK
Branch: master
Clone or download
dennwc transformer: add Drop operation to Fields (#369)
* transformer: add Drop operation to Fields

Signed-off-by: Denys Smirnov <denys@sourced.tech>
Latest commit 5a1a268 Feb 21, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets driver: update versions of sdk and bblfshd in the skeleton Feb 14, 2019
build build: add comments to test/benchmark code Dec 3, 2018
cmd address review feedback Feb 18, 2019
driver address review feedback Feb 18, 2019
etc driver: update versions of sdk and bblfshd in the skeleton Feb 14, 2019
internal/docker docker-based integration tests Jul 19, 2018
protocol Improve some role descriptions, update bindata. Feb 20, 2019
uast transformer: add Drop operation to Fields (#369) Feb 21, 2019
.gitignore Initial commit Feb 21, 2017
.travis.yml update ci config Dec 3, 2018
DCO
Gopkg.lock enable opentracing in drivers and grpc Nov 27, 2018
Gopkg.toml switch back to master branch of xpath library Sep 19, 2018
LICENSE Initial commit Feb 21, 2017
MAINTAINERS
Makefile move common main function from template to driver package Feb 7, 2018
README.md sdk test: add documentation Feb 18, 2019
common.go refactoring: moving to CLI helper Nov 22, 2018
doc.go change sdk version in import paths May 3, 2018

README.md

sdk Build Status codecov license GitHub release

Babelfish SDK contains the tools and libraries required to create a Babelfish driver for a programming language.

Build

Dependencies

The Babelfish SDK has the following dependencies:

Make sure that you've correctly set your GOROOT and GOPATH environment variables.

Install

Babelfish SDK gets installed using either Go:

$ go get -t -v gopkg.in/bblfsh/sdk.v2/...

or make command:

$ make install

These commands will install both bblfsh-sdk and bblfsh-sdk-tools programs at $GOPATH/bin/.

Contribute

The SDK provides scaffolding templates for creating a new language driver. These templates are converted to Go code that ends up in bblfsh-sdk tool. Use make to update these templates:

$ make
go get -v github.com/jteeuwen/go-bindata/...
go get -v golang.org/x/tools/cmd/cover/...
cat protocol/internal/testdriver/main.go | sed -e 's|\([[:space:]]\+\).*//REPLACE:\(.*\)|\1\2|g' \
	> etc/skeleton/driver/main.go.tpl
chmod -R go=r ${GOPATH}/src/github.com/bblfsh/sdk/etc/build; \
go-bindata \
	-pkg build \
	-modtime 1 \
	-nocompress \
	-prefix ${GOPATH}/src/github.com/bblfsh/sdk/etc/build \
	-o assets/build/bindata.go \
	${GOPATH}/src/github.com/bblfsh/sdk/etc/build/...
chmod -R go=r ${GOPATH}/src/github.com/bblfsh/sdk/etc/skeleton; \
go-bindata \
	-pkg skeleton \
	-modtime 1 \
	-nocompress \
	-prefix ${GOPATH}/src/github.com/bblfsh/sdk/etc/skeleton \
	-o assets/skeleton/bindata.go \
	${GOPATH}/src/github.com/bblfsh/sdk/etc/skeleton/...

You can validate this process has been properly done before submitting changes:

$ make validate-commit

If the code has not been properly generated, this command will show a diff of the changes that have not been processed and will end up with a message like:

generated bindata is out of sync
make: *** [Makefile:66: validate-commit] Error 2

Review the process if this happens.

On the other hand, If you need to regenerate proto and proteus files, you must run go generate from protocol/ directory:

$ cd protocol/
$ go generate

It regenerates all proto and proteus files under protocol/ and uast/ directories.

Usage

Babelfish SDK helps both setting up the initial structure of a new driver and keeping that structure up to date.

Creating the driver's initial structure

Let's say we're creating a driver for mylang. The first step is going to the location where we want the repository for the driver to be bootstrapped:

$ cd $GOPATH/src/github.com/bblfsh

Now the driver should be bootstrapped with bblfsh-sdk. This will create a git repository, and some directories and files required by every driver. They will be overwritten if they exist, like the README.md file in the example below.

$ bblfsh-sdk init mylang alpine
initializing driver "mylang", creating new manifest
creating file "manifest.toml"
creating file "Makefile"
creating file "driver/main.go"
creating file "driver/normalizer/normalizer.go"
creating file ".git/hooks/pre-commit"
creating file ".gitignore"
creating file ".travis.yml"
creating file "Dockerfile.build.tpl"
creating file "driver/normalizer/normalizer_test.go"
creating file "Dockerfile.tpl"
creating file "LICENSE"
managed file "README.md" has changed, discarding changes
$ git add -A
$ git commit -m 'initialize repository'

Note that this adds a pre-commit git hook, which will verify these files are up to date before every commit and will disallow commits if some of the managed files are changed. You can by-pass this with git commit --no-verify.

You can find the driver skeleton used here at etc/skeleton.

Keeping managed files updated

Whenever the managed files are updated, drivers need to update them. bblfsh-sdk can be used to perform some of this updates in managed files. For example, if the README template is updated, running bblfsh-sdk update will overwrite it.

$ bblfsh-sdk update
managed file "README.md" has changed, discarding changes

bblfsh-sdk doesn't update the SDK itself.

For further details of how to construct a language driver, take a look at Implementing the driver section in documentation.

Testing the driver

bbflsh-sdk also includes a testing framework for a driver. In order to run test for a particular dirver, change to it's directory and run:

$ bblfsh-sdk test

This will:

  • compile a "test binary" that parses content of the ./fixtures directory of the driver
  • create a docker image with all dependencies, native driver and a test binary
  • run this test binary inside a Docker container, using that image

Overall, SDK supports 3 different kind of tests for a driver:

  • UnitTests, parsing content of ./fixtures and applying UAST transformations. Described above.
  • Integration tests, using content of ./fixtures/_integration*
  • Benchmarks, using content of ./fixtures/bench_*

First two always run, benchmarks are only triggered by bblfsh-sdk test --bench.

License

GPLv3, see LICENSE