Thrift improved
Switch branches/tags
Clone or download
brianshannan-wf Merge pull request #1036 from Workiva/release_2_23_0
INFENG-6308 INFENG-6309 RELEASE 2.23.0
Latest commit 7a28bcc Oct 11, 2018
Permalink
Failed to load latest commit information.
.github add glide for examples Apr 18, 2018
Godeps rename _workspace to vendor Nov 8, 2017
compiler release 2.23.0 Oct 10, 2018
contrib Add frame parser script utility Aug 18, 2016
documentation docs: doctoc and extra spaces Dec 28, 2016
examples release 2.23.0 Oct 10, 2018
lib release 2.23.0 Oct 10, 2018
scripts Merge pull request #1024 from Workiva/python_nats_version_testing2 Aug 6, 2018
test release 2.23.0 Oct 10, 2018
vendor Add glide.yaml and glide.lock for CLI; update vendored deps Mar 12, 2018
.gitignore remove godep and vendored dependencies from go library Mar 12, 2018
.travis.yml Don't install virtualenv wrapper or Dart, as they aren't used. Apr 15, 2018
CONTRIBUTING.md Update contributing information Aug 17, 2017
Dockerfile INFENG-6094 publish build artifacts Aug 2, 2018
LICENSE license Jul 3, 2017
MAINTAINERS.md Update maintainers, add contributing Aug 16, 2016
Makefile Merge pull request #1 from charliestrawn/top-level-makefile Apr 24, 2018
README.md Java: Support use_vendor Jul 9, 2018
aviary.yaml RED-1848: update aviary.yaml with proper excludes Sep 6, 2017
codecov.yml Rename codecov.yaml back to codecov.yml Nov 28, 2016
glide.lock Add glide.yaml and glide.lock for CLI; update vendored deps Mar 12, 2018
glide.yaml Add glide.yaml and glide.lock for CLI; update vendored deps Mar 12, 2018
main.go Compiler: make -use-vendor a generator option for Go Jan 31, 2017
skynet.yaml upgrade versions of python nats libraries Aug 2, 2018
smithy.yaml upgrade versions of python nats libraries Aug 2, 2018

README.md

Frugal

Build Status

Frugal is an extension of Apache Thrift which provides additional functionality. Key features include:

  • request headers
  • request multiplexing
  • request interceptors
  • per-request timeouts
  • thread-safe clients
  • code-generated pub/sub APIs
  • support for Go, Java, Dart, and Python (2.7 and 3.5)

Frugal is intended to act as a superset of Thrift, meaning it implements the same functionality as Thrift with some additional features. For a more detailed explanation, see the documentation.

Installation

Homebrew

brew install frugal

Download

Pre-compiled binaries for OS X and Linux are available from the Github releases tab. Currently, adding these binaries is a manual process. If a downloadable release is missing, notify the messaging team to have it added.

If go is already installed and setup you can also simply:

$ go get github.com/Workiva/frugal

From Source

Our usage of godep has been deprecated as we move to glide. Once the deprecation period is over, we will remove both the Godeps/ and vendor/ folder, relying solely on glide for dependency management

  1. Install go and setup GOPATH.

  2. Clone the frugal repo

    $ mkdir -p $GOPATH/src/github.com/Workiva && cd $_
    $ git clone git@github.com:Workiva/frugal.git
  3. Install the CLI binary

    $ cd $GOPATH/src/github.com/Workiva/frugal
    $ curl https://glide.sh/get | sh  # get glide if necessary
    $ glide install  # get dependencies
    $ go install

When generating go, be aware the frugal go library and the frugal compiler have separate dependencies.

Usage

Define your Frugal file which contains your pub/sub interface, or scopes, and Thrift definitions.

# event.frugal

// Anything allowed in a .thrift file is allowed in a .frugal file.
struct Event {
    1: i64 ID,
    2: string Message
}

// Scopes are a Frugal extension for pub/sub APIs.
scope Events {
    EventCreated: Event
}

Generate the code with frugal. Currently, only Go, Java, Dart, and Python are supported.

$ frugal -gen=go event.frugal

By default, generated code is placed in a gen-* directory. This code can then be used as such:

// publisher.go
func main() {
    conn, err := nats.Connect(nats.DefaultURL)
    if err != nil {
        panic(err)
    }

    var (
        protocolFactory  = frugal.NewFProtocolFactory(thrift.NewTBinaryProtocolFactoryDefault())
        transportFactory = frugal.NewFNatsScopeTransportFactory(conn)
        provider         = frugal.NewFScopeProvider(transportFactory, protocolFactory)
        publisher        = event.NewEventsPublisher(provider)
    )
    publisher.Open()
    defer publisher.Close()

    event := &event.Event{ID: 42, Message: "Hello, World!"}
    if err := publisher.PublishEventCreated(frugal.NewFContext(""), event); err != nil {
        panic(err)
    }
}
// subscriber.go
func main() {
    conn, err := nats.Connect(nats.DefaultURL)
    if err != nil {
        panic(err)
    }

    var (
        protocolFactory  = frugal.NewFProtocolFactory(thrift.NewTBinaryProtocolFactoryDefault())
        transportFactory = frugal.NewFNatsScopeTransportFactory(conn)
        provider         = frugal.NewFScopeProvider(transportFactory, protocolFactory)
        subscriber       = event.NewEventsSubscriber(provider)
    )

    _, err = subscriber.SubscribeEventCreated(func(ctx *frugal.FContext, e *event.Event) {
        fmt.Println("Received event:", e.Message)
    })
    if err != nil {
        panic(err)
    }

    wait := make(chan bool)
    log.Println("Subscriber started...")
    <-wait
}

Prefixes

By default, Frugal publishes messages on the topic <scope>.<operation>. For example, the EventCreated operation in the following Frugal definition would be published on Events.EventCreated:

scope Events {
    EventCreated: Event
}

Custom topic prefixes can be defined on a per-scope basis:

scope Events prefix foo.bar {
    EventCreated: Event
}

As a result, EventCreated would be published on foo.bar.Events.EventCreated.

Prefixes can also define variables which are provided at publish and subscribe time:

scope Events prefix foo.{user} {
    EventCreated: Event
}

This variable is then passed to publish and subscribe calls:

var (
    event = &event.Event{ID: 42, Message: "hello, world!"}
    user  = "bill"
)
publisher.PublishEventCreated(frugal.NewFContext(""), event, user)

subscriber.SubscribeEventCreated(user, func(ctx *frugal.FContext, e *event.Event) {
    fmt.Printf("Received event for %s: %s\n", user, e.Message)
})

Generated Comments

In Thrift, comments of the form /** ... */ are included in generated code. In Frugal, to include comments in generated code, they should be of the form /**@ ... */.

/**@
 * This comment is included in the generated code because
 * it has the @ sign.
 */
struct Foo {}

/**@ This comment is included too. */
service FooService {
    /** This comment isn't included because it doesn't have the @ sign. */
    Foo getFoo()
}

Annotations

Annotations are extra directive in the IDL that can alter the way code is generated. Some common annotations are listed below

Annotation Values Allowed Places Description
vendor Optional location Namespaces, Includes See vendoring includes
deprecated Optional description Service methods, Struct/union/exception fields Marks a method or field as deprecated (if supported by the language, or in a comment otherwise), and logs a warning if a deprecated method is called.

Vendoring Includes

Frugal does not generate code for includes by default. The -r flag is required to recursively generate includes. If -r is set, Frugal generates the entire IDL tree, including code for includes, in the same output directory (as specified by -out) by default. Since this can cause problems when using a library that uses a Frugal-generated object generated with the same IDL in two or more places, Frugal provides special support for vendoring dependencies through a vendor annotation on includes and namespaces.

The vendor annotation is used on namespace definitions to indicate to any consumers of the IDL where the generated code is vendored so that consumers can generate code that points to it. This cannot be used with * namespaces since it is language-dependent. Consumers then use the vendor annotation on includes they wish to vendor. The value provided on the include-side vendor annotation, if any, is ignored.

When an include is annotated with vendor, Frugal will skip generating the include if use_vendor language option is set since this flag indicates intention to use the vendored code as advertised by the vendor annotation.

If no location is specified by the vendor annotation, the behavior is defined by the language generator.

The vendor annotation is currently only supported by Go, Dart and Java.

The example below illustrates how this works.

bar.frugal ("providing" IDL):

namespace go bar (vendor="github.com/Workiva/my-repo/gen-go/bar")
namespace dart bar (vendor="my-repo/gen-go")
namespace java bar (vendor="com.workiva.bar.custom.pkg")

struct Struct {}

foo.frugal ("consuming" IDL):

include "bar.frugal" (vendor)

service MyService {
    bar.Struct getStruct()
}
frugal -r -gen go:package_prefix=github.com/Workiva/my-other-repo/gen-go,use_vendor foo.frugal

When we run the above command to generate foo.frugal, Frugal will not generate code for bar.frugal since use_vendor is set and the "providing" IDL has a vendor path set for the Go namespace. Instead, the generated code for foo.frugal will reference the vendor path specified in bar.frugal (github.com/Workiva/my-repo/gen-go/bar).

Thrift Parity

Frugal is intended to be a superset of Thrift, meaning valid Thrift should be valid Frugal. File an issue if you discover an inconsistency in compatibility with the IDL.