-
Notifications
You must be signed in to change notification settings - Fork 225
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
46 additions
and
217 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,252 +1,81 @@ | ||
| # Gizmo Microservice Toolkit [](https://godoc.org/github.com/NYTimes/gizmo) [](https://travis-ci.org/NYTimes/gizmo) [](https://coveralls.io/github/NYTimes/gizmo?branch=master) | ||
|
|
||
|  | ||
| <p align="center"> | ||
| <img src="http://graphics8.nytimes.com/images/blogs/open/2015/gizmo.png"/> | ||
| </p> | ||
|
|
||
| This toolkit provides packages to put together server and pubsub daemons with the following features: | ||
|
|
||
| * standardized configuration and logging | ||
| * health check endpoints with configurable strategies | ||
| * configuration for managing pprof endpoints and log levels | ||
| * structured logging containing basic request information | ||
| * useful metrics for endpoints | ||
| * graceful shutdowns | ||
| * basic interfaces to define our expectations and vocabulary | ||
| * Standardized configuration and logging | ||
| * Health check endpoints with configurable strategies | ||
| * Configuration for managing pprof endpoints and log levels | ||
| * Basic interfaces to define expectations and vocabulary | ||
| * Structured logging containing basic request information | ||
| * Useful metrics for endpoints | ||
| * Graceful shutdowns | ||
|
|
||
| In this toolkit, you will find: | ||
| ### Packages | ||
|
|
||
| ## The `config` packages | ||
| #### [`server`](https://godoc.org/github.com/NYTimes/gizmo/server) | ||
|
|
||
| The `config` package contains a handful of useful functions to load to configuration structs from JSON files, JSON blobs in Consul k/v or environment variables. | ||
| The `server` package is the bulk of the toolkit and relies on `server.Config` to manage `Server` implementations. | ||
|
|
||
| The subpackages contain structs meant for managing common configuration options and credentials. There are currently configs for: | ||
| It offers 2 server implementations: | ||
|
|
||
| * Go Kit Metrics | ||
| * MySQL | ||
| * MongoDB | ||
| * Oracle | ||
| * PostgreSQL | ||
| * AWS (S3, DynamoDB, ElastiCache) | ||
| * GCP | ||
| * Gorilla's `securecookie` | ||
| 1. [`SimpleServer`](https://godoc.org/github.com/NYTimes/gizmo/server#SimpleServer), which is capable of handling basic HTTP and JSON requests via 5 of the available `Service` implementations: `SimpleService`, `JSONService`, `ContextService`, `MixedService` and a `MixedContextService`. | ||
|
|
||
| The package also has a generic `Config` type in the `config/combined` subpackage that contains all of the above types. It's meant to be a 'catch all' convenience struct that many applications should be able to use. | ||
| 2. [`RPCServer`](https://godoc.org/github.com/NYTimes/gizmo/server#RPCServer), which is capable of serving a gRPC server on one port and JSON endpoints on another. This kind of server can only handle the `RPCService` implementation. | ||
|
|
||
| ## The `server` package | ||
|
|
||
| This package is the bulk of the toolkit and relies on `server.Config` for any managing `Server` implementations. A server must implement the following interface: | ||
| #### [`server/kit`](https://godoc.org/github.com/NYTimes/gizmo/server/kit) | ||
|
|
||
| ```go | ||
| // Server is the basic interface that defines what expect from any server. | ||
| type Server interface { | ||
| Register(Service) error | ||
| Start() error | ||
| Stop() error | ||
| } | ||
| ``` | ||
| This is an experimental package in Gizmo! | ||
|
|
||
| The package offers 2 server implementations: | ||
| * The rationale behind this package: | ||
| * A more opinionated server with fewer choices | ||
| * go-kit is used for serving HTTP/JSON & gRPC is used for serving HTTP2/RPC | ||
| * Monitoring and metrics are handled by a sidecar (ie. Cloud Endpoints) | ||
| * Logs always go to stdout/stderr | ||
| * Using Go's 1.8 graceful HTTP shutdown | ||
| * Services using this package are meant for deploy to GCP with GKE and Cloud Endpoints. | ||
|
|
||
| `SimpleServer`, which is capable of handling basic HTTP and JSON requests via 5 of the available `Service` implementations: `SimpleService`, `JSONService`, `ContextService`, `MixedService` and a `MixedContextService`. A service and these implementations will be defined below. | ||
|
|
||
| `RPCServer`, which is capable of serving a gRPC server on one port and JSON endpoints on another. This kind of server can only handle the `RPCService` implementation. | ||
| #### [`config`](https://godoc.org/github.com/NYTimes/gizmo/config) | ||
|
|
||
| The `Service` interface is minimal to allow for maximum flexibility: | ||
| ```go | ||
| type Service interface { | ||
| Prefix() string | ||
| The `config` package contains a handful of useful functions to load to configuration structs from JSON files, JSON blobs in Consul k/v, or environment variables. | ||
|
|
||
| // Middleware provides a hook for service-wide middleware. | ||
| Middleware(http.Handler) http.Handler | ||
| } | ||
| ``` | ||
| * There are also many structs for common configuration options and credentials of different Cloud Services and Databases. | ||
| * The package also has a generic `Config` type in the `config/combined` subpackage that contains all of the above types. It's meant to be a 'catch all' convenience struct that many applications should be able to use. | ||
|
|
||
| The 5 service types that are accepted and hostable on the `SimpleServer`: | ||
| #### [`pubsub`](https://godoc.org/github.com/NYTimes/gizmo/pubsub) | ||
|
|
||
| The `pubsub` package contains two (`publisher` and `subscriber`) generic interfaces for publishing data to queues as well as subscribing and consuming data from those queues. | ||
|
|
||
| ```go | ||
| type SimpleService interface { | ||
| Service | ||
| There are 4 implementations of `pubsub` interfaces: | ||
|
|
||
| // router - method - func | ||
| Endpoints() map[string]map[string]http.HandlerFunc | ||
| } | ||
| * For pubsub via Amazon's SNS/SQS, you can use the [`pubsub/aws`](https://godoc.org/github.com/NYTimes/gizmo/pubsub/aws) package | ||
|
|
||
| type JSONService interface { | ||
| Service | ||
| * For pubsub via Google's Pubsub, you can use the [`pubsub/gcp`](https://godoc.org/github.com/NYTimes/gizmo/pubsub/gcp) package | ||
|
|
||
| // Ensure that the route syntax is compatible with the router | ||
| // implementation chosen in cfg.RouterType. | ||
| // route - method - func | ||
| JSONEndpoints() map[string]map[string]JSONEndpoint | ||
| // JSONMiddleware provides a hook for service-wide middleware around JSONEndpoints. | ||
| JSONMiddleware(JSONEndpoint) JSONEndpoint | ||
| } | ||
|
|
||
| type MixedService interface { | ||
| Service | ||
|
|
||
| // route - method - func | ||
| Endpoints() map[string]map[string]http.HandlerFunc | ||
|
|
||
| // Ensure that the route syntax is compatible with the router | ||
| // implementation chosen in cfg.RouterType. | ||
| // route - method - func | ||
| JSONEndpoints() map[string]map[string]JSONEndpoint | ||
| // JSONMiddleware provides a hook for service-wide middleware around JSONEndpoints. | ||
| JSONMiddleware(JSONEndpoint) JSONEndpoint | ||
| } | ||
| * For pubsub via Kafka topics, you can use the [`pubsub/kafka`](https://godoc.org/github.com/NYTimes/gizmo/pubsub/kafka) package | ||
|
|
||
| type ContextService interface { | ||
| Service | ||
| * For publishing via HTTP, you can use the [`pubsub/http`](https://godoc.org/github.com/NYTimes/gizmo/pubsub/http) package | ||
|
|
||
| // route - method - func | ||
| ContextEndpoints() map[string]map[string]ContextHandlerFunc | ||
| // ContextMiddleware provides a hook for service-wide middleware around ContextHandler | ||
| ContextMiddleware(ContextHandler) ContextHandler | ||
| } | ||
|
|
||
| type MixedContextService interface { | ||
| ContextService | ||
|
|
||
| // route - method - func | ||
| JSONEndpoints() map[string]map[string]JSONContextEndpoint | ||
| JSONContextMiddleware(JSONContextEndpoint) JSONContextEndpoint | ||
| } | ||
| ``` | ||
| #### [`pubsub/pubsubtest`](https://godoc.org/github.com/NYTimes/gizmo/pubsub/pubsubtest) | ||
|
|
||
| Where `JSONEndpoint`, `JSONContextEndpoint`, `ContextHandler` and `ContextHandlerFunc` are defined as: | ||
| The `pubsub/pubsubtest` package contains test implementations of the `pubsub.Publisher`, `pubsub.MultiPublisher`, and `pubsub.Subscriber` interfaces that will allow developers to easily mock out and test their `pubsub` implementations. | ||
|
|
||
| ```go | ||
| type JSONEndpoint func(*http.Request) (int, interface{}, error) | ||
| #### [`web`](https://godoc.org/github.com/NYTimes/gizmo/web) | ||
|
|
||
| type JSONContextEndpoint func(context.Context, *http.Request) (int, interface{}, error) | ||
| The `web` package has a handful of very useful functions for parsing types from request queries and payloads. | ||
|
|
||
| type ContextHandler interface { | ||
| ServeHTTPContext(context.Context, http.ResponseWriter, *http.Request) | ||
| } | ||
| #### Examples | ||
|
|
||
| type ContextHandlerFunc func(context.Context, http.ResponseWriter, *http.Request) | ||
| ``` | ||
| * Several reference implementations utilizing `server` and `pubsub` are available in the [`examples`](https://github.com/NYTimes/gizmo/tree/master/examples) subdirectory. | ||
| * There are also examples within the GoDoc: [here](https://godoc.org/github.com/NYTimes/gizmo/examples) | ||
|
|
||
| Also, the one service type that works with an `RPCServer`: | ||
|
|
||
| ```go | ||
| type RPCService interface { | ||
| ContextService | ||
|
|
||
| Service() (grpc.ServiceDesc, interface{}) | ||
|
|
||
| // Ensure that the route syntax is compatible with the router | ||
| // implementation chosen in cfg.RouterType. | ||
| // route - method - func | ||
| JSONEndpoints() map[string]map[string]JSONContextEndpoint | ||
| // JSONMiddleware provides a hook for service-wide middleware around JSONContextEndpoints. | ||
| JSONMiddlware(JSONContextEndpoint) JSONContextEndpoint | ||
| } | ||
| ``` | ||
|
|
||
| The `Middleware(..)` functions offer each service a 'hook' to wrap each of its endpoints. This may be handy for adding additional headers or context to the request. This is also the point where other, third-party middleware could be easily plugged in (i.e. oauth, tracing, metrics, logging, etc.) | ||
|
|
||
| ## The `pubsub` packages | ||
|
|
||
| The base `pubsub` package contains three generic interfaces for publishing data to queues and subscribing and consuming data from those queues. | ||
|
|
||
| ```go | ||
| // Publisher is a generic interface to encapsulate how we want our publishers | ||
| // to behave. Until we find reason to change, we're forcing all publishers | ||
| // to emit protobufs. | ||
| type Publisher interface { | ||
| // Publish will publish a message. | ||
| Publish(ctx context.Context, key string, msg proto.Message) error | ||
| // Publish will publish a []byte message. | ||
| PublishRaw(ctx context.Context, key string, msg []byte) error | ||
| } | ||
|
|
||
| // MultiPublisher is an interface for publishers who support sending multiple | ||
| // messages in a single request, in addition to individual messages. | ||
| type MultiPublisher interface { | ||
| Publisher | ||
|
|
||
| // PublishMulti will publish multiple messages with a context. | ||
| PublishMulti(context.Context, []string, []proto.Message) error | ||
| // PublishMultiRaw will publish multiple raw byte array messages with a context. | ||
| PublishMultiRaw(context.Context, []string, [][]byte) error | ||
| } | ||
|
|
||
| // Subscriber is a generic interface to encapsulate how we want our subscribers | ||
| // to behave. For now the system will auto stop if it encounters any errors. If | ||
| // a user encounters a closed channel, they should check the Err() method to see | ||
| // what happened. | ||
| type Subscriber interface { | ||
| // Start will return a channel of raw messages. | ||
| Start() <-chan SubscriberMessage | ||
| // Err will contain any errors returned from the consumer connection. | ||
| Err() error | ||
| // Stop will initiate a graceful shutdown of the subscriber connection. | ||
| Stop() error | ||
| } | ||
| ``` | ||
|
|
||
| Where a `SubscriberMessage` is an interface that gives implementations a hook for acknowledging/delete messages. Take a look at the docs for each implementation in `pubsub` to see how they behave. | ||
|
|
||
| There are currently major 4 implementations of the `pubsub` interfaces: | ||
|
|
||
| For pubsub via Amazon's SNS/SQS, you can use the `pubsub/aws` package. | ||
|
|
||
| For pubsub via Google's Pubsub, you can use the `pubsub/gcp` package. This package offers 2 ways of publishing to Google PubSub. `gcp.NewPublisher` uses the RPC client and `gcp.NewHTTPPublisher` will publish over plain HTTP, which is useful for the App Engine standard environment. | ||
|
|
||
| For pubsub via Kafka topics, you can use the `pubsub/kafka` package. | ||
|
|
||
| For publishing via HTTP, you can use the `pubsub/http` package. | ||
|
|
||
| The `MultiPublisher` interface is only implemented by `pubsub/gcp`. | ||
|
|
||
| ## The `pubsub/pubsubtest` package | ||
|
|
||
| This package contains 'test' implementations of the `pubsub.Publisher`, `pubsub.MultiPublisher`, and `pubsub.Subscriber` interfaces that will allow developers to easily mock out and test their `pubsub` implementations: | ||
|
|
||
| ```go | ||
| type TestPublisher struct { | ||
| // Published will contain a list of all messages that have been published. | ||
| Published []TestPublishMsg | ||
|
|
||
| // GivenError will be returned by the TestPublisher on publish. | ||
| // Good for testing error scenarios. | ||
| GivenError error | ||
|
|
||
| // FoundError will contain any errors encountered while marshalling | ||
| // the protobuf struct. | ||
| FoundError error | ||
| } | ||
|
|
||
| type TestSubscriber struct { | ||
| // ProtoMessages will be marshalled into []byte used to mock out | ||
| // a feed if it is populated. | ||
| ProtoMessages []proto.Message | ||
|
|
||
| // JSONMEssages will be marshalled into []byte and used to mock out | ||
| // a feed if it is populated. | ||
| JSONMessages []interface{} | ||
|
|
||
| // GivenErrError will be returned by the TestSubscriber on Err(). | ||
| // Good for testing error scenarios. | ||
| GivenErrError error | ||
|
|
||
| // GivenStopError will be returned by the TestSubscriber on Stop(). | ||
| // Good for testing error scenarios. | ||
| GivenStopError error | ||
|
|
||
| // FoundError will contain any errors encountered while marshalling | ||
| // the JSON and protobuf struct. | ||
| FoundError error | ||
| } | ||
| ``` | ||
|
|
||
| ## The `web` package | ||
|
|
||
| This package contains a handful of very useful functions for parsing types from request queries and payloads. | ||
|
|
||
| ## Examples | ||
|
|
||
| * Several reference implementations utilizing `server` and `pubsub` are available in the ['examples'](https://github.com/NYTimes/gizmo/tree/master/examples) subdirectory. | ||
| <sub><strong>If you experience any issues please create an issue and/or reach out on the #gizmo channel in the [Gophers Slack Workspace](https://invite.slack.golangbridge.org) with what you've found.</strong></sub> | ||
|
|
||
| <sub>The Gizmo logo was based on the Go mascot designed by Renée French and copyrighted under the Creative Commons Attribution 3.0 license.</sub> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters