Permalink
Cannot retrieve contributors at this time
Name already in use
A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
bigmachine/doc.go
Go to fileThis commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
112 lines (85 sloc)
3.75 KB
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Copyright 2018 GRAIL, Inc. All rights reserved. | |
// Use of this source code is governed by the Apache 2.0 | |
// license that can be found in the LICENSE file. | |
/* | |
Package bigmachine implements a vertically integrated stack for | |
distributed computing in Go. Go programs written with bigmachine are | |
transparently distributed across a number of machines as | |
instantiated by the backend used. (Currently supported: EC2, local | |
machines, unit tests.) Bigmachine clusters comprise a driver node | |
and a number of bigmachine nodes (called "machines"). The driver | |
node can create new machines and communicate with them; machines can | |
call each other. | |
Computing model | |
On startup, a bigmachine program calls driver.Start. Driver.Start | |
configures a bigmachine instance based on a set of standard flags | |
and then starts it. (Users desiring a lower-level API can use | |
bigmachine.Start directly.) | |
import ( | |
"github.com/grailbio/bigmachine" | |
"github.com/grailbio/bigmachine/driver" | |
... | |
) | |
func main() { | |
flag.Parse() | |
// Additional configuration and setup. | |
b, shutdown := driver.Start() | |
defer shutdown() | |
// Driver code... | |
} | |
When the program is run, driver.Start returns immediately: the program | |
can then interact with the returned bigmachine B to create new | |
machines, define services on those machines, and invoke methods on | |
those services. Bigmachine bootstraps machines by running the same | |
binary, but in these runs, driver.Start never returns; instead it | |
launches a server to handle calls from the driver program and other | |
machines. | |
A machine is started by (*B).Start. Machines must be configured with | |
at least one service: | |
m, err := b.Start(ctx, bigmachine.Services{ | |
"MyService": &MyService{Param1: value1, ...}, | |
}) | |
Users may then invoke methods on the services provided by the | |
returned machine. A services's methods can be invoked so long as | |
they are of the form: | |
Func(ctx context.Context, arg argType, reply *replyType) error | |
See package github.com/grailbio/bigmachine/rpc for more details. | |
Methods are named by the sevice and method name, separated by a dot | |
('.'), e.g.: "MyService.MyMethod": | |
if err := m.Call(ctx, "MyService.MyMethod", arg, &reply); err != nil { | |
log.Print(err) | |
} else { | |
// Examine reply | |
} | |
Since service instances must be serialized so that they can be transmitted | |
to the remote machine, and because we do not know the service types | |
a priori, any type that can appear as a service must be registered with | |
gob. This is usually done in an init function in the package that declares | |
type: | |
type MyService struct { ... } | |
func init() { | |
// MyService has method receivers | |
gob.Register(new(MyService)) | |
} | |
Vertical computing | |
A bigmachine program attempts to appear and act like a single program: | |
- Each machine's standard output and error are copied to the driver; | |
- bigmachine provides aggregating profile handlers at /debug/bigmachine/pprof | |
so that aggregate profiles may be taken over the full cluster; | |
- command line flags are propagated from the driver to the machine, | |
so that a binary run can be configured in the usual way. | |
The driver program maintains keepalives to all of its machines. Once | |
this is no longer maintained (e.g., because the driver finished, or | |
crashed, or lost connectivity), the machines become idle and shut | |
down. | |
Services | |
A service is any Go value that implements methods of the form given | |
above. Services are instantiated by the user and registered with | |
bigmachine. When a service is registered, bigmachine will also | |
invoke an initialization method on the service if it exists. | |
Per-machine initialization can be performed by this method.The form | |
of the method is: | |
Init(*Service) error | |
If a non-nil error is returned, the machine is considered failed. | |
*/ | |
package bigmachine |