graphpipe-go provides a variety of functions to help you easily serve
and access ml models using the very speedy GraphPipe protocol.
Additionally, this package provides reference server implementations for common ML model formats, including:
For an overview of GraphPipe, read our project documentation
If you are interested in learning a bit more about how the go servers and clients work, read on:
Most users integrating GraphPipe into their application will be interested making client calls. To make remote calls, we provide three different APIs
// Remote is the simple function for making a remote model request with a
// single input and output and no config. It performs introspection and
// automatic type conversion on its input and output. It will use the server
// defaults for input and output.
func Remote(uri string, in interface{}) (interface{}, error)
// MultiRemote is the complicated function for making a remote model request.
// It supports multiple inputs and outputs, custom clients, and config strings.
// If inputNames or outputNames is empty, it will use the default inputs and
// outputs from the server. For multiple inputs and outputs, it is recommended
// that you Specify inputNames and outputNames so that you can control
// input/output ordering. MultiRemote also performs type introspection for
// inputs and outputs.
func MultiRemote(client *http.Client, uri string, config string, ins []interface{}, inputNames, outputNames []string) ([]interface{}, error)
// MultiRemoteRaw is the actual implementation of the remote model
// request using NativeTensor objects. The raw call is provided
// for requests that need optimal performance and do not need to
// be converted into native go types.
func MultiRemoteRaw(client *http.Client, uri string, config string, inputs []*NativeTensor, inputNames, outputNames []string) ([]*NativeTensor, error) {
In similar fashion to the serving model, the client for making remote calls is made up of three functions, Remote, MultiRemote, and MultiRemoteRaw.
The functions range from simple to complex. The first three of those will
convert your native Go types into tensors and back, while the last one uses
graphpipe tensors throughout.
There are two Serve functions, Serve and ServeRaw, that both create standard Go http listeners and support caching with BoltDB.
For applications where the manipulation of tensors will mostly be in go, Serve
provides a wrapper to your apply function that allows you to work with
standard go types rather than explicit graphpipe data objects and converts
between them for you. From the Go docs:
// Serve offers multiple inputs and outputs and converts tensors
// into native datatypes based on the shapes passed in to this function
// plus any additional shapes implied by your apply function.
// If cache is true, will attempt to cache using cache.db as cacheFile
func Serve(listen string, cache bool, apply interface{}, inShapes, outShapes [][]int64) error {}
As an example, here is a simple way to construct a graphpipe identity server, which can receive a graphpipe network request, and echo it back to the client:
package main
import (
"github.com/Sirupsen/logrus"
graphpipe "github.com/oracle/graphpipe-go"
)
func main() {
logrus.SetLevel(logrus.InfoLevel)
useCache := false // toggle caching on/off
inShapes := [][]int64(nil) // Optionally set input shapes
outShapes := [][]int64(nil) // Optionally set output shapes
if err := graphpipe.Serve("0.0.0.0:9000", useCache, apply, inShapes, outShapes); err != nil {
logrus.Errorf("Failed to serve: %v", err)
}
}
func apply(requestContext *graphpipe.RequestContext, ignore string, in interface{}) interface{} {
return in // using the graphpipe.Serve interface, graphpipe automatically converts
// go native types to tensors.
}
You can find a docker-buildable example of this server here.
For applications that will be passing the tensors directly to another system for processing and don't need conversion to standard Go types, ServeRaw provides a lower-level interface. For examples of apps that use ServeRaw, see graphpipe-tf and graphpipe-onnx.
As you might expect, Serve uses ServeRaw underneath the hood.