gRPC error after being idle for a while

[Scapal]

I have a Go server exposing a REST API, using Dgraph as my database.

I create a single grpc connection once and reuse the dgo client during the entire service lifespan.

Whenever my server app is idle for too long, I get the following error during my next query or mutation:
rpc error: code = Unavailable desc = transport is closing

If I retry, it works again. So it seems the Dgraph server is closing the connection without the dgo client knowing about it.

Is this a dgo, a grpc problem or a Dgraph bug?

Tested with Dgraph v1.0.9 and v1.0.10

google.golang.org/grpc v1.16.0
github.com/dgraph-io/dgo v0.0.0-20181102011806-23d7ac35e2c7

[srfrog]

This is not a bug. Currently Dgraph doesn't support long-running connections so it uses the default connection timeout (90 secs?). It will disconnect if it hasn't received any requests during that period.

For this to work as you expect, we would need to add a keep-alive policy to the server. Then you could have the client connect and stay as long as the policy is held. The keep-alive sends a packet periodically to keep the connection. Most likely this would add new a parameter flag to alpha, but the default state has to remain the same as now, or we could end up with zombie connections.

In the mean time, you could just reconnect every 90 secs or so.

[Scapal]

Could the dgo client then automatically retry once when getting that specific error ?
When the connection is closed by the server, the next dgo request gets an error transport is closing, but the following requests are working fine.

[manishrjain]

Instead of a mechanism at the server, you could just ping the server every minute in a goroutine from the client.

[Scapal]

Since it will probably be the most common use case, you should probably have a clear documented pattern for long connections

[manishrjain]

This is the default Grpc behavior, not specific to Dgraph.

https://github.com/grpc/grpc/blob/master/doc/keepalive.md

[Scapal]

Instead of a mechanism at the server, you could just ping the server every minute in a goroutine from the client.

When you say "ping the server" you mean "perform a simple costless query" or is there an actual ping query on Dgraph ?

[manishrjain]

You can call CheckVersion API. It should do the same job as an echo.

[Scapal]

I've set the gRPC dial option grpc.WithDefaultCallOptions(grpc.FailFast(false)) to see if that fixes the problem without requiring pings.

[Scapal]

I've set the gRPC dial option grpc.WithDefaultCallOptions(grpc.FailFast(false)) to see if that fixes the problem without requiring pings.

It doesn't work, so I went for the CheckVersion keepalive:

import (
	"context"
	"time"

	"github.com/dgraph-io/dgo/protos/api"
	log "github.com/sirupsen/logrus"
)

type DgraphKeepAlive struct {
	ticker *time.Ticker
	stop   chan struct{}
	done   chan struct{}
	client api.DgraphClient
}

func NewDgraphKeepAlive(client api.DgraphClient, d time.Duration) *DgraphKeepAlive {
	dka := DgraphKeepAlive{
		ticker: time.NewTicker(d),
		stop:   make(chan struct{}),
		done:   make(chan struct{}),
		client: client,
	}

	go func() {
		defer dka.ticker.Stop()
		defer close(dka.done)
		for {
			select {
			case <-dka.stop:
				log.Info("Stopped DGraph Keepalive")
				return
			case <-dka.ticker.C:
				log.Debug("DGraph Keepalive Ping")
				err := Ping(dka.client)
				if err != nil {
					log.Errorf("Error during Dgraph keepalive ping: %v", err)
				}
			}
		}
	}()
	return &dka
}

func (dka *DgraphKeepAlive) Stop() {
	close(dka.stop)
	<-dka.done
}

func Ping(c api.DgraphClient) error {
	var check api.Check
	_, err := c.CheckVersion(context.TODO(), &check)
	return err
}