curl log REST endpoint causing memory error

[zhxu73]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:

Argo server hit a invalid memory address or nil pointer dereference when query the /api/v1/workflows/{namespace}/{name}/{podName}/log REST endpoint

{"error":{"grpc_code":13,"http_code":500,"message":"runtime error: invalid memory address or nil pointer dereference","http_status":"Internal Server Error"}}

What you expected to happen:

returning actual logs of the pod, similar to argo logs or kubectl logs

How to reproduce it (as minimally and precisely as possible):

• minikube v1.11.0
• create argo namespace
• modify the default install manifest yaml (https://raw.githubusercontent.com/argoproj/argo/v2.9.1/manifests/install.yaml) to use a NodePort for argo-server service, so that the service is exposed

apiVersion: v1
kind: Service
metadata:
  name: argo-server
spec:
  type: NodePort
  ports:
  - name: web
    port: 2746
    targetPort: 2746
    nodePort: 32746
  selector:
    app: argo-server

• kubectl apply the modified yaml
• role binding for cli

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default -n argo

• submit a hello world workflow using argo cli (https://github.com/argoproj/argo/blob/master/examples/hello-world.yaml)
• get the workflow name and the pod name using argo cli (which are the same for the hello world workflow)
• get the service url, minikube service argo-server --url -n argo
• curl the log endpoint, uses the workflow name and pod name from above
  curl http://172.17.0.3:32746/api/v1/workflows/argo/$workflow_name/$pod_name/log
• curl return a 500 along with a json saying invalid memory address or nil pointer dereference

Anything else we need to know?:

Environment:

• Argo version:
  v2.9.1
• Kubernetes version :

  buildDate: "2020-06-26T03:47:41Z"
  compiler: gc
  gitCommit: e6503f8d8f769ace2f338794c914a96fc335df0f
  gitTreeState: clean
  gitVersion: v1.18.5
  goVersion: go1.13.9
  major: "1"
  minor: "18"
  platform: linux/amd64
serverVersion:
  buildDate: "2020-04-16T11:36:15Z"
  compiler: gc
  gitCommit: a17149e1a189050796ced469dbd78d380f2ed5ef
  gitTreeState: clean
  gitVersion: v1.16.9
  goVersion: go1.13.9
  major: "1"
  minor: "16"
  platform: linux/amd64

Logs

kubectl logs -n argo $(kubectl get pods -l app=argo-server -n argo -o name)

time="2020-07-08T00:09:19Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference\ngoroutine 162 [running]:\nruntime/debug.Stack(0xc00059aaa8, 0x1cf0f60, 0x37786b0)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/argoproj/argo/util/grpc.PanicLoggerStreamServerInterceptor.func1.1(0xc000108460, 0xc00059b920)\n\t/go/src/github.com/argoproj/argo/util/grpc/interceptor.go:31 +0x6e\npanic(0x1cf0f60, 0x37786b0)\n\t/usr/local/go/src/runtime/panic.go:679 +0x1b2\ngithub.com/argoproj/argo/util/logs.WorkflowLogs(0x2776fc0, 0xc00019ed80, 0x27357c0, 0xc00091ee10, 0x27c6260, 0xc0008fe000, 0x27771c0, 0xc0004a8360, 0x7f9d79d884d0, 0xc0008be1c0, ...)\n\t/go/src/github.com/argoproj/argo/util/logs/workflow-logger.go:69 +0x65f\ngithub.com/argoproj/argo/server/workflow.(*workflowServer).PodLogs(0xc000388060, 0xc0004a8360, 0x2791aa0, 0xc0008be1c0, 0xc000388060, 0xc0008fe000)\n\t/go/src/github.com/argoproj/argo/server/workflow/workflow_server.go:385 +0x2e2\ngithub.com/argoproj/argo/pkg/apiclient/workflow._WorkflowService_PodLogs_Handler(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0xc00019ed80, 0x0)\n\t/go/src/github.com/argoproj/argo/pkg/apiclient/workflow/workflow.pb.go:1708 +0x109\ngithub.com/argoproj/argo/server/auth.(*gatekeeper).StreamServerInterceptor.func1(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0xc000230fe0, 0x2409090, 0x0, 0x0)\n\t/go/src/github.com/argoproj/argo/server/auth/gatekeeper.go:68 +0xcc\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0x0, 0x203000)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:49 +0x5f\ngithub.com/argoproj/argo/util/grpc.glob..func2(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0xc000230fe0, 0xc000231000, 0xc, 0x1fc8b23)\n\t/go/src/github.com/argoproj/argo/util/grpc/interceptor.go:45 +0x4e\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0x0, 0x0)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:49 +0x5f\ngithub.com/argoproj/argo/util/grpc.PanicLoggerStreamServerInterceptor.func1(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0xc000230fe0, 0xc000231040, 0x0, 0x0)\n\t/go/src/github.com/argoproj/argo/util/grpc/interceptor.go:35 +0xa8\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1eed180, 0xc000388060, 0x278b2c0, 0xc000231280, 0x21, 0xbfb9618bc950a178)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:49 +0x5f\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.StreamServerInterceptor.func1(0x1eed180, 0xc000388060, 0x278b4a0, 0xc00075e240, 0xc000230fe0, 0xc000231100, 0xc000231220, 0xc000230fe0)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/logging/logrus/server_interceptors.go:66 +0x176\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1eed180, 0xc000388060, 0x278b4a0, 0xc00075e240, 0xc00083ec10, 0x40c698)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:49 +0x5f\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x1eed180, 0xc000388060, 0x278b4a0, 0xc00075e240, 0xc000230fe0, 0x2409090, 0x2776fc0, 0xc00019eba0)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:58 +0xcf\ngoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc000190340, 0x279d500, 0xc0005d3080, 0xc0000e2b00, 0xc0003880f0, 0x3784de0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1336 +0x486\ngoogle.golang.org/grpc.(*Server).handleStream(0xc000190340, 0x279d500, 0xc0005d3080, 0xc0000e2b00, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1409 +0xcbc\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00025b500, 0xc000190340, 0x279d500, 0xc0005d3080, 0xc0000e2b00)\n\t/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xbb\ncreated by google.golang.org/grpc.(*Server).serveStreams.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1\n"
time="2020-07-08T00:09:19Z" level=error msg="finished streaming call with code Internal" error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" grpc.code=Internal grpc.method=PodLogs grpc.service=workflow.WorkflowService grpc.start_time="2020-07-08T00:09:19Z" grpc.time_ms=32.451 span.kind=server system=grpc

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

Panic in workflow-logger.go line 69: the request is missing log options - and that should be catered for.

@zhxu73 this can be easily fixed by adding a nil check at line 68 - would you be interested in submitting a PR.

In the meantime - you maybe able to workaround by adding log options to you request:

/api/v1/workflows/{namespace}/{name}/{podName}/log?logOptions.timestamps=true