Add grpc/http multiplexed server with stubbed core handler to verify (#…

…2789)

* Add multiplexed grpc/http service with stubbed core RegisteredPlugins call to verify.

* go mod tidy after rebase

* Update to use v1alpha1 and service-specific scope (ie. core.plugins.v1alpha1 rather than core.v1).

* Remove comments

* Fix Makefile target name and add test target.

cmd/kubeapps-apis/Makefile

@@ -18,9 +18,25 @@
 # See https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
 cli-dependencies:
 	go get \
-		github.com/spf13/cobra/cobra
+		github.com/spf13/cobra/cobra \
+		google.golang.org/grpc/cmd/protoc-gen-go-grpc \
+		google.golang.org/protobuf/cmd/protoc-gen-go
+
+# Not sure if it's worth having targets for buf, but at least the targets can
+# be tooling agnostic.
+# NOTE: I have an issue with buf in that if I try to update the protobuf dependencies
+# (defined in the buf.yaml) with `buf beta mod update`, the `buf.lock` file updates
+# accordingly, but then am unable to run `buf generate` or any other command that
+# pulls the dependencies to the local cache. I've reported this and created a minimal
+# demo of the issue (at least, for me) at: https://github.com/bufbuild/buf/issues/294#issuecomment-837839297
+# I'll update once I've resolved that.
+generate:
+	buf generate
 
 run:
 	go run main.go
 
-.PHONY: cli-dependencies run
+test:
+	go test ./...
+
+.PHONY: cli-dependencies generate run test

cmd/kubeapps-apis/README.md

@@ -2,6 +2,20 @@
 
 This is an experimental service that we are not yet including in our released Kubeapps chart, providing an extensible protobuf-based API service enabling Kubeapps (or other packaging UIs) to interact with various Kubernetes packaging formats in a consistent way.
 
+## Extensible
+
+The API definitions are using Google protocol buffers - an Interface Definition Language that was created to ensure that API messages and API services can be defined, extended and maintained with the least friction, in a language-neutral way (though we are exclusively using go here).
+
+gRPC is an RPC implementation that uses protocol buffers, providing RPC functionality across languages. Importantly, even though we'll be writing the API service in go, applications using the service can use any language they want and get pre-built clients for that language.
+
+If you are unfamiliar with protocol buffers and gRPC, it's worthwhile to at least read the [introduction](https://grpc.io/docs/what-is-grpc/introduction/) and [core concepts](https://www.grpc.io/docs/what-is-grpc/core-concepts/).
+
+Though it is possible to use a JS/TypeScript client, it is also possible to use a protocol buffer extensions provided by Google that enables optionally exposing the RPCs as http requests, demo'd here. This is made possible via the [grpc-gateway project](https://github.com/grpc-ecosystem/grpc-gateway) which proxies JSON http requests through to the relevant gRPC call. As shown in the demo (and by many others examples online), both http and grpc can be served on the same port without issue.
+
+Together, this enables the best of both worlds: a well known and used Interface Definition Language for defining extensible APIs that can also be exposed via a rest-like http interface.
+
+Finally, we've also chosen to use the [buf](https://buf.build/) tool for generating the code from the proto files. In the past we've used `protoc` (proto buffer compiler) and its extensions directly, but `buf` allows you to specify a simple yaml config instead, and also provides a `lint` command to ensure that your choice of API structure follows best practise, as well as ensuring you're aware when you break backwards compatability.
+
 ## CLI
 
 Similar to most go commands, we've used [Cobra](https://github.com/spf13/cobra) for the CLI interface. Currently there is only a root command to run server, but we may later add a `version` subcommand or a `new-plugin` subcommand, but even without these it provides a lot of useful defaults for config, env var support etc.
@@ -16,18 +30,30 @@ make run
 I0511 11:39:56.444553 4116647 server.go:25] Starting server on :50051
 ```
 
-You can then verify the (currently echo) service via http:
+You can then verify the (currently stubbed) configured plugins endpoint via http:
 
 
 ```bash
-curl http://localhost:50051/helm/v1/available-packages
+curl http://localhost:50051/core/plugins/v1alpha1/configured-plugins
+
+{"plugins":["foobar.package.v1"]}
+```
 
-Hello, "/helm/v1/available-packages"
+or via gRPC (using the [grpcurl tool](https://github.com/fullstorydev/grpcurl)):
+
+```bash
+grpcurl -plaintext localhost:50051 kubeappsapis.core.plugins.v1alpha1.PluginsService.GetConfiguredPlugins
+{
+  "plugins": [
+    "foobar.package.v1"
+  ]
+}
 ```
 
+
 ## Hacking
 
-A few exra tools will be needed to contribute to the development of this service.
+A few extra tools will be needed to contribute to the development of this service.
 
 ### Install go cli deps
 
@@ -38,3 +64,18 @@ make cli-dependencies
 ```
 
 This will ensure that the cobra command is available should you need to add a sub-command.
+
+### Install buf
+
+Grab the latest binary from the [buf releases](https://github.com/bufbuild/buf/releases).
+
+You can now try changing the url in the proto file (such as in `proto/kubeappsapis/core/v1/core.proto`) and then run:
+
+```
+buf generate
+make run
+```
+
+and then verify that the RegisteredPlugins RPC call is exposed via HTTP at the new URL path that you specified.
+
+You can also use `buf lint` to ensure that the proto IDLs are sane (ie. extensible, no backwards incompatible changes etc.)

cmd/kubeapps-apis/buf.gen.yaml

@@ -0,0 +1,14 @@
+version: v1beta1
+plugins:
+  - name: go
+    out: gen
+    opt:
+      - module=github.com/kubeapps/kubeapps/cmd/kubeapps-apis/gen
+  - name: go-grpc
+    out: gen
+    opt:
+      - module=github.com/kubeapps/kubeapps/cmd/kubeapps-apis/gen
+  - name: grpc-gateway
+    out: gen
+    opt:
+      - module=github.com/kubeapps/kubeapps/cmd/kubeapps-apis/gen

cmd/kubeapps-apis/buf.lock

@@ -0,0 +1,9 @@
+# Generated by buf. DO NOT EDIT.
+deps:
+  - remote: buf.build
+    owner: beta
+    repository: googleapis
+    branch: main
+    commit: aced890b7e944f9ca174481ad810ca5f
+    digest: b1-guLHfQhGOhlEA-QaPyQLoT-ZVFGJkRkluWgkzPT0ma0=
+    create_time: 2021-03-04T14:52:47.094456Z

cmd/kubeapps-apis/buf.yaml

@@ -0,0 +1,13 @@
+version: v1beta1
+name: "buf.build/kubeapps/kubeapps"
+build:
+  roots:
+    - proto
+lint:
+  use:
+    - DEFAULT
+  except:
+    - RPC_REQUEST_RESPONSE_UNIQUE
+deps:
+  - buf.build/beta/googleapis
+  - buf.build/grpc-ecosystem/grpc-gateway