Ability to extend default KCP server with http.Handler

[mjudeikis]

Im working on project, based on KCP where I would like to extend existing KCP API server with some arbitrary http.Handers
This would be very similar how now KCP is being enabled to use syncer tunners :

if kcpfeatures.DefaultFeatureGate.Enabled(kcpfeatures.SyncerTunnel) {
	apiHandler = tunneler.WithSyncerTunnel(apiHandler)
}

This way in my use-case I would be able to dynamically provision routes under handler in my control.
Proposal would be:

1. Add Extra options:

type ExtraOptions struct {
        ...
	AdditionalAPIHandlers    []func(apiHandler http.Handler) http.HandlerFunc

	BatteriesIncluded []string
}

2. Add loading of the handers to the chain:

// For library usage we allow to extend the handler chain
// with user provided handlers. This allows to add custom
// http handlers to the and integrate more tightly with the
// kcp server.
if len(opts.Extra.AdditionalAPIHandlers) > 0 {
	for _, h := range opts.Extra.AdditionalAPIHandlers {
		apiHandler = h(apiHandler)
	}
}

3. NOT expose this via flags but leave ability to do this in library usage:

serverOptions := options.NewOptions(rootDir)
serverOptions.GenericControlPlane.Logs.Config.Verbosity = config.VerbosityLevel(2)
serverOptions.Extra.AdditionalAPIHandlers = []func(h http.Handler) http.HandlerFunc{
	faroserver.WithCustomTunnels,
}

completed, err := serverOptions.Complete()
if err != nil {
	return err
}

This would enable to hook almost any handlers (assuming security and logic is out of KCP scope for those),

Would something like this would be acceptable OR maybe there is already way to achieve this today?

[ncdc]

We have

kcp/pkg/server/config.go

Lines 338 to 344 in 7b33e59

	// Add a mux before the chain, for other handlers with their own handler chain to hook in. For example, when
	// the virtual workspace server is running as part of kcp, it registers /services with the mux so it can handle
	// that path itself, instead of the rest of the handler chain above handling it.
	mux := http.NewServeMux()
	mux.Handle("/", apiHandler)
	*c.preHandlerChainMux = append(*c.preHandlerChainMux, mux)
	apiHandler = mux

but it's currently not exported. @sttts would it be worth exporting? @mjudeikis if it were, could you register a path handler, or do you need it lower level?

Either way, I'm in favor of supporting this use case.

[mjudeikis]

I was thinking about lower level so it's more flexibile for others to extend for wider usage. For my case path works. But in reality it is very limiting :)

[ncdc]

Yeah, makes sense!

[sttts]

serverOptions.Extra.AdditionalAPIHandlers is a little misplaced. I could see this in the Config object.

[sttts]

Can't you already get to the GenericAPIServer within the server struct? And then add handlers? (UnprotectedHandlerSomething)

[mjudeikis]

I found this genericapiserver.NewEmptyDelegateWithCustomHandler and s.GenericControlPlane.GenericAPIServer.UnprotectedHandler() But I dont think I can use them so I could add handler to close chain itself.

I was not able to find right way to extend this with existing machinery. If you sure there is some handler to achieve this, I would really appreciate some pointer as these 2 gave me no luck :/

[sttts]

s.GenericControlPlane.GenericAPIServer.Handler.NonGoRestfulMux.HandleFunc

[mjudeikis]

t := faroserver.NewTunneler()
			s.GenericControlPlane.GenericAPIServer.Handler.NonGoRestfulMux.HandleFunc("/services/faros-tunnels", t.CustomTunnels)
```
Like a charm :)

[mjudeikis]

I think I spoke too early. Basically this is "In the chain" So all kube rbac applies to it. So if you want to make something outside the chain (tunnel, webserver, authentication entry point) - This does not work.