Proto schema registry refresh optimizations #1040
Conversation
backend/pkg/schema/client.go
Outdated
| errors := make([]error, len(subjectsRes.Subjects)) | ||
| hasErrors := atomic.Bool{} |
There was a problem hiding this comment.
I think we should do errors := make([]error, 0, len(subjectsRes.Subjects)) and then we can check the length of errors to see if we have errors. I think in most cases we do not expect every single subject to return an error.
It feels a bit weird to return a slice of schemas and a slice of errors. At this point we can probably just refactor the function to return schema + error via a channel and handle the error and returned schema on the receiving side.
There was a problem hiding this comment.
The intent is to be able to get all the schemas we successfully fetched, even if we failed to retrieve some subset of all the schemas in SR. So we return all the successfully fetched schemas and a collection of errors.
This is the case for both GetSchemasIndividually() and GetSchemas(), and then GetProtoDescriptors() we are able to log all the failed schemas, but still be able to parse and compile the successfully retrieved ones so we could potentially use them.
| s.requestGroup.Forget(key) | ||
| return nil, fmt.Errorf("failed to get schema from registry: %w", err) | ||
| _, err, _ := s.requestGroup.Do(key, func() (any, error) { | ||
| schemasRes, errs := s.registryClient.GetSchemas(ctx, false) |
There was a problem hiding this comment.
schemaRes will be nil as far as I can tell if all retrieved schemas errored (e.g. schema registry not reachable). I think this func will still work if this is the case, but would be good to test the case where schemaRes is nil.
Rather than using singleflight we could use https://github.com/twmb/go-cache to simplify some things, but I assume you tried to keep as much of the original code as possible
Iterative work for optimizing some of the proto schema registry refreshing.
Changes
GetSchemasBySubject(),GetSchemas(),GetSchemasIndividually()to return pointer types.gciseems to incorrectly sort the"maps"import. Updated the tooling hoping it would address the issue, but it didn't seem to help, so I commented that step out for now.I will try and add some tests for this soon.