fix: support ConsumerGroups in KongRawStateToKongState

[programmer04]

What this PR does / why we need it:

Support properly ConsumerGroups when fallback to the last valid configuration. Function configfetcher.KongRawStateToKongState now supports ConsumerGroup entities. It means that if KIC fetches the last valid configuration from Gateways, it will include all ConsumerGroups that a Gateway may return.

Which issue this PR fixes:

Fixes #5377

Special notes for your reviewer:

Having explicitly omitted checking this in internal/dataplane/configfetcher/kongrawstate_test.go:318 caused that the test hasn't caught it during the work on ConsumerGroup. I thing the same may happen for Vault 🤔

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c9e95e9) 69.3% compared to head (cc7d0f7) 69.4%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #5438   +/-   ##
=====================================
  Coverage   69.3%   69.4%           
=====================================
  Files        176     176           
  Lines      22379   22386    +7     
=====================================
+ Hits       15529   15556   +27     
+ Misses      5925    5911   -14     
+ Partials     925     919    -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[czeslavo]

Having explicitly omitted checking this in internal/dataplane/configfetcher/kongrawstate_test.go:318 caused that the test hasn't caught it during the work on ConsumerGroup. I thing the same may happen for Vault 🤔

Yup, the intention was that once implementing a new field in KongState we will remove the exclude from the test. But I think we might improve that and instead of asserting all of KongRawState - excluded fields to be covered in tests to assert all of kongstate.KongState fields to be covered as that's the struct KIC defines and once we add anything there, it should be also covered in the mapping from KongRawState. That would be something like this:

--- a/internal/dataplane/configfetcher/kongrawstate_test.go
+++ b/internal/dataplane/configfetcher/kongrawstate_test.go
@@ -277,7 +277,7 @@ func TestKongRawStateToKongState(t *testing.T) {
 
                        // Collect all fields that are tested in this test case.
                        if tt.kongRawState != nil {
-                               testedKongRawStateFields.Insert(extractNotEmptyFieldNames(*tt.kongRawState)...)
+                               testedKongRawStateFields.Insert(extractNotEmptyFieldNames(*tt.expectedKongState)...)
                        }
 
                        var state *kongstate.KongState
@@ -290,12 +290,12 @@ func TestKongRawStateToKongState(t *testing.T) {
                })
        }
 
-       ensureAllKongRawStateFieldsAreTested(t, testedKongRawStateFields.UnsortedList())
+       ensureAllKongStateFieldsAreTested(t, testedKongRawStateFields.UnsortedList())
 }
 
-// extractNotEmptyFieldNames returns the names of all non-empty fields in the given KongRawState.
+// extractNotEmptyFieldNames returns the names of all non-empty fields in the given KongState.
 // This is to programmatically find out what fields are used in a test case.
-func extractNotEmptyFieldNames(s utils.KongRawState) []string {
+func extractNotEmptyFieldNames(s kongstate.KongState) []string {
        var fields []string
        typ := reflect.ValueOf(s).Type()
        for i := 0; i < typ.NumField(); i++ {
@@ -308,22 +308,13 @@ func extractNotEmptyFieldNames(s utils.KongRawState) []string {
        return fields
 }
 
-// ensureAllKongRawStateFieldsAreTested verifies that all fields in KongRawState are tested.
+// ensureAllKongStateFieldsAreTested verifies that all fields in KongState are tested.
 // It uses the testedFields slice to determine what fields were actually tested and compares
-// it to the list of all fields in KongRawState, excluding fields that KIC doesn't support.
-func ensureAllKongRawStateFieldsAreTested(t *testing.T, testedFields []string) {
-       kongRawStateFieldsKICDoesntSupport := []string{
-               // These are fields that KIC explicitly doesn't support.
-               "SNIs",
-               "ConsumerGroups",
-               "CustomEntities",
-               "Vaults",
-               "RBACRoles",
-               "RBACEndpointPermissions",
-       }
-       allKongRawStateFields := func() []string {
+// it to the list of all fields in KongState, excluding fields that KIC doesn't support.
+func ensureAllKongStateFieldsAreTested(t *testing.T, testedFields []string) {
+       allKongStateFields := func() []string {
                var fields []string
-               typ := reflect.ValueOf(utils.KongRawState{}).Type()
+               typ := reflect.ValueOf(kongstate.KongState{}).Type()
                for i := 0; i < typ.NumField(); i++ {
                        fields = append(fields, typ.Field(i).Name)
                }
@@ -331,11 +322,7 @@ func ensureAllKongRawStateFieldsAreTested(t *testing.T, testedFields []string) {
        }()
 
        // Meta test - ensure we have testcases covering all fields in KongRawState.
-       for _, field := range allKongRawStateFields {
-               if lo.Contains(kongRawStateFieldsKICDoesntSupport, field) {
-                       t.Logf("skipping field %s - explicitly unsupported", field)
-                       continue
-               }
+       for _, field := range allKongStateFields {
                assert.True(t, lo.Contains(testedFields, field), "field %s not tested", field)
        }
 }

This catches Vaults, ConsumerGroups and Licenses as not tested. 👍

[programmer04]

Neat idea @czeslavo. I checked it and it detects the field Plugins too as false-positive I think. The option is to explicitly ignore it... Also, I discovered that KongRawState doesn't contain the field Licenses, it has to be added