Skip to content

Commit

Permalink
refactor(kubernetes): Clean up KubernetesCacheUtils class (#4764)
Browse files Browse the repository at this point in the history
* refactor(kubernetes): Extract getHandler to a function

This is a minor refactor to make the upcoming changes cleaner;
just pull out a function.

* refactor(kubernetes): Fix type safety and null safety

resourceModelFromCacheData is very unsafe; it does an unchecked
cast and can also return null. The goal of that function is to
allow custom overrides for how to convert cache data into a
ServerGroup or a ServerGroupManager.  (We have not implemented
the ability to customize other types.)

The problem is that we are checking if the handler is of the
generic ModelHandler instead of the specific type we are
interested in.  We already have ServerGroupHandler and
ServerGroupManagerHandler to handle these two specific cases.
Split the generic resourceModelFromCacheData into to specific
ones to handle these two cases.  By checking if the handler
implements the specific case, we can avoid unchecked casts.

Finally, rather than return null if the handler doesn't specify
an override, just use the default implementation. The goal here
is only to allow people to override this if they want; if they
don't override, we can just use the default.

* refactor(kubernetes): Clean up CustomKubernetesHandlerFactory

This class implemented the (raw) ModelHandler, but only ever
implemented the default implementations for the kind in question.
As we now fall back to the default implementations in cases where
there is no override, we don't need to explicitly override them
here.

* refactor(kubernetes): Remove ModelHandler class

Now that we're using the more specific handler classes to detect
custom implementations of fromCacheData, we don't need the
ModelHandler interface at all anymore; remove it.

* refactor(kubernetes): Add relationshipTypes helper function

Minor refactor to abstract away translating a Spinnaker kind into
its relationship types.

* refactor(kubernetes): Add relationshipKeys function

Minor refactor to pull common logic looking up relationships on
a CacheData object into a function.

* refactor(kubernetes): Rename to function

Rename aggregateRelationshipsBySpinnakerKind as getRelationshipKeys,
because it does the same thing as the relationshipKeys function we
just created, but for a SpinnakerKind. Also collect to an
ImmutableList and update the return type.

Similarly remove the OfSpinnakerKind from getRelationships, becase
the function signature implies what the input is.

* refactor(kubernetes): Update return type of mapByRelationship

It will now return a Map<String, Collection<CacheData>> instead of
a Map<String, List<CacheData>> because it's not actually important
to callers that the values are List<?>. This will make the next
commit possible.

* refactor(kubernetes): mapByRelationship returns a MultiMap

This is a much better data structure for mapping keys to multiple
values. In particular, it has the property that .get(key) returns
an empty list rather than null if there are no mappings, so we can
avoid a lot of defaulting.

* refactor(kubernetes): Inline a variable

Inline the keys variable in loadRelationshipsFromCache; also
collect the relationship keys to a Set rather than to a List.

This cuts down the number of times we look up load balancers from
4 to 2; we'll still need to get that down to 1 but this is progress.

* refactor(kubernetes): Rename getTransitiveRelationships

Rename this function to something shorter and clearer; also update
it to only accept a single CacheData as all callers are wrapping a
single CacheData in a collection when calling it.

* refactor(kubernetes): Require callers of to create the filter

Simple refactor to push requiring the cache filter to be created
to callers. (Actually caller as this is only used in one place.)

* refactor(kubernetes): Require callers of getRelationships to pass data

getRelationships is inherently ineffecient, as it encourages callers to
require a lookup of data they already have.  In 2 of the 3 callers we
are either looking up the same data in a loop, or already have the data
we're looking up.

Require callers to first look up the cache data, then pass it in to
get its relationships.

* refactor(kubernetes): Rename loadRelationshipsFromCache

Use getRelationships to be consistent with how we're naming other
functions in this class.

Also update the single-data-consuming version to call the list
version with a singleton.

* refactor(kubernetes): Remove loadRelationshipsFromCache

This is identical to getRelationships.

Also the whole class is non-null now, so move the annotation to
the class level.

* refactor(kubernetes): Cleanup of KubernetesCacheUtils

Make the class package-private as it is only used in the package.
Also add Javadoc to some methods.

* refactor(kubernetes): Replace kindMap lookups with getRelationships

We already have a function in cacheUtils that looks up all relationships
for a SpinnakerKind; replace cases where callers map the SpinnakerKind
to KubernetesKinds with just calling this function.

* style(kubernetes): Fix comments

Co-Authored-By: Maggie Neterval <mneterval@google.com>

Co-authored-by: Maggie Neterval <mneterval@google.com>
  • Loading branch information
ezimanyi and maggieneterval committed Jul 30, 2020
1 parent 43a28f6 commit 3d90bde
Show file tree
Hide file tree
Showing 12 changed files with 168 additions and 241 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
import com.netflix.spinnaker.clouddriver.kubernetes.caching.Keys;
import com.netflix.spinnaker.clouddriver.model.Cluster;
import com.netflix.spinnaker.moniker.Moniker;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import lombok.Value;

Expand All @@ -43,8 +43,8 @@ public KubernetesV2Cluster(String rawKey) {

public KubernetesV2Cluster(
String rawKey,
List<KubernetesV2ServerGroup> serverGroups,
List<KubernetesV2LoadBalancer> loadBalancers) {
Collection<KubernetesV2ServerGroup> serverGroups,
Collection<KubernetesV2LoadBalancer> loadBalancers) {
Keys.ClusterCacheKey key = (Keys.ClusterCacheKey) Keys.parseKey(rawKey).get();
this.name = key.getName();
this.accountName = key.getAccount();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package com.netflix.spinnaker.clouddriver.kubernetes.caching.view.model;

import com.google.common.collect.ImmutableList;
import com.google.common.collect.Multimap;
import com.netflix.spinnaker.cats.cache.CacheData;
import com.netflix.spinnaker.clouddriver.kubernetes.caching.Keys;
import com.netflix.spinnaker.clouddriver.kubernetes.caching.agent.KubernetesCacheDataConverter;
Expand All @@ -27,8 +28,6 @@
import com.netflix.spinnaker.clouddriver.model.LoadBalancerProvider;
import com.netflix.spinnaker.clouddriver.model.LoadBalancerServerGroup;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
Expand All @@ -54,8 +53,8 @@ private KubernetesV2LoadBalancer(

public static KubernetesV2LoadBalancer fromCacheData(
CacheData cd,
List<CacheData> serverGroupData,
Map<String, ? extends Collection<CacheData>> serverGroupToInstanceData) {
Collection<CacheData> serverGroupData,
Multimap<String, CacheData> serverGroupToInstanceData) {
if (cd == null) {
return null;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,35 +17,36 @@

package com.netflix.spinnaker.clouddriver.kubernetes.caching.view.provider;

import static com.google.common.collect.ImmutableList.toImmutableList;
import static com.google.common.collect.ImmutableSet.toImmutableSet;

import com.google.common.collect.ImmutableCollection;
import com.google.common.collect.ImmutableListMultimap;
import com.google.common.collect.ImmutableMultimap;
import com.google.common.collect.ImmutableSet;
import com.netflix.spinnaker.cats.cache.Cache;
import com.netflix.spinnaker.cats.cache.CacheData;
import com.netflix.spinnaker.cats.cache.RelationshipCacheFilter;
import com.netflix.spinnaker.clouddriver.kubernetes.caching.Keys;
import com.netflix.spinnaker.clouddriver.kubernetes.caching.agent.KubernetesCacheDataConverter;
import com.netflix.spinnaker.clouddriver.kubernetes.caching.view.model.ManifestBasedModel;
import com.netflix.spinnaker.clouddriver.kubernetes.caching.view.provider.data.KubernetesV2CacheData;
import com.netflix.spinnaker.clouddriver.kubernetes.description.KubernetesResourceProperties;
import com.netflix.spinnaker.clouddriver.kubernetes.description.KubernetesSpinnakerKindMap;
import com.netflix.spinnaker.clouddriver.kubernetes.description.SpinnakerKind;
import com.netflix.spinnaker.clouddriver.kubernetes.description.manifest.KubernetesManifest;
import com.netflix.spinnaker.clouddriver.kubernetes.description.manifest.KubernetesKind;
import com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesHandler;
import com.netflix.spinnaker.clouddriver.kubernetes.op.handler.ModelHandler;
import java.util.ArrayList;
import com.netflix.spinnaker.kork.annotations.NonnullByDefault;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
@NonnullByDefault
@Slf4j
public class KubernetesCacheUtils {
class KubernetesCacheUtils {
private final Cache cache;
private final KubernetesSpinnakerKindMap kindMap;
private final KubernetesAccountResolver resourcePropertyResolver;
Expand All @@ -60,115 +61,100 @@ public KubernetesCacheUtils(
this.resourcePropertyResolver = resourcePropertyResolver;
}

public Collection<CacheData> getAllKeys(String type) {
Collection<CacheData> getAllKeys(String type) {
return cache.getAll(type);
}

public Collection<String> getAllKeysMatchingPattern(String type, String key) {
Collection<String> getAllKeysMatchingPattern(String type, String key) {
return cache.filterIdentifiers(type, key);
}

public Collection<CacheData> getAllDataMatchingPattern(String type, String key) {
Collection<CacheData> getAllDataMatchingPattern(String type, String key) {
return cache.getAll(type, getAllKeysMatchingPattern(type, key));
}

public Optional<CacheData> getSingleEntry(String type, String key) {
Optional<CacheData> getSingleEntry(String type, String key) {
return Optional.ofNullable(cache.get(type, key));
}

public Optional<CacheData> getSingleEntryWithRelationships(
String type, String key, String... to) {
return Optional.ofNullable(cache.get(type, key, RelationshipCacheFilter.include(to)));
Optional<CacheData> getSingleEntryWithRelationships(
String type, String key, RelationshipCacheFilter cacheFilter) {
return Optional.ofNullable(cache.get(type, key, cacheFilter));
}

private Collection<String> aggregateRelationshipsBySpinnakerKind(
CacheData source, SpinnakerKind kind) {
return kindMap.translateSpinnakerKind(kind).stream()
.map(g -> source.getRelationships().get(g.toString()))
.filter(Objects::nonNull)
.flatMap(Collection::stream)
.filter(Objects::nonNull)
.collect(Collectors.toList());
/** Gets the data for all relationships of a given Spinnaker kind for a CacheData item. */
ImmutableCollection<String> getRelationshipKeys(
CacheData cacheData, SpinnakerKind spinnakerKind) {
return relationshipTypes(spinnakerKind)
.flatMap(t -> relationshipKeys(cacheData, t))
.collect(toImmutableList());
}

public Collection<CacheData> getTransitiveRelationship(
String from, List<String> sourceKeys, String to) {
Collection<CacheData> sourceData =
cache.getAll(from, sourceKeys, RelationshipCacheFilter.include(to));
return cache.getAll(
to,
sourceData.stream()
.map(CacheData::getRelationships)
.filter(Objects::nonNull)
.map(r -> r.get(to))
.filter(Objects::nonNull)
.flatMap(Collection::stream)
.collect(Collectors.toList()));
/** Gets the data for all relationships of a given type for a CacheData item. */
Collection<CacheData> getRelationships(CacheData cacheData, String relationshipType) {
return getRelationships(ImmutableSet.of(cacheData), relationshipType);
}

public Collection<CacheData> getAllRelationshipsOfSpinnakerKind(
/**
* Gets the data for all relationships of a given Spinnaker kind for a collection of CacheData
* items.
*/
Collection<CacheData> getRelationships(
Collection<CacheData> cacheData, SpinnakerKind spinnakerKind) {
return kindMap.translateSpinnakerKind(spinnakerKind).stream()
.map(kind -> loadRelationshipsFromCache(cacheData, kind.toString()))
return relationshipTypes(spinnakerKind)
.map(kind -> getRelationships(cacheData, kind))
.flatMap(Collection::stream)
.collect(Collectors.toList());
}

public Collection<CacheData> loadRelationshipsFromCache(
CacheData source, String relationshipType) {
return loadRelationshipsFromCache(ImmutableSet.of(source), relationshipType);
}

public Collection<CacheData> loadRelationshipsFromCache(
/** Gets the data for all relationships of a given type for a collection of CacheData items. */
private Collection<CacheData> getRelationships(
Collection<CacheData> sources, String relationshipType) {
List<String> keys =
return cache.getAll(
relationshipType,
sources.stream()
.map(CacheData::getRelationships)
.filter(Objects::nonNull)
.map(r -> r.get(relationshipType))
.filter(Objects::nonNull)
.flatMap(Collection::stream)
.collect(Collectors.toList());

return cache.getAll(relationshipType, keys);
.flatMap(cd -> relationshipKeys(cd, relationshipType))
.collect(toImmutableSet()));
}

/*
* Builds a map of all keys belonging to `sourceKind` that are related to any entries in `targetData`
*/
public Map<String, List<CacheData>> mapByRelationship(
ImmutableMultimap<String, CacheData> mapByRelationship(
Collection<CacheData> targetData, SpinnakerKind sourceKind) {
Map<String, List<CacheData>> result = new HashMap<>();

for (CacheData datum : targetData) {
Collection<String> sourceKeys = aggregateRelationshipsBySpinnakerKind(datum, sourceKind);
ImmutableListMultimap.Builder<String, CacheData> builder = ImmutableListMultimap.builder();
targetData.forEach(
datum ->
getRelationshipKeys(datum, sourceKind)
.forEach(sourceKey -> builder.put(sourceKey, datum)));
return builder.build();
}

for (String sourceKey : sourceKeys) {
List<CacheData> storedData = result.getOrDefault(sourceKey, new ArrayList<>());
storedData.add(datum);
result.put(sourceKey, storedData);
}
/** Returns a stream of all relationships of a given type for a given CacheData. */
private Stream<String> relationshipKeys(CacheData cacheData, String type) {
Collection<String> relationships = cacheData.getRelationships().get(type);
// Avoiding creating an Optional here as this is deeply nested in performance-sensitive code.
if (relationships == null) {
return Stream.empty();
}
return relationships.stream();
}

return result;
/** Given a spinnaker kind, returns a stream of the relationship types representing that kind. */
private Stream<String> relationshipTypes(SpinnakerKind spinnakerKind) {
return kindMap.translateSpinnakerKind(spinnakerKind).stream().map(KubernetesKind::toString);
}

@SuppressWarnings("unchecked")
public <T extends ManifestBasedModel> T resourceModelFromCacheData(
KubernetesV2CacheData cacheData) {
KubernetesHandler getHandler(KubernetesV2CacheData cacheData) {
Keys.InfrastructureCacheKey key =
(Keys.InfrastructureCacheKey) Keys.parseKey(cacheData.primaryData().getId()).get();
KubernetesManifest manifest = KubernetesCacheDataConverter.getManifest(cacheData.primaryData());

KubernetesResourceProperties properties =
resourcePropertyResolver
.getResourcePropertyRegistry(key.getAccount())
.get(manifest.getKind());
KubernetesHandler handler = properties.getHandler();
if (handler instanceof ModelHandler) {
return (T) ((ModelHandler) handler).fromCacheData(cacheData);
} else {
return null;
}
// TODO(ezimanyi): The kind is also stored directly on the cache data; get it from there instead
// of reading it from the manifest.
KubernetesKind kind =
KubernetesCacheDataConverter.getManifest(cacheData.primaryData()).getKind();
return resourcePropertyResolver
.getResourcePropertyRegistry(key.getAccount())
.get(kind)
.getHandler();
}
}
Loading

0 comments on commit 3d90bde

Please sign in to comment.