Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gh17060 gateway single host kube #17557

Merged
merged 50 commits into from
Aug 31, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
a2bb8c9
first treafik config generator implementation
sparkoo Aug 4, 2020
56a3f8e
refactor generating gateway config
sparkoo Aug 4, 2020
1417611
fix TraefikGatewayConfigGeneratorTest
sparkoo Aug 4, 2020
5d3e625
test GatewayServerExposer
sparkoo Aug 4, 2020
9e275c4
nullsafer way how to get port
sparkoo Aug 4, 2020
3cfabf4
create route configmaps in che namespace
sparkoo Aug 6, 2020
e0ec48c
create gateway configmaps in che namespace
sparkoo Aug 12, 2020
641d6ae
gateway prefix path without slash
sparkoo Aug 12, 2020
eb28dce
Merge branch 'master' into gh17060-gatewaySingleHostKube
sparkoo Aug 12, 2020
2fcd61d
cleanup
sparkoo Aug 12, 2020
b9de188
rename GatewayRouterProvisioner to Resolver and create configmaps in …
sparkoo Aug 12, 2020
6e599ec
minor fixes
sparkoo Aug 12, 2020
cc9ed29
format
sparkoo Aug 12, 2020
cf56415
license headers
sparkoo Aug 12, 2020
7b8497b
fix GatewayTlsProvisionerTest
sparkoo Aug 13, 2020
3fc2524
test GatewayRouterResolver
sparkoo Aug 13, 2020
067b3b0
test cheNamespace cleanup
sparkoo Aug 13, 2020
97e3de3
test creating configmaps from gateway configs in KubernetesInternalRu…
sparkoo Aug 13, 2020
de5520c
CheNamespace and CheInstallationLocation tests, refactors
sparkoo Aug 13, 2020
31e8fa0
format, license headers, javadocs,
sparkoo Aug 13, 2020
4035eb3
test generating traefik configuration
sparkoo Aug 13, 2020
f42a4b2
javadocs GatewayRouteConfig
sparkoo Aug 13, 2020
ad2b416
Merge branch 'master' into gh17060-gatewaySingleHostKube
sparkoo Aug 13, 2020
1e163fb
javadocs
sparkoo Aug 13, 2020
a563b34
fix CheInstallationLocation test
sparkoo Aug 17, 2020
721495f
resolve servers only for configmaps that has machinename annotation
sparkoo Aug 17, 2020
62e2314
add X-Forwarded-Proto header to traefik config
sparkoo Aug 18, 2020
66b9d3e
refactor so KubernetesInternalRuntime only creates configmaps
sparkoo Aug 18, 2020
3cd2006
get rid of GatewayRouteConfig
sparkoo Aug 19, 2020
9508e6c
javadocs and refactorings
sparkoo Aug 20, 2020
5e6051d
revert unnecessary change
sparkoo Aug 20, 2020
e30d279
cleanup, refactor, javadocs of GatewayRouteProvisioner
sparkoo Aug 20, 2020
8480ba9
cleanup, better exception messages
sparkoo Aug 20, 2020
183d58c
javadocs and minor refactorings
sparkoo Aug 20, 2020
8d796ae
javadocs and minor refactors
sparkoo Aug 20, 2020
d5cc155
remove singlehost from openshift for now
sparkoo Aug 20, 2020
a82b883
verify that gatewayRouterProvisioner is called
sparkoo Aug 20, 2020
60b626a
fix test checking proper configmaps are created in che namespace
sparkoo Aug 20, 2020
e4db204
CheNamespace checks configmaps for proper annotations + fix and add t…
sparkoo Aug 20, 2020
49375ae
test GatewayRouterProvisioner
sparkoo Aug 20, 2020
69a03fb
test GatewayTlcProvisioner
sparkoo Aug 20, 2020
28c8fbe
test GatewayServerExposer
sparkoo Aug 20, 2020
fe3a96c
test Traefik config generator
sparkoo Aug 20, 2020
ee77b6b
revert unnecessary change
sparkoo Aug 20, 2020
83e4774
license headers
sparkoo Aug 20, 2020
c77a4ee
fix NPE when checking object to create in che installation namespace
sparkoo Aug 20, 2020
8f1c14e
javadocs
sparkoo Aug 20, 2020
4d6337c
rename CheKubernetesClientFactory -> CheServerKubernetesClientFactory…
sparkoo Aug 21, 2020
ec1201d
expose paths with ending slash
sparkoo Aug 21, 2020
fe6e99e
Merge branch 'master' into gh17060-gatewaySingleHostKube
sparkoo Aug 31, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,13 @@
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.DefaultHostExternalServiceExposureStrategy;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.ExternalServerExposer;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.ExternalServiceExposureStrategy;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.GatewayRouteConfigGenerator;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.GatewayServerExposer;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.IngressServerExposer;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.IngressServiceExposureStrategyProvider;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.MultiHostExternalServiceExposureStrategy;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.SingleHostExternalServiceExposureStrategy;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.external.TraefikGatewayRouteConfigGenerator;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.secure.SecureServerExposer;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.secure.SecureServerExposerFactory;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.secure.SecureServerExposerFactoryProvider;
Expand Down Expand Up @@ -161,6 +163,7 @@ protected void configure() {
bind(ExternalServiceExposureStrategy.class)
.toProvider(IngressServiceExposureStrategyProvider.class);

bind(GatewayRouteConfigGenerator.class).to(TraefikGatewayRouteConfigGenerator.class);
MapBinder<WorkspaceExposureType, ExternalServerExposer<KubernetesEnvironment>>
exposureStrategies =
MapBinder.newMapBinder(
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Copyright (c) 2012-2018 Red Hat, Inc.
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
* which is available at https://www.eclipse.org/legal/epl-2.0/
*
* SPDX-License-Identifier: EPL-2.0
*
* Contributors:
* Red Hat, Inc. - initial API and implementation
*/
package org.eclipse.che.workspace.infrastructure.kubernetes.server.external;

import java.util.Map;

/**
* Generates config for single external server that we want to expose in the Gateway.
*
* <p>Implementation provides configuration for specific Gateway technology (e.g., Traefik).
*/
public interface GatewayRouteConfigGenerator {

/**
* Generates content of configuration for service, defined by passed parameters, that should be
* exposed by the Gateway. Returned {@code Map<String, String>} will be used as a value of
* ConfigMap.
*
* <p>Implementation must ensure that Gateway configured with returned content will route the
* requests on {@code path} into {@code serviceUrl}. Also it must strip {@code path} from request
* url.
*
* @param name name of the service
* @param serviceUrl url of service we want to route to
* @param path path to route and strip
* @return full content of configuration for the service
*/
Map<String, String> generate(String name, String serviceUrl, String path);
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,29 +11,127 @@
*/
package org.eclipse.che.workspace.infrastructure.kubernetes.server.external;

import com.google.common.collect.ImmutableMap;
import io.fabric8.kubernetes.api.model.ConfigMap;
import io.fabric8.kubernetes.api.model.ConfigMapBuilder;
import io.fabric8.kubernetes.api.model.ServicePort;
import java.util.HashMap;
import java.util.Map;
import javax.inject.Inject;
import org.eclipse.che.api.core.model.workspace.config.ServerConfig;
import org.eclipse.che.api.workspace.server.model.impl.ServerConfigImpl;
import org.eclipse.che.commons.annotation.Nullable;
import org.eclipse.che.workspace.infrastructure.kubernetes.Annotations;
import org.eclipse.che.workspace.infrastructure.kubernetes.environment.KubernetesEnvironment;
import org.eclipse.che.workspace.infrastructure.kubernetes.server.KubernetesServerExposer;

/**
* Uses Traefik gateway configured with ConfigMaps to expose servers.
*
* <p>TODO: implement
* Uses gateway configured with ConfigMaps to expose servers.
*
* @param <T> type of environment
*/
public class GatewayServerExposer<T extends KubernetesEnvironment>
sparkoo marked this conversation as resolved.
Show resolved Hide resolved
implements ExternalServerExposer<T> {

private static final Map<String, String> GATEWAY_CONFIGMAP_LABELS =
ImmutableMap.<String, String>builder()
.put("app", "che")
.put("role", "gateway-config")
.build();

private final ExternalServiceExposureStrategy strategy;
private final GatewayRouteConfigGenerator gatewayConfigGenerator;

@Inject
public GatewayServerExposer(
ExternalServiceExposureStrategy strategy,
GatewayRouteConfigGenerator gatewayConfigGenerator) {
this.strategy = strategy;
this.gatewayConfigGenerator = gatewayConfigGenerator;
}

/**
* Exposes service port on given service externally (outside kubernetes cluster) using the Gateway
* specific configurations.
*
* @param k8sEnv Kubernetes environment
* @param machineName machine containing servers
* @param serviceName service associated with machine, mapping all machine server ports
* @param serverId non-null for a unique server, null for a compound set of servers that should be
* exposed together.
* @param servicePort specific service port to be exposed externally
* @param externalServers server configs of servers to be exposed externally
*/
@Override
public void expose(
T k8sEnv,
String machineName,
@Nullable String machineName,
String serviceName,
String serverId,
ServicePort servicePort,
Map<String, ServerConfig> externalServers) {
throw new UnsupportedOperationException("Not implemented yet.");

if (serverId == null) {
// this is the ID for non-unique servers
serverId = servicePort.getName();
}
ConfigMap traefikGatewayConfig =
generateTraefikConfig(machineName, serviceName, serverId, servicePort, externalServers);

k8sEnv.getConfigMaps().put(traefikGatewayConfig.getMetadata().getName(), traefikGatewayConfig);
}

private ConfigMap generateTraefikConfig(
String machineName,
String serviceName,
String serverId,
ServicePort servicePort,
Map<String, ServerConfig> serversConfigs) {
final String serverName = KubernetesServerExposer.makeServerNameValidForDns(serverId);
final String name = createName(serviceName, serverName);
final String serviceClusterUrl = createServiceUrl(serviceName, servicePort);
final String path = ensureEndsWithSlash(strategy.getExternalPath(serviceName, serverName));
final Map<String, String> configData =
gatewayConfigGenerator.generate(name, serviceClusterUrl, path);
final Map<String, String> annotations = createAnnotations(serversConfigs, path, machineName);

return new ConfigMapBuilder()
.withNewMetadata()
.withName(name)
.withLabels(GATEWAY_CONFIGMAP_LABELS)
.withAnnotations(annotations)
.endMetadata()
.withData(configData)
.build();
}

private String ensureEndsWithSlash(String path) {
return path.endsWith("/") ? path : path + '/';
}

private String createName(String serviceName, String serverName) {
return serviceName + "-" + serverName;
}

private String createServiceUrl(String serviceName, ServicePort servicePort) {
return "http://"
+ serviceName
+ ".che.svc.cluster.local:"
sparkoo marked this conversation as resolved.
Show resolved Hide resolved
+ servicePort.getTargetPort().getIntVal().toString();
sparkoo marked this conversation as resolved.
Show resolved Hide resolved
}

private Map<String, String> createAnnotations(
Map<String, ServerConfig> serversConfigs, String path, String machineName) {
Map<String, ServerConfig> configsWithPaths = new HashMap<>();
for (String scKey : serversConfigs.keySet()) {
ServerConfigImpl sc = new ServerConfigImpl(serversConfigs.get(scKey));
sc.setPath(path);
configsWithPaths.put(scKey, sc);
}

return Annotations.newSerializer()
.servers(configsWithPaths)
.machineName(machineName)
.annotations();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
/*
* Copyright (c) 2012-2018 Red Hat, Inc.
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
* which is available at https://www.eclipse.org/legal/epl-2.0/
*
* SPDX-License-Identifier: EPL-2.0
*
* Contributors:
* Red Hat, Inc. - initial API and implementation
*/
package org.eclipse.che.workspace.infrastructure.kubernetes.server.external;

import static com.fasterxml.jackson.dataformat.yaml.YAMLGenerator.Feature.WRITE_DOC_START_MARKER;

import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator;
import java.io.IOException;
import java.io.StringWriter;
import java.util.Collections;
import java.util.Map;

/**
* Traefik configuration for single route looks like this (values in {} are parameters of {@link
* GatewayRouteConfigGenerator#generate(String, String, String)} method):
*
* <pre>
* http:
* routers:
* {name}:
* rule: "PathPrefix(`{path}`)"
* service: {name}
* middlewares:
* - "{name}"
* priority: 100
* services:
* {name}:
* loadBalancer:
* servers:
* - url: '{serviceUrl}'
* middlewares:
* {name}:
* stripPrefix:
* prefixes:
* - "{path}"
* </pre>
*/
public class TraefikGatewayRouteConfigGenerator implements GatewayRouteConfigGenerator {
sparkoo marked this conversation as resolved.
Show resolved Hide resolved

/**
* Generates Traefik specific configuration for single service.
*
* @param name name of the service
* @param serviceUrl url of service we want to route to
* @param path path to route and strip
* @return traefik service route config
*/
@Override
public Map<String, String> generate(String name, String serviceUrl, String path) {
StringWriter sw = new StringWriter();

try {
YAMLGenerator generator =
Copy link
Member Author

@sparkoo sparkoo Aug 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about using YAMLGenerator. We could use ObjectMapper, but then we would need to create all objects in configuration yaml or use multiple levels of Map<String, Map<String, Map<....
So I don't think it's so bad... Or any better idea?

YAMLFactory.builder().disable(WRITE_DOC_START_MARKER).build().createGenerator(sw);

generator.writeStartObject();
generator.writeFieldName("http");
generator.writeStartObject();

generator.writeFieldName("routers");
generateRouters(generator, name, path);

generator.writeFieldName("services");
generateServices(generator, name, serviceUrl);

generator.writeFieldName("middlewares");
generateMiddlewares(generator, name, path);

generator.flush();

return Collections.singletonMap(name + ".yml", sw.toString());
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException(e);
sparkoo marked this conversation as resolved.
Show resolved Hide resolved
}
}

/**
* generates Routers part of Traefik config
*
* <pre>
* {name}:
* rule: "PathPrefix(`{path}`)"
* service: "{name}"
* middlewares:
* - "{name}"
* priority: 100
* </pre>
*/
private void generateRouters(YAMLGenerator generator, String name, String path)
throws IOException {
generator.writeStartObject();
generator.writeFieldName(name);
generator.writeStartObject();
generator.writeFieldName("rule");
generator.writeString("PathPrefix(`" + path + "`)");
generator.writeFieldName("service");
generator.writeString(name);
generator.writeFieldName("middlewares");
generator.writeStartArray();
generator.writeString(name);
generator.writeEndArray();
generator.writeFieldName("priority");
generator.writeNumber(100);
generator.writeEndObject();
generator.writeEndObject();
}

/**
* generates Services part of Traefik config
*
* <pre>
* {name}:
* loadBalancer:
* servers:
* - url: "{serviceUrl}"
* </pre>
*/
private void generateServices(YAMLGenerator generator, String name, String serviceUrl)
throws IOException {
generator.writeStartObject();
generator.writeFieldName(name);
generator.writeStartObject();
generator.writeFieldName("loadBalancer");
generator.writeStartObject();
generator.writeFieldName("servers");
generator.writeStartArray();
generator.writeStartObject();
generator.writeFieldName("url");
generator.writeString(serviceUrl);
generator.writeEndObject();
generator.writeEndArray();
generator.writeEndObject();
generator.writeEndObject();
generator.writeEndObject();
}

/**
* generates Middlewares part of Traefik config
*
* <pre>
* {name}:
* stripPrefix:
* prefixes:
* - "{path}"
* </pre>
*/
private void generateMiddlewares(YAMLGenerator generator, String name, String path)
throws IOException {
generator.writeStartObject();
generator.writeFieldName(name);
generator.writeStartObject();
generator.writeFieldName("stripPrefix");
generator.writeStartObject();
generator.writeFieldName("prefixes");
generator.writeStartArray();
generator.writeString(path);
generator.writeEndArray();
generator.writeEndObject();
generator.writeEndObject();
generator.writeEndObject();
}
}
Loading