Introduce EnterpriseSearch CRD #2688

sebgl · 2020-03-10T16:55:07Z

Introduce the EnterpriseSearch CRD and controller in ECK.

It seems to be working fine so far but has not been extensively tested.
To be used, it requires the image field to be set with a valid Docker image.

This is obviously work in progress, since:

parts of the controller should be refactored to share common code with APMServer and Kibana
most of the association controller should be refactored to share common code with APMServer and Kibana
there are 0 unit tests (mostly because I wanted to go fast in the prototype phase, and because many unit tests should be written along with the refactoring above)
there are 0 E2E tests
it is missing a few features (podDisruptionBudget, secret config refs, version upgrade handling, etc.)

Bootstrap the Enterprise Search CRD & controller

This is cleaner, more similar to what we do for other resources, and avoids leaking secrets in Pod environment variables. Note that Pods do not get automatically rotated when config changes so far.

This is heavily based on the APMServer association controller. Would largely benefit from some common refactoring in the future.

I don't know why this isn't there in the first place. I plan to investigate in the future, but for now this is required to perform status updates.

Setup the EnterpriseSearch - Elasticsearch association controller

We want Pods to be rotated with the new config.

Let's query /swiftype-app-version which seems to be the default out there. The initial bootstrap is pretty slow so I set an InitialDelaySeconds of 60sec.

I *think* this user can end up being useful if Elasticsearch is down hence the native realm cannot be used.

This fixes CRD generation following a rebase on master.

Generate self-signed certificates for Enterprise Search and enable TLS encryption by default. This can be disabled in the EnterpriseSearch spec, and user-provided certificates can be used instead.

Depending on the TLS spec, let's use the correct protocol: http or https.

Automatically rotate the Pod if: - the ent search config changes - the ent search TLS certs change - the referenced Elasticsearch certs change This is done through hashing them all and using that hash as a label in the podTemplate.

* Minor cosmetic housekeeping * goimports files * Add missing license headers * Fix linter warnings * Add api docs

jonasll · 2020-03-10T17:23:55Z

config/crds/patches/kustomization.yaml

@@ -42,3 +42,10 @@ patchesJson6902:
      kind: CustomResourceDefinition
      name: kibanas.kibana.k8s.elastic.co
    path: apm-kibana-patches.yaml
+  # custom patches for EnterpriseSearch


Proper name here is Enterprise Search, two words.

This really refers to the EnterpriseSearch CRD (the technical implementation), I'd be in favor of keeping it as is. Also this is def not a user-facing file.

jonasll · 2020-03-10T17:25:41Z

docs/reference/api-docs.asciidoc

+[id="{anchor_prefix}-github-com-elastic-cloud-on-k8s-pkg-apis-enterprisesearch-v1beta1-enterprisesearch"]
+=== EnterpriseSearch 
+
+EnterpriseSearch represents an Enterprise Search resource in a Kubernetes cluster.


Can we make sure to outline that EnterpriseSearch here is a technical way of referring to Enterprise Search, the product? Naming is extra sensitive and we need make sure that we perhaps wrap it in EnterpriseSearch a pre wrap or something.

This doc is auto-generated from the following go code comment:

// EnterpriseSearch represents an Enterprise Search resource in a Kubernetes cluster. type EnterpriseSearch struct {

There's a go linter rule that specifies each commented struct must start with // <structName> ..., so it's not easy to change it to eg. <pre>EnterpriseSearch</pre>.

I can change it to something like EnterpriseSearch is a Kubernetes CRD to represent Enterprise Search. if that makes more sense?

This is the CRD API doc, we'll definitely have more user-facing docs in the future in which it's important we make that distinction clear 👍

jonasll · 2020-03-10T17:26:15Z

docs/reference/api-docs.asciidoc

+[id="{anchor_prefix}-github-com-elastic-cloud-on-k8s-pkg-apis-enterprisesearch-v1beta1-enterprisesearchlist"]
+=== EnterpriseSearchList 
+
+EnterpriseSearchList contains a list of EnterpriseSearch


Same comment on communicating tech-level items vs. product name.

pebrc

LGTM I like the flatter package structure compared to our other controller implementations. I have more or less ignored all TODOs and the whole association controller and the fact that there are no tests in the assumption we will address this in follow up PRs. So I 👍 for merging as is and addressing any issues after that.

pkg/apis/enterprisesearch/v1beta1/enterprisesearch_types.go

pkg/controller/enterprisesearch/enterprisesearch_controller.go

pkg/controller/enterprisesearch/state.go

pkg/controller/entsearchassociation/association_controller.go

sebgl added 20 commits February 17, 2020 14:07

Add Enterprise Search CRD

207376e

Add the Enterprise Search controller - minimal deployment ok

2b984a5

Add a working minimal Enterprise Search sample

d4dae4f

Merge pull request #1 from sebgl/ent-search-bootstrap

d8fba95

Bootstrap the Enterprise Search CRD & controller

Reconcile a configuration secret instead of env vars (#2)

e286001

This is cleaner, more similar to what we do for other resources, and avoids leaking secrets in Pod environment variables. Note that Pods do not get automatically rotated when config changes so far.

Setup the EnterpriseSearch - Elasticsearch association controller

3053c7e

This is heavily based on the APMServer association controller. Would largely benefit from some common refactoring in the future.

Patch the EnterpriseSearch CRD to fix the subresources

77544ad

I don't know why this isn't there in the first place. I plan to investigate in the future, but for now this is required to perform status updates.

Merge pull request #3 from sebgl/es-association

22adc70

Setup the EnterpriseSearch - Elasticsearch association controller

Label Pods with a hash of the config for auto rotation (#4)

612a409

We want Pods to be rotated with the new config.

Reuse the JAVA_OPTS const (#5)

c8bd16a

Setup a readiness probe for smooth deployment rotation (#6)

857325e

Let's query /swiftype-app-version which seems to be the default out there. The initial bootstrap is pretty slow so I set an InitialDelaySeconds of 60sec.

Generate the default enterprise_search user in a secret (#7)

2619476

I *think* this user can end up being useful if Elasticsearch is down hence the native realm cannot be used.

Strip out private Docker image from default sample (#8)

09c313c

Merge branch 'master' into enterprise-search

4921eb2

Fix EnterpriseSearch CRD generation (#9)

1f30227

This fixes CRD generation following a rebase on master.

Use self-signed TLS certificates by default (#10)

d8160ec

Generate self-signed certificates for Enterprise Search and enable TLS encryption by default. This can be disabled in the EnterpriseSearch spec, and user-provided certificates can be used instead.

Switch http/https protocol in readiness probe and config (#11)

4155954

Depending on the TLS spec, let's use the correct protocol: http or https.

Inject a hash of TLS certs for pod rotation (#12)

b5031cd

Automatically rotate the Pod if: - the ent search config changes - the ent search TLS certs change - the referenced Elasticsearch certs change This is done through hashing them all and using that hash as a label in the podTemplate.

Cleanup the code files (#13)

775ddbf

* Minor cosmetic housekeeping * goimports files * Add missing license headers * Fix linter warnings * Add api docs

Merge branch 'master' into enterprise-search

08ad2d0

sebgl added >feature Adds or discusses adding a feature to the product v1.1.0 labels Mar 10, 2020

jonasll suggested changes Mar 10, 2020

View reviewed changes

sebgl added 2 commits March 11, 2020 11:14

Struct comments improvements

bfcd4cf

Regenerate CRDs and api docs

d46f6a4

pebrc approved these changes Mar 12, 2020

View reviewed changes

sebgl added 4 commits March 12, 2020 14:54

Fixes from PR review

35fbf49

Merge branch 'master' into enterprise-search

1aa7e3c

Rely on the global scheme.Scheme

8103630

Setup RBAC for Enterprise Search

f7087a6

sebgl merged commit a1bd830 into master Mar 13, 2020

anyasabo added v1.2.0 and removed v1.1.0 labels Mar 31, 2020

sebgl deleted the enterprise-search branch May 25, 2020 07:08

pebrc added the release-highlight Candidate for the ECK release highlight summary label Jun 4, 2020

charith-elastic changed the title ~~Add the EnterpriseSearch CRD and controller~~ Introduce EnterpriseSearch CRD Jun 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce EnterpriseSearch CRD #2688

Introduce EnterpriseSearch CRD #2688

sebgl commented Mar 10, 2020 •

edited

Loading

jonasll Mar 10, 2020

sebgl Mar 11, 2020

jonasll Mar 10, 2020

sebgl Mar 11, 2020

jonasll Mar 10, 2020

pebrc left a comment

Introduce EnterpriseSearch CRD #2688

Introduce EnterpriseSearch CRD #2688

Conversation

sebgl commented Mar 10, 2020 • edited Loading

jonasll Mar 10, 2020

Choose a reason for hiding this comment

sebgl Mar 11, 2020

Choose a reason for hiding this comment

jonasll Mar 10, 2020

Choose a reason for hiding this comment

sebgl Mar 11, 2020

Choose a reason for hiding this comment

jonasll Mar 10, 2020

Choose a reason for hiding this comment

pebrc left a comment

Choose a reason for hiding this comment

sebgl commented Mar 10, 2020 •

edited

Loading