Merge istio/installer and istio/operator into istio/istio #20106

* update installer sha. * update vfs. * update testdata.

* Support Operator e2e * Support Operator e2e * Load Images for kind nodes * Load Images for kind nodes * Load Images for kind nodes * Load Images for kind nodes * Review Comments * Review Comments

* Update readme for controller local debugging * Address comment

As per YAML specifications, YAML documents should begin with `---` and conclude with `...`. Kubernetes accepts pretty sloppy yaml and just requires a `---` although other tools, such as `yq` require properly formatted YAML in order to operate correctly.

…io#469) * auto on and make run demo * make run-build * run with container BUILD_WITH_CONTAINER=1 make gen

* WIP: Refine e2e implementation This is a refinement based upon the prior PRs in this area. Depends-On: istio/tools#483 * Use proper image with jq

* Update to latest common-files * Address reviewer comments.

* Turn off DNS certificate provisioning * Run "make gen" and include the changes * make gen after unset hub and tag

* Update README * Updates * Comments * Remove local build instructions * Add instructions for local running * Remove redundant readme

* update sha for installer to include automtls change * make gen with container build * manual update the default.yaml, ci not failing dueo to configmap * change default.yaml * refresh golden * fix licenses

* update go mod file * update hub and tag to use istio.io/pkg DockerBuildInfo * add ldflags tests for hub and tag * set if condition to check unknow string instead of empty string * update go.mod go.sum * fix lint error * run make gen to update go.sum * add licenses update

* Add remote profile * Update assets

…, tolerations: '[] ->') output (istio#486)

* Prune citadel secrets * Add istio.io/dns-key-and-cert into secret map

* Add statsv2 install option * Address comment * Address comment * Add make gen update

* Turn off DNS certificate provisioning * make update-goldens * make gen-check * Update installer.sha

…s. (istio#476) * Exclude the overridden values when comparing the old with the new ones. * Add --show-overrides flag to show all changed values. * Fix lint. Remove unused recursive params. Reorder flags. * Remove empty lines between imports. * Fix lint, reording imports. * Move genOverlayICPS() to a better place, reducing imports in upgrade.go. * Use the untranslated input as a mask * Improve isPathInTree() func based on PR comments. * Renaming test struct with more readable fields. * Remove --show--overrides as it is a common use case to have seprate flag, and we already have method (profile diff) to archieve it. * Use the existing name.GetFromTreePath() instead of duplicating tree traversals. * Fix rebasing parameter change. * Improve tests, and fix a bug for not matching the leaf path from overrides.

* Fix cryptic messages. * Improve the command examples.

* use force flag for set override * Update test * fix conflict

* Match CRD files in istio/istio * add a newline * add gen files

* Fix duplicate field * update gen

And remove values_gke.yaml Closes istio#18576

* Update installer SHA * Update golden test files

* Fix reverse translate issue * update test result * Fix typo * Fix conflict

This is not used and adds confusion

* Add reminder message if user fail with values schema * Address comment

These are not being applied properly, so cannot access telemetry components when mtls is enabled

* add permissions to be able to view logs in kiali console * `make gen` results

* Fix enablement path for CNI * Fix tests

) * Add multicluster remote gateway install YAML for mesh command * Fixes to values * Update assets * Fix meshNetworks path * Update assets

* Use global trust domain * make gen * Fix stuff

* Add CoreDNS feature and component * Update values

…rivileges (istio#492) * add istio-reader clusterrole to access remote clusters with reduced privileges This also fixes an issue where mixer doesn't have sufficient access to read replicationcontrollers in remote clusters. * make gen

* update hub and tag examples * fix lint * update docs * lint * move up grafana to latest * update version * Revert "update version" This reverts commit 5cba0da9f0fdd3ac05ef4de0054872504ae3007a. * update version

* Kill unneeded/broken DestinationRules See istio#18654 This is especially bad because the config will be reconciled, so even if manually fixed will still break * api-server only when mtls enabled * gen

* automtls aspect in the enable mesh tls. * make gen

Fixes istio#18676

* Align injection template to fix SDS This may not fully fix SDS but is part of what is required * Fix mesh config * regen

…, but not supported by GetFromTreePath() (istio#532) * Fix GetFromTreePath issues when input path is longer than tree depth. * Refactor the tree traversal code. * Remove unreachable code. * Add complex tests, fix an issue of recursion on list node. * Add one more negtive test.

* Add a few more values as typed config * regen

* move to correct block... * move around the comment as well

* add missing istio-reader service account and binding * make gen

* Remove demo-auth profile Fixes istio#18646 * Really kill the demo-auth

* Sync installer * Fix some values changes * update goldens

* Update jaeger yaml to include the jaeger-agent svc Per slack discussions and istio#18658. * Update tracing.gen.yaml * Update tracing.gen.yaml * Update tracing.gen.yaml

User is supposed to create it. Closes istio#18763

* Add missing remote resources * Move services * Update configmap * Add default value * Fix gen and lint * Rename crds chart to base * Missed some renames * Make gen * Missing yaml * Update generated files using container build * Move clusterrole * Manual fix to genpatch

* Enable ingress in remote profile * Fix gencheck

This now directly matches istio/istio. The problem here is it is impossible to override the default data source, and you can only have one default, so there are conflicts.

* Moving the stdout output to l.logAndError(), also change a logAndOutput into l.logAndError(). * Also improve the output from kube client.

* Update envoy metrics service and access log service config This PR migrates to the newer flags for the envoy metrics service and access log service config that support setting up the TLS context and keepalive config. * Fix citadel.yaml template

* [kiali] issue-18798 fix some wrong yaml. Another (trivial) change is that this alphabetizes a couple settings to make comparison with the old helm charts easier. * gen check results

This was out of sync with the values.yaml from installer

* Update reverse translation for coredns * update golden

* Fix podDNSSearchNamespaces validation issue. * Re-run make gen * Update with comments, adjust order of the field PodDNSSearchNamespaces * Fix failed tests.

istio#528) * move the istio-reader serviceaccount and binding to the base directory * make gen

Without this change, there are 10,000 lines logged for `manifest generate` which is way to noisy. These should be debug logging only, and only useful user facing logs on info.

* Modify prom operator root path * Make gen

This will help with issues where things are not reproducible

* Updated mixer config to bring in latest changes Updated to get to 3194864 from istio/istio master branch. * Updated after running make gen

* Update galley-dashboard.json To match changes to istio/istio in istio#18757 * Update grafana.gen.yaml * Update istio-grafana.gen.yaml

* Add fields to values schema * Make gen * Add field descriptions * Make gen

* Add health check to Citadel. * Small fix. * Small fix.

* Sync labels with istio/istio I don't fully understand the details of it, but Kubernetes gets mad when you change labels sometimes, our tools depend on some labels (like istioctl), and users may have tools/scripts that do as well. For consistency, make these the same as istio/istio * Fix mixer * Fix mixer * Fix injector

* Update common files * Update installer SHA * Remove protoc_gen_k8s_support_plugins It is causing errors like this: ``` istio.io/operator/pkg/apis/istio/v1alpha1 imports github.com/gogo/protobuf/protobuf/google/protobuf: module github.com/gogo/protobuf@latest found (v1.3.1), but does not contain package github.com/gogo/protobuf/protobuf/google/protobuf common/Makefile.common.mk:67: recipe for target 'tidy-go' failed make: *** [tidy-go] Error 1 make: *** [Makefile:111: gen-check] Error 2 ``` * make gen-check * update golden files

* Installation option doc cleanup * regen

istio#577) * Create namespace before namespaced resources in Base component, fixing namespaces not found errors. * Fix output. * Wait for NS ready; Move waitForCrd out from err branch. * Remove duplicate code blocks. Replace nonNsCRDKindObjects() with objectsNotInLists(). * Remove the caller of nonNsCRDKindObjects

* Move gateway and coredns enablement in sample config * Update assets

* update for telemetryv2 * add gen * add gen

From pilot.telemetry.v2

* Telemetry v2 off by default * Gen * gen again

* Move telemetry out of pilot * Update

* add namespace for envoyfilter. * move imported config to template. * change to root config namespace. * fix gen check.

The current command does not work. It results in "invalid version".

* Use gobuild.sh to build operator * Make gen * Restore deleted line * Add comment for posterity, convert mesh target also

* update migrate script * Address comment * Address comment * Address comment

* Remove gen * Fix noauth * run build * Fix ingress * final fixes and gitignore * Deflake noauth * Fix noauth

* Update crd-all.gen.yaml * Fix crd * Remove invalid stuff

This unblocks https://github.com/istio/istio/pull/18991/files

* update minimal profile * update gen * update goldens

* remove egress sds * Remove extra role and rolebinding

…stio#623) * Ignore a warning in kubectl apply when namespace is already created. * Add TODO and k8s issue number. * Fix PR comments.

Signed-off-by: Gary Brown <gary@brownuk.com>

* sync values types * add more missing fields * mark k8s fields as deprecated

* Remove original cr yaml from logging * fix lint

See istio#19043 for prior discussion. Both pilot and galley set max client ages in order to promote rebalancing, but the default `stream_idle_timeout` value of `5m` is lower than the default max client age of `30m`. This is causing reconnects to occur more frequently than intended. This change disables the idle timeout entirely, giving control back to pilot and galley over when the client connections are killed. Mixer does not seem to set the max client age, so the `stream_idle_timeout` is only being set for the static listener for outbound connections to galley.

* Improve the user facing outputs. * Fix lint

* Clean up code * Lint

* Support SSE for galley config * Fix lint

fixes istio#18827 backport of istio#18989

* update telemetry v2 config * fix

Several people have complained about failed tests during `make update-goldens`. This PR turns on the verbose flag to aid in debugging both there, as well as `make test`.

Fixes: istio#19227 This PR mirrors: istio#19231 I am not confident this change is correct, however, it does revert back to the original 1.3 behavior. I don't understand why we want two services (zipkin and tracing). Should not tracing be sufficient? If we must retain zipkin and we want tracing to run on port 9411 (vs what it has been running on which is port 80...) zipkin must run on some other port.

Signed-off-by: Jon Yucel <jon.yucel@getcruise.com>

* Update common files * Use same image as CI so generators work correctly

This gives us more control of the output. In cases where we supply stdOut and stdErr (test cases), if we abort the execution upon an error, the caller does not have the chance to collect the output and print to user. By using `RunE` instead of `Run` we don't need to abort the execution and therefore break the workflow. By returning an error the caller can properly handle the error.

* Remove unnecessary env variables * update goldens

* Add unit test and fix for ICP.DeepCopy() DeepCopy() always failed with an error, which was only logged and wasn't handled properly. The error occured when unmarshalling CreatedTimestamp, which meant that all remaining fields were never unmarshalled. * Add copyright header * Update go.mod * Add additional test and fix import formatting

* Export Logger and NewLogger * Fix test error * Fix lint error

For istio#17158

This should resolve the jaeger regression around ports. Additionally other features have been added to the installer. Fixes: istio#19227

…tio#646) * Disable leader election when LEADER_ELECTION_NAMESPACE is not set Debugging the controller is practically impossible, since it loses the leader election lock when it's suspended for too long. This causes the process to be terminated. * Improve warning

* Fix logging on windows /dev/null does not exist on windows. Instead, just don't log at all. Also clean up some file logging stuff, this is dead code. * Fix split path * Fix license * Fix format

* Add cluster domain for policy and telemetry * address comment

Previously, the controller read the ICP object twice - first as an Unstructured object, and then from the local cache. This was a workaround around the issue where ICP.DeepCopy(), which is called when you get the object from the cache, didn't work properly. With DeepCopy now fixed, we can get the ICP from the cache and there's no need to also get it from the API server.

…abled flags (istio#578) * more granular stackdriver control with logging.enabled and metrics.enabled flags Signed-off-by: Jon Yucel <jon.yucel@getcruise.com> * set the default values for stackdriver logging and metrics to true

Manual CP of istio#18976

…stio#659) Replaces low-level array handling code with apimachinery's util/sets. Also fixes panic that can occur if there's a conflict during finalizer removal. The code re-reads the ICP, but doesn't re-check if the finalizer is still present in the new ICP. If it's missing, finalizerIndex is -1 and thus finalizers[:finalizerIndex] will panic.

* Operator init and remove commands * Build fix * Restore deleted files * Review comments * Fix merge conflicts * Regen assets * Lint

* Fix reconcile conflict error * Fix lint error

* Add base feature and component and empty profile * Update goldens * Update tests * Update goldens

This copies from istio/istio

Signed-off-by: Juraci Paixão Kröhling <jpkroehling@redhat.com>

* add stackdriver envoyfilter for telemetryv2 * update fields name * add envoyfilters for different versions * longer waiting time for test * update default * lint issue

* Add nil check when determining reflect.Kind of types. Added test. * Fix comment

…o#672)

* Fixes for mixer mesh config * Policy disabled will remove mixerCheckServer * decouple v2 and v1. Support telemetry v2 with no mixer at all * disable telemetry v2

)

…nto different goroutines. (istio#679)

* fix(templates): Correct OWNER urls. This PR is meant to cover fixes such as: - istio#18485 - istio#19513 * Update injection-template.yaml * Update deployment.yaml

…tio#685)

…d target install. (istio#671) * upgrade uses ICPS to compare changes between current install and target install. * Remove values-based upgrade check. * Skip the validation for the old ICPS generation. * Remove unsed function readValuesFromInjectorConfigMap(). * Address PR comments and replace values with ICPS. * Fix lint.

* watch Istio resources changes in istio-operator * fix test error * Review Comments * Fix e2e test failure * Fix e2e test failure * Fix e2e failure * debug e2e failure * debug operator e2e * debug e2e * debug e2e * debug e2e * remove e2e debug change

We cannot be cloning from github blindly, as this will break our private security builds. This PR adds an option to pass an INSTALLER_DIR option to the scripts and skip the cloning. Additionally, do not run vfsgen when you do docker build. This should be an explicit step, otherwise the code magically changes during a build and you spend all day debugging it.

Signed-off-by: Dinar Valeev <k0da@opensuse.org>

…lPlaneSecurity (istio#697) * add a feature validation * remove cmm.

* remove old mixer types * merge common pilot config types into a single block

* Sync config map with master * Update values.yaml

* Update installer SHA * Update again

* Upgrade libraries to Kubernetes 1.17 * Add licenses * remove legacy kinds from pruning * Reduce what we watch * Fix lint

* Add the istiod SA and Cluster binding. Using kubect apply -k ...cluster will support both istio and istiod in same cluster. * Initial yaml for istiod. For now kustomize only - no need for helm ( we may add a small helm wrapper later) * Initial merging of istiod enablement * Remove old istiod * Fix name * Forgot to add 2 files * Add the settings for SDS/secure token * Fix knative test * Fix the prow kind * Added few comments and remove one more 'values' for istiod

* Update the version for operator; Try version from DockerInfo.Tag as operator binary version; Use the latest version if target version does not exist. * Fix version/version_test. * Improve tests.

* update of migrate logic * fix test * remove profile related change * update gen * fix cni path * add new test for overlay

…o#707) * Fallback to previous versions when failing to create translator and reversed translator. * Replace a magic number with constant, add comments for it.

I have test both of these out

* Add missing auto-mtls env var This is causing failed tests when using operator, because ingress starts without waiting for certs. See istio#19746. This matches istio/istio. * Disable for noauth * Fix knative

* Sync installer * Update value * Exclude injector config * Add comment * Update docs

…o Tag. (istio#710) * Update operator version to 1.5.0; support setting version by BuildInfo Tag. * Add Version Unmarshal back. * Fix lint.

Signed-off-by: Chun Lin Yang <clyang@cn.ibm.com>

* Fix for disablePolicyCheck * controlPlaneSecurity off

…ecks (istio#623) Signed-off-by: Chun Lin Yang <clyang@cn.ibm.com>

Signed-off-by: Chun Lin Yang <clyang@cn.ibm.com>

Fixes prometheus/prometheus#6512

…stio#722)

* Fix args.wait not being used. * Fix import.

Fixes istio#17318

* Add TODO comments of version 1.5 for translate * make fmt.

* Fixes for istiod * Fix knative namespace * Fix clusterrole for cert watch * Fix lint

* Sync installer SHA Pick up changes to upgrade prom, fix istiod, and fix injection template * fix demo * Fix mandiff * Fix test

…lot endpoints. (istio#635) This patch allows you to override global.remotePilotAddress for remote clusters. See istio#19639 for a full issue write-up. Fixes istio#19639

* xds: use typedstruct Signed-off-by: Kuat Yessenov <kuat@google.com> * change indent Signed-off-by: Kuat Yessenov <kuat@google.com>

* update translate config * fix test and address comment * fix lint

* Set CNI namespace as kube-system Signed-off-by: Chun Lin Yang <clyang@cn.ibm.com> * Fix test failure issues

This patch sets `SilenceUsage` to true. Currently mesh command always outputs usage when it fails and makes difficult to find the main error message.

Major change here is a new panel added to mesh dashboard and galley metrics name fixes

* Fixes for istiod injection Right now there is no way to remove the sidecar injector deployment when using istiod. This means you will always have a bunch of config that is unused. Instead, when istiod is enabled, remove the injector stuff. Alternative approach would be to add an explicit flag for this, like istiod.disableinjector or something, but I am not sure that is needed at this time. Additionally, add back features from the old injection to avoid regressions. These are critical features, not just legacy junk. Finally, reuse the old name for the configmap so existing tooling like istioctl can continue to function correctly. * Fix tests

* Fix pilot readiness probe Not sure why this is defined here, it is a duplicate of installer... * gen

* Sync installer I was going to wait for the repo merge, but I wanted to fix any failing tests that may pop up to make that merge smoother * fix e2e

* add issue link to the ObjectMeta marshalling. * fix test.

* limit cni in kube-system Signed-off-by: Chun Lin Yang <clyang@cn.ibm.com> * keep configurable

* Migrate code to use IstioOperator API in istio/api * Fix manifest generation * Prometheus test case * Refactor addons and gateways * Add gateway indexing * Fix remaining tests * Fix controller test * Update profiles with new CR headers * Update readme * Fix dependency processing * Add passthrough for values to gateways * Add addon name field * More test fixes * Add default gateway override test case * More test fixes * Update goldens * Lint * Make gen * Update goldens * Fix mandiff * Fix remaining mandiff profiles * Update goldens * Fix addon status * Remove lingering references to IstioControlPlane * Make gen * Update fields to refer to istiooperator * Update to v1alpha1 * Change CRD meta to plural * Remove lingering v1alpha2, icp references * Move CR back to istio-system * Rename local vars from icp to iop * Change deployment filenames back to v1alpha2 since tests expect it * Make gen * attempt to fix test Co-authored-by: John Howard <howardjohn@google.com>

…tio#645) * Mount Citadel CA cert to data plane and config istiod CA provider * Change the mount path * Remove an environmental variable

* Support customized labels for istio-ingress Signed-off-by: Chun Lin Yang <clyang@cn.ibm.com> * update istio-egress

* fix iop status update issue. * fix lint. * update status with instance upadter. * add back the status subresource. * move status out of spec.

* remove dynamic rest mapper from openshift. * fix tests.

Merge istio/installer and istio/operator into istio/istio #20106

Merge istio/installer and istio/operator into istio/istio #20106

Commits on Oct 26, 2019

Commits on Oct 28, 2019

Commits on Oct 29, 2019

Commits on Oct 30, 2019

Commits on Oct 31, 2019

Commits on Nov 1, 2019

Commits on Nov 2, 2019

Commits on Nov 3, 2019

Commits on Nov 4, 2019

Commits on Nov 5, 2019

Commits on Nov 6, 2019

Commits on Nov 7, 2019

Commits on Nov 8, 2019

Commits on Nov 9, 2019

Commits on Nov 10, 2019

Commits on Nov 11, 2019

Commits on Nov 12, 2019

Commits on Nov 13, 2019

Commits on Nov 14, 2019

Commits on Nov 15, 2019

Commits on Nov 16, 2019

Commits on Nov 17, 2019

Commits on Nov 18, 2019

Commits on Nov 20, 2019

Commits on Nov 21, 2019

Commits on Nov 22, 2019

Commits on Nov 24, 2019

Commits on Nov 25, 2019

Commits on Nov 26, 2019

Commits on Nov 27, 2019

Commits on Nov 28, 2019

Commits on Dec 1, 2019

Commits on Dec 2, 2019

Commits on Dec 3, 2019

Commits on Dec 4, 2019

Commits on Dec 5, 2019

Commits on Dec 6, 2019

Commits on Dec 7, 2019

Commits on Dec 9, 2019

Commits on Dec 10, 2019

Commits on Dec 11, 2019

Commits on Dec 12, 2019

Commits on Dec 13, 2019

Commits on Dec 15, 2019

Commits on Dec 16, 2019

Commits on Dec 17, 2019

Commits on Dec 18, 2019

Commits on Dec 19, 2019

Commits on Dec 20, 2019

Commits on Dec 22, 2019

Commits on Dec 23, 2019

Commits on Dec 24, 2019

Commits on Dec 26, 2019

Commits on Dec 27, 2019

Commits on Dec 28, 2019

Commits on Dec 29, 2019

Commits on Jan 1, 2020

Commits on Jan 2, 2020

Commits on Jan 3, 2020

Commits on Jan 5, 2020

Commits on Jan 6, 2020

Commits on Jan 7, 2020

Commits on Jan 8, 2020

Commits on Jan 10, 2020

Commits on Jan 12, 2020