Move to using Istio 1.1.1. #3353

tcnghia · 2019-03-04T17:18:35Z

Proposed Changes

Move to using Isio 1.1.1.

Release Note

Move to using Isio 1.1.1.

tcnghia · 2019-03-07T02:21:44Z

/retest

vendor/github.com/knative/test-infra/scripts/library.sh

tcnghia · 2019-03-07T18:50:24Z

@adrcunha that is correct.

Istio 1.1 split the installation into two steps: the first installer runs to install all the CRDs. So it reduces the existing race condition by allowing us to check for the installer to complete first.

tanzeeb · 2019-03-12T15:30:31Z

I think the 503s in these conformance tests are because of segfaults in the istio-ingressgateway. Logs from my cluster:

[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x0          
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:67] Backtrace (use tools/stack_decode.py to get line numbers):       
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7f71d9903390]                                
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: std::_Function_handler<>::_M_invoke() [0x8ae399]             
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x896cfd]  
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: event_process_active_single_queue [0xc2fa44]                 
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: event_base_loop [0xc2e62c]                                   
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Event::DispatcherImpl::run() [0x896bc3]               
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Server::WorkerImpl::threadRoutine() [0x8922c2]        
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__inv
[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: start_thread [0x7f71d98f96ba]                                
 warn    Epoch 1 terminated with an error: signal: segmentation fault (core dumped)                                                                                        
 warn    Aborted all epochs                                                                                                                                                
 info    Epoch 1: set retry delay to 200ms, budget to 9                                                                                                                    
 info    Reconciling retry (budget 9)                                                                                                                                      
 info    Epoch 0 starting                                                                                                                                                  
 info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --servi

The 503s are non-deterministic otherwise, happens at different times, on different tests, and never happens when you run a specific test.

Update:

I haven't seen a segfault all afternoon, but I still see the 503s.

In addition, I had an EnvoyFilter running the whole time. I still see 503s after removing it.

tcnghia · 2019-03-13T19:09:00Z

/retest

tcnghia · 2019-03-14T00:02:53Z

/retest

chaodaiG · 2019-03-14T01:36:39Z

/retest

chaodaiG · 2019-03-14T02:23:09Z

/test pull-knative-serving-upgrade-tests

ZhiminXiang · 2019-03-19T21:53:28Z

third_party/istio-1.1-rc/download-istio.sh

+  > ../istio-crds.yaml
+
+# Create a custom cluster local gateway, based on the Istio custom-gateway template.
+helm template --namespace=istio-system \


We also need key gateways.istio-ingressgateway.sds.enabled=true in order to support #1767 and #3052.

tcnghia · 2019-03-26T06:34:20Z

Conformance tests and scale tests are passing consistently now.

Tests that fail consistently:

TestDestroyPodInFlight: we may have to add back our graceful shutdown patch.
TestServiceToServiceCall/*
TestAutoscaleUpDownUp
TestActivatorOverload
TestGrpc*FromZero

tcnghia · 2019-03-26T06:34:39Z

/test pull-knative-serving-integration-tests

tcnghia · 2019-03-26T14:02:06Z

Only TestDestroyPodInFlight and TestServiceToServiceCall/* are failing now.

/test pull-knative-serving-integration-tests

tcnghia · 2019-03-26T18:03:25Z

/test pull-knative-serving-integration-tests

tcnghia · 2019-03-26T20:20:18Z

/test pull-knative-serving-unit-tests

tcnghia · 2019-03-26T21:31:32Z

looks like all tests passing now

tcnghia · 2019-03-26T22:34:57Z

Summary:

The PR Envoy Graceful Shutdown istio/istio#11485 doesn't seem to satisfy our TestDestroyPodInflight test. We will keep the prestop sleep patch as a result. I will work with Istio test infra to translate this test into a test on Istio side, may be post-submit.
The PR clarified mesh connect timeout fields based on code impl istio/istio#12089 missed 1.1.1 by 3 days, so we may have to wait until 1.1.2 to get it. We will keep the connectTimeout patch as a result.

Other than that our tests seem to be happy with Istio 1.1.1

tcnghia · 2019-03-29T23:18:01Z

/retest

tanzeeb · 2019-04-01T15:10:52Z

/test pull-knative-serving-upgrade-tests
/test pull-knative-serving-integration-tests

mattmoor

/lgtm
/approve

knative-prow-robot · 2019-04-02T15:21:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattmoor, tcnghia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~hack/OWNERS~~ [mattmoor]
~~test/OWNERS~~ [mattmoor,tcnghia]
~~third_party/OWNERS~~ [mattmoor,tcnghia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

knative-prow-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 4, 2019

knative-prow-robot requested review from adrcunha and lichuqiang March 4, 2019 17:18

mattmoor-sockpuppet reviewed Mar 4, 2019

View reviewed changes

adrcunha reviewed Mar 7, 2019

View reviewed changes

vendor/github.com/knative/test-infra/scripts/library.sh Outdated Show resolved Hide resolved

tcnghia force-pushed the istio-1.1 branch from eb7ec08 to 4539e00 Compare March 7, 2019 21:27

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2019

tcnghia force-pushed the istio-1.1 branch from 4539e00 to a30f2cc Compare March 7, 2019 21:29

knative-prow-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2019

tcnghia changed the title ~~[WIP] Move to using Istio 1.1.0-rc.2 to test our compatibility.~~ [WIP] Move to using Istio 1.1.0-rc.4 to test our compatibility. Mar 13, 2019

ZhiminXiang reviewed Mar 19, 2019

View reviewed changes

markusthoemmes mentioned this pull request Mar 25, 2019

Upgrade Istio to 1.1 #3515

Closed

tcnghia force-pushed the istio-1.1 branch from d83533e to 4906be5 Compare March 25, 2019 22:10

knative-prow-robot added area/API API objects and controllers area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features labels Mar 25, 2019

tcnghia changed the title ~~[WIP] Move to using Istio 1.1.0-rc.4 to test our compatibility.~~ [WIP] Move to using Istio 1.1.0 to test our compatibility. Mar 25, 2019

mattmoor-sockpuppet reviewed Mar 25, 2019

View reviewed changes

tcnghia changed the title ~~[WIP] Move to using Istio 1.1.0 to test our compatibility.~~ [WIP] Move to using Istio 1.1.1 to test our compatibility. Mar 26, 2019

tcnghia force-pushed the istio-1.1 branch 2 times, most recently from 7b32c38 to c8d68fc Compare March 26, 2019 17:46

tcnghia force-pushed the istio-1.1 branch from 01b751b to 3999771 Compare March 26, 2019 19:00

tcnghia force-pushed the istio-1.1 branch from 4495437 to 87ac90a Compare March 26, 2019 21:43

googlebot added the cla: yes Indicates the PR's author has signed the CLA. label Mar 29, 2019

tcnghia force-pushed the istio-1.1 branch from 87ac90a to 14f1e2f Compare March 30, 2019 14:36

Check in Istio 1.1.1 to enable testing, but do not release its yamls.

9e4208c

tcnghia force-pushed the istio-1.1 branch from 14f1e2f to 9e4208c Compare March 30, 2019 14:52

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 30, 2019

Run tests with 1.1.1 too.

36cad51

knative-prow-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 2, 2019

tcnghia changed the title ~~[WIP] Move to using Istio 1.1.1 to test our compatibility.~~ Move to using Istio 1.1.1. Apr 2, 2019

knative-prow-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 2, 2019

mattmoor reviewed Apr 2, 2019

View reviewed changes

knative-prow-robot assigned mattmoor Apr 2, 2019

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 2, 2019

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 2, 2019

knative-prow-robot merged commit 8a6c8f9 into knative:master Apr 2, 2019

adrcunha mentioned this pull request Apr 4, 2019

Default Knative Serving installation function is broken knative/test-infra#674

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move to using Istio 1.1.1. #3353

Move to using Istio 1.1.1. #3353

tcnghia commented Mar 4, 2019 •

edited

tcnghia commented Mar 7, 2019

tcnghia commented Mar 7, 2019

tanzeeb commented Mar 12, 2019 •

edited

tcnghia commented Mar 13, 2019

tcnghia commented Mar 14, 2019

chaodaiG commented Mar 14, 2019

chaodaiG commented Mar 14, 2019

ZhiminXiang Mar 19, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019 •

edited

tcnghia commented Mar 29, 2019

tanzeeb commented Apr 1, 2019

mattmoor left a comment

knative-prow-robot commented Apr 2, 2019

Move to using Istio 1.1.1. #3353

Move to using Istio 1.1.1. #3353

Conversation

tcnghia commented Mar 4, 2019 • edited

Proposed Changes

tcnghia commented Mar 7, 2019

tcnghia commented Mar 7, 2019

tanzeeb commented Mar 12, 2019 • edited

tcnghia commented Mar 13, 2019

tcnghia commented Mar 14, 2019

chaodaiG commented Mar 14, 2019

chaodaiG commented Mar 14, 2019

ZhiminXiang Mar 19, 2019

Choose a reason for hiding this comment

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019

tcnghia commented Mar 26, 2019 • edited

tcnghia commented Mar 29, 2019

tanzeeb commented Apr 1, 2019

mattmoor left a comment

Choose a reason for hiding this comment

knative-prow-robot commented Apr 2, 2019

tcnghia commented Mar 4, 2019 •

edited

tanzeeb commented Mar 12, 2019 •

edited

tcnghia commented Mar 26, 2019 •

edited