Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

redeploying a kubernetes service results in temporary outage #22

Closed
rhs opened this issue Apr 20, 2017 · 2 comments
Closed

redeploying a kubernetes service results in temporary outage #22

rhs opened this issue Apr 20, 2017 · 2 comments

Comments

@rhs
Copy link
Contributor

rhs commented Apr 20, 2017

when I redeploy a kubernetes service sitting behind a given ambassador route, there seem to be 5-10 seconds of downtime... this doesn't happen if I'm connecting directly to the external ip of the service
@kflynn kflynn added this to Test in Launch! Apr 25, 2017
@kflynn
Copy link
Member

kflynn commented Apr 26, 2017 •
edited

The issue seems to be that, since Envoy only queries the SDS every 15 seconds by default, it can have an endpoint that's stale if the endpoint goes away right after the query (as can happen when the service is updated).

The simplest way to work around this for Ambassador-as-an-API-gateway is likely just to let Kubernetes handle load balancing for the moment. That costs a lot of features, but it's dirt simple.

Other things we could do:

  • drop the 15-second poll to 1 second
  • document that you must do rolling upgrades to avoid the outage (ugh)
  • teach Envoy how to accept updates over a websocket asynchronously ( ;) and also ugh)

@kflynn
Copy link
Member

kflynn commented May 3, 2017

Closing since we switched away from SDS for launch. Also see #47.

@kflynn kflynn closed this as completed May 3, 2017
@kflynn kflynn removed this from Drivers in Launch! May 3, 2017
@vaibhavrtk vaibhavrtk mentioned this issue Feb 13, 2019
Closed
kflynn pushed a commit that referenced this issue Apr 10, 2019
@richarddli
Merge pull request #22 from datawire/config
fb2a979 
delete obsolete docs
LukeShu added a commit that referenced this issue Sep 6, 2019
@LukeShu
Merge pull request #24 from datawire/lukeshu/merged-sidecar-pt1
754ff2a 
This is a bundle of "stable" work pulled out from working toward #22
that I figure I should try to land sooner rather than later.  None of
the commits are really thematically related to eachother.  See
individual commits for details.
rhs added a commit that referenced this issue Oct 9, 2019
@rhs
Merge pull request #22 from datawire/k8s-namespaces
5e8a52d 
Support Kubernetes namespaces
acookin added a commit that referenced this issue Aug 18, 2021
@acookin
envoy v2 fixes maybz (#22)
5cf7440 
* envoy v2 fixes maybz

* envoy v2 fixes maybz
acookin added a commit that referenced this issue Aug 24, 2021
@acookin
envoy v2 fixes maybz (#22)
1fb9b96 
* envoy v2 fixes maybz

* envoy v2 fixes maybz

Signed-off-by: Flynn <flynn@datawire.io>
acookin added a commit that referenced this issue Aug 24, 2021
@acookin @aidanhahn
@AliceProxy @kflynn
[v2.0.2-ea] Release Branch (#3701)
b23ceab 
* ensure pod log uploading when pytests fail

- incorrect if() clause in yaml caused failure to upload pod logs,
  making it hard to diagnose failures

Signed-off-by: Alice Nodelman <anode@datawire.io>

* fixup links in readme (#3682)

Signed-off-by: Alix Cook <alixcook@datawire.io>

* fix chart readme (#3690)

Signed-off-by: Alix Cook <alixcook@datawire.io>

* Document similar implementation in DevPortal (#3691)

* Document similar implementation in DevPortal

Signed-off-by: alex <alex@datawire.io>

* Added comment to the AmbassadorMapping CRD docs attribute

Signed-off-by: alex <alex@datawire.io>

* make generate

Signed-off-by: Alix Cook <alixcook@datawire.io>

Co-authored-by: Alix Cook <alixcook@datawire.io>

* split out t_mappingtests into namespace sets (default and plain)

    - some of the mappingtests are designed to run in the default namespace,
      some in the plain namespace.  Split up these tests into different py
      files to make it easier to understand and execute
    - update pytest harness to start an httpbin for each namespace
      defined in tests under execution
    - update plain test gold file to use correct namespace

Signed-off-by: Alice Nodelman <anode@datawire.io>

* add devel (#3696)

Signed-off-by: Alix Cook <alixcook@datawire.io>

* Update for v2.0.2-ea (#3698)

Signed-off-by: Alix Cook <alixcook@datawire.io>

* Update for v2.0.2-ea

Signed-off-by: Alix Cook <alixcook@datawire.io>

* update envoy commit

Signed-off-by: Aidan Hahn <aidanhahn@datawire.io>

* dont push manifests if IS_PRIVATE

Signed-off-by: Flynn <flynn@datawire.io>

* updates for embargoed

Signed-off-by: Flynn <flynn@datawire.io>

* envoy v3 should be default

Signed-off-by: Flynn <flynn@datawire.io>

* run make generate

Signed-off-by: Flynn <flynn@datawire.io>

* Update for v1.14.0.

* dont push manifests if IS_PRIVATE
* updates for embargoed
* Update for v1.14.0

Signed-off-by: Alix Cook <alixcook@datawire.io>

* fixup changelog

Signed-off-by: Flynn <flynn@datawire.io>

* gotest fix (#16)

* gotest fix

* undo

Signed-off-by: Flynn <flynn@datawire.io>

* remove tests that were commented out on master

Signed-off-by: Aidan Hahn <aidanhahn@datawire.io>
Signed-off-by: Flynn <flynn@datawire.io>

* enable deprecated apis

Signed-off-by: Flynn <flynn@datawire.io>

* pytest corrections (#18)

Signed-off-by: Aidan Hahn <aidanhahn@datawire.io>
Signed-off-by: Flynn <flynn@datawire.io>

* gzip fix (#19)

* gzip fix

* update ambex

* update ambex

* update circle

Signed-off-by: Flynn <flynn@datawire.io>

* pytest envoy v2 test fixes (#21)

Signed-off-by: Flynn <flynn@datawire.io>

* envoy v2 fixes maybz (#22)

* envoy v2 fixes maybz

* envoy v2 fixes maybz

Signed-off-by: Flynn <flynn@datawire.io>

* also push rc for emissary privaet

Signed-off-by: Flynn <flynn@datawire.io>

* allow chunked length (#24)

* expose chunked length

Signed-off-by: AliceProxy <aliceproxy@protonmail.com>

improve test coverage

Signed-off-by: AliceProxy <aliceproxy@protonmail.com>

Remove v2listener and v2Test

Signed-off-by: AliceProxy <aliceproxy@protonmail.com>

add setting to protos

Signed-off-by: AliceProxy <aliceproxy@protonmail.com>

make generate

Signed-off-by: AliceProxy <aliceproxy@protonmail.com>

* comment out test

* test fixes

* UNDO THIS

* mkay

* comment out test

Co-authored-by: AliceProxy <aliceproxy@protonmail.com>
Signed-off-by: Flynn <flynn@datawire.io>

* update pf gomod

Signed-off-by: Flynn <flynn@datawire.io>

* extauth needs "ignore_case" for auth and request headers

Signed-off-by: Aidan Hahn <aidanhahn@datawire.io>
Signed-off-by: Flynn <flynn@datawire.io>

* yank deprecated config option

Signed-off-by: Aidan Hahn <aidanhahn@datawire.io>
Signed-off-by: Flynn <flynn@datawire.io>

* update t_chunked_length test

Signed-off-by: Aidan Hahn <aidanhahn@datawire.io>
Signed-off-by: Flynn <flynn@datawire.io>

* Log ambex snapshots

Signed-off-by: Flynn <flynn@datawire.io>

* ambex logging cleanup

Signed-off-by: Flynn <flynn@datawire.io>

* Update goldfiles for V3 #1 (from d15d3b4)

Signed-off-by: Flynn <flynn@datawire.io>

* fix fake_mapping_cors_test

Signed-off-by: Flynn <flynn@datawire.io>

* fix chunked length

Signed-off-by: Flynn <flynn@datawire.io>

* fix envoy_api_version

Signed-off-by: Flynn <flynn@datawire.io>

* make go-mod-tidy

Signed-off-by: Flynn <flynn@datawire.io>

* Update t_chunked_length for V2

Signed-off-by: Flynn <flynn@datawire.io>

* Update gold files

Signed-off-by: Flynn <flynn@datawire.io>

* CHANGELOG

Signed-off-by: Flynn <flynn@datawire.io>

* make generate

Signed-off-by: Flynn <flynn@datawire.io>

* Temporary fix for repo name.

Signed-off-by: Flynn <flynn@datawire.io>

* remove fake mapping test, make generate

Signed-off-by: Flynn <flynn@datawire.io>

* bump envoy in 2.0.2

* fix manifest stuff

* fix is private thing again

Co-authored-by: Alice Nodelman <anode@datawire.io>
Co-authored-by: Alex Gervais <alex@datawire.io>
Co-authored-by: Aidan Hahn <aidanhahn@datawire.io>
Co-authored-by: Flynn <flynn@datawire.io>
Co-authored-by: Aidan Hahn <87328374+aidanhahn@users.noreply.github.com>
Co-authored-by: AliceProxy <aliceproxy@protonmail.com>
Co-authored-by: Flynn <kflynn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rhs @kflynn