New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

istio-statsd-prom-bridge is in crashloopbackoff in Knative v0.1.1 deployment #1921

Closed
simingweng opened this Issue Aug 22, 2018 · 14 comments

Comments

Projects
None yet
@simingweng

simingweng commented Aug 22, 2018

Expected Behavior

Following the v0.1.1 Knative installation guide, the step of installing Istio with command

curl -L https://raw.githubusercontent.com/knative/serving/v0.1.1/third_party/istio-0.8.0/istio.yaml \ | sed 's/LoadBalancer/NodePort/' \ | kubectl apply -f -

should bring up the the Istio pods

Actual Behavior

istio-statsd-prom-bridge pod ends up in crashloopbackoff

Steps to Reproduce the Problem

  1. execute the command above
  2. kubectl -n istio-system get pods

Additional Info

this is due to the latest v0.7.0 release of prometheus statsd_exporter Docker image, in which there was a breaking change introduced to switch from single dash to double dash for command line arguments.

Knative v0.1.1 istio manifest file is using the "latest" tag, therefore the v0.7.0 statsd_exporter is automatically in, and
args: - '-statsd.mapping-config=/etc/statsd/mapping.conf'
no longer works.

@simingweng

This comment has been minimized.

Show comment
Hide comment
@simingweng

simingweng commented Aug 22, 2018

/assign @mdemirhan

@mattmoor

This comment has been minimized.

Show comment
Hide comment
@mattmoor
Member

mattmoor commented Aug 22, 2018

@pzmrzy

This comment has been minimized.

Show comment
Hide comment
@pzmrzy

pzmrzy Aug 24, 2018

I also get same error when I install kNative with minikube and GKE(Master version
1.10.6-gke.2) by following the GKE guild
Works after change to install istio-1.0.1
kubectl apply -f https://raw.githubusercontent.com/knative/serving/master/third_party/istio-1.0.1/istio.yaml

pzmrzy commented Aug 24, 2018

I also get same error when I install kNative with minikube and GKE(Master version
1.10.6-gke.2) by following the GKE guild
Works after change to install istio-1.0.1
kubectl apply -f https://raw.githubusercontent.com/knative/serving/master/third_party/istio-1.0.1/istio.yaml

@yunlzheng

This comment has been minimized.

Show comment
Hide comment
@yunlzheng

yunlzheng Aug 25, 2018

- '-statsd.mapping-config=/etc/statsd/mapping.conf'

The statsd-prom-bridge args is wrong.

statsd_exporter: error: unknown short flag '-s', try --help

This should be

containers:
      - args:
        - --statsd.mapping-config=/etc/statsd/mapping.conf ## Update this deployment manual

yunlzheng commented Aug 25, 2018

- '-statsd.mapping-config=/etc/statsd/mapping.conf'

The statsd-prom-bridge args is wrong.

statsd_exporter: error: unknown short flag '-s', try --help

This should be

containers:
      - args:
        - --statsd.mapping-config=/etc/statsd/mapping.conf ## Update this deployment manual
@sdake

This comment has been minimized.

Show comment
Hide comment
@sdake

sdake Aug 26, 2018

Contributor

The correct solution is to use what was tested with Istio 0.8.0 - which is statsd v0.6.0. I'll correct this problem in the Istio 0.8 branch. Apologies for the use of :latest tags, we are/have removed them from the upstream templating for Istio.

Contributor

sdake commented Aug 26, 2018

The correct solution is to use what was tested with Istio 0.8.0 - which is statsd v0.6.0. I'll correct this problem in the Istio 0.8 branch. Apologies for the use of :latest tags, we are/have removed them from the upstream templating for Istio.

sdake added a commit to sdake/istio that referenced this issue Aug 26, 2018

statsd-exporter introduced knative regression
stats-exporter:latest introduced a breaking change documented here
knative/serving#1921

The correct solution is to use statsd-exporter that shipped with
Istio 0.8.0 - which was v0.6.0.
@tcnghia

This comment has been minimized.

Show comment
Hide comment
@tcnghia

tcnghia Aug 27, 2018

Contributor

@yanweiguo do you know how this will impact our monitoring components? thanks

Contributor

tcnghia commented Aug 27, 2018

@yanweiguo do you know how this will impact our monitoring components? thanks

@yanweiguo

This comment has been minimized.

Show comment
Hide comment
@yanweiguo

yanweiguo Aug 27, 2018

Contributor

@akyyy Yao, could you have a look for this?

Contributor

yanweiguo commented Aug 27, 2018

@akyyy Yao, could you have a look for this?

@tcnghia

This comment has been minimized.

Show comment
Hide comment
@tcnghia

tcnghia Aug 28, 2018

Contributor

It looks like this an Istio installation with istio-statsd-prome-bridge crashing doesn't affect Knative functionalities.

We should still fix the issue to avoid confusions from users.

Contributor

tcnghia commented Aug 28, 2018

It looks like this an Istio installation with istio-statsd-prome-bridge crashing doesn't affect Knative functionalities.

We should still fix the issue to avoid confusions from users.

rshriram added a commit to istio/istio that referenced this issue Aug 29, 2018

statsd-exporter introduced knative regression (#8249)
stats-exporter:latest introduced a breaking change documented here
knative/serving#1921

The correct solution is to use statsd-exporter that shipped with
Istio 0.8.0 - which was v0.6.0.
@bobcatfish

This comment has been minimized.

Show comment
Hide comment
@bobcatfish

bobcatfish Sep 19, 2018

Contributor

So it seems like this still happens when you follow along with https://github.com/knative/docs/blob/master/install/Knative-with-GKE.md

image

Contributor

bobcatfish commented Sep 19, 2018

So it seems like this still happens when you follow along with https://github.com/knative/docs/blob/master/install/Knative-with-GKE.md

image

@cforce

This comment has been minimized.

Show comment
Hide comment
@cforce

cforce Sep 28, 2018

It is also reproducable with docker for windows edge (statsd_exporter: error: unknown short flag '-s', try --help).When using https://github.com/knative/serving/blob/master/third_party/istio-1.0.2/istio.yaml issue is gone

cforce commented Sep 28, 2018

It is also reproducable with docker for windows edge (statsd_exporter: error: unknown short flag '-s', try --help).When using https://github.com/knative/serving/blob/master/third_party/istio-1.0.2/istio.yaml issue is gone

@mdemirhan

This comment has been minimized.

Show comment
Hide comment
@mdemirhan

mdemirhan Oct 1, 2018

Member

#1922 was submitted to fix this. Will prioritize getting that in soon.

Member

mdemirhan commented Oct 1, 2018

#1922 was submitted to fix this. Will prioritize getting that in soon.

@mdemirhan

This comment has been minimized.

Show comment
Hide comment
@mdemirhan

mdemirhan Oct 15, 2018

Member

This should be fixed now (#1922). Let me know if you hit issues.

Member

mdemirhan commented Oct 15, 2018

This should be fixed now (#1922). Let me know if you hit issues.

@mdemirhan mdemirhan closed this Oct 15, 2018

@mdemirhan

This comment has been minimized.

Show comment
Hide comment
@mdemirhan

mdemirhan Oct 15, 2018

Member

@tcnghia mentioned that we need to cut a new release. Reopening the issue and assigning to @adrcunha to cut a new release.

/assign @adrcunha

Member

mdemirhan commented Oct 15, 2018

@tcnghia mentioned that we need to cut a new release. Reopening the issue and assigning to @adrcunha to cut a new release.

/assign @adrcunha

@mdemirhan mdemirhan reopened this Oct 15, 2018

@mdemirhan mdemirhan removed their assignment Oct 15, 2018

@adrcunha

This comment has been minimized.

Show comment
Hide comment
@adrcunha

adrcunha Oct 15, 2018

Contributor

Synced with Mustafa, better cutting a new release.

/close

Contributor

adrcunha commented Oct 15, 2018

Synced with Mustafa, better cutting a new release.

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment