Update to promtail 1.4.1 fail

[jmvizcainoio]

Describe the bug

When i try to upgrade using loki-stack helm chart to the latest version , promtail doesn't start. The pod will be stay in CrashLoopBackOff state.

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (1.4.1)
2. Started Promtail (1.4.1) to tail '...'

Expected behavior
Upgrate promtail to latest version 1.4.1

Environment:

• Infrastructure: [Kubertetes 1.16.7 deployed using kops in aws ]
• Deployment tool: [e.g., helm 3.0.1]
• loki-stack chart 0.34.2
• promtail version 1.4.1

Screenshots, Promtail config, or terminal output

when i get the logs of the pod that is crashing

level=debug ts=2020-04-09T07:14:39.497565402Z caller=manager.go:42 msg="detected pipe from stdin"
level=warn ts=2020-04-09T07:14:39.497613744Z caller=stdin_target_manager.go:90 msg="too many scrape configs, skipping 5 configs."
level=info ts=2020-04-09T07:14:39.498103397Z caller=server.go:147 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-04-09T07:14:39.498416273Z caller=main.go:67 msg="Starting Promtail" version="(version=1.4.1, branch=HEAD, revision=2445ccd2)"

If i change the docker image to 1.4.0 works fine.

I tested in a docker cluster in my laptop using k8s 1.5.5 and works fine.

[cyriltovena]

So it works fine with 1.15 kubernetes but it seems to think there is a pipe when running in kubernetes 1.16.7. hum.

[cyriltovena]

Any chance you're running the container with stdi: true or tty:true ?

[jmvizcainoio]

@cyriltovena Thanks by the response,

I'm not sure about that, i'm using the default config in the helm chart.

just helm upgrade --install loki-stack loki/loki-stack

Thanks

Juanma

[slim-bean]

We had another instance of the same behavior by someone in public slack, different setup, was actually maing their own image, however the output is the same.

I'm thinking that we should remove the code to try to detect if someone is piping to promtail, checking if stdin is a character device isn't proving to be reliable enough and troubleshooting why it's going wrong is extremely difficult.

I think i'm going to suggest we remove the auto detect code and instead add --stdin as a flag if someone would like promtail to read from stdin.

@cyriltovena WDYT?

[cyriltovena]

Yep will do in the meantime use 1.4.0 Le ven. 10 avr. 2020 à 21:19, Ed Welch <notifications@github.com> a écrit :

…

We had another instance of the same behavior by someone in public slack, different setup, was actually maing their own image, however the output is the same. I'm thinking that we should remove the code to try to detect if someone is piping to promtail, checking if stdin is a character device isn't proving to be reliable enough and troubleshooting why it's going wrong is extremely difficult. I think i'm going to suggest we remove the auto detect code and instead add --stdin as a flag if someone would like promtail to read from stdin. @cyriltovena <https://github.com/cyriltovena> WDYT? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1921 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIBF3L3OI5XCOMXORY3KLTRL7ARPANCNFSM4MEQEBOA> .

[avg07]

I have same issue.

helm upgrade --install loki-stack loki/loki-stack

level=warn ts=2020-04-15T17:36:08.883877563Z caller=stdin_target_manager.go:90 msg="too many scrape configs, skipping 4 configs." level=info ts=2020-04-15T17:36:08.884794311Z caller=server.go:147 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses" level=info ts=2020-04-15T17:36:08.885237532Z caller=main.go:67 msg="Starting Promtail" version="(version=1.4.1, branch=HEAD, revision=2445ccd2)"

kubernetes v1.16.2

[jwoconnell]

Same CrashloopBackoff behavior here with promtail 1.4.1 and kubernetes 1.14.3. promtail 1.4.0 worked.

$ helm upgrade --install loki loki/loki-stack

[avg07]

Fixed in:
CHART: loki-stack-0.37.3
APP VERSION: v1.5.0