ReadNamespacedPodLog / ReadNamespacedPodLogAsync Stopped working starting in Version 5+

[scubakiz]

The following code worked fine until I updated the .NET library to 5/6:

     var log = _kubernetesClient.ReadNamespacedPodLog(
                podName,
                currentNamespace,
                container: containerName,
                tailLines: returnRailLines,
                timestamps: showLogTimestamps,
                previous: previousLog);

The async version worked too:

     var log = await _kubernetesClient.ReadNamespacedPodLogAsync(
                podName,
                currentNamespace,
                container: containerName,
                tailLines: returnRailLines,
                timestamps: showLogTimestamps,
                previous: previousLog);

After upgrading to 5.0.20 (and then 6.0.26), both versions of these simply block forever. Nothing is ever returned and no error is shown.

I tried upgrading to version 7.0, and then the entire app stopped working, so I couldn't test the above functionality. That may be because I'm connecting to a 1.21.9 cluster. Getting version 6.0 to work would be fine for now.

Platform: .Net 5.0
Kubectl version:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.9", GitCommit:"37f338aa38e0427e127162afe462e2f4150f0ba3", GitTreeState:"clean", BuildDate:"2022-02-07T20:49:26Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

[tg123]

hmm I did not remember there was a change related to log and .net runtime

here is an example
https://github.com/kubernetes-client/csharp/blob/master/examples/logs/Logs.cs

and test used of log

csharp/tests/E2E.Tests/MnikubeTests.cs

Line 345 in 8e86191

public async Task LogStreamTestAsync()

can you please confirm if the examples work well?

[scubakiz]

I tried the first example and it fails everywhere.

Brand new console app, .Net 5. I compiled it with 4.0.26, 5.0.20 & 6.0.26.

I tried them connected to an AKS cluster (see version above) and an EKS cluster:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

They all return an internal error in the SDK:

Microsoft.Rest.HttpOperationException
HResult=0x80131500
Message=Operation returned an invalid status code 'BadRequest'
Source=KubernetesClient
StackTrace:
at k8s.Kubernetes.d__115.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in //src/libraries/System.Private.CoreLib/src/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 56
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 173
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 150
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 551
at ConsoleApp3.Logs.

d__0.MoveNext() in C:\Users\makizhne\source\repos\ConsoleApp3\ConsoleApp3\Program.cs:line 24
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in //src/libraries/System.Private.CoreLib/src/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 56
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 173
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 150
at System.Runtime.CompilerServices.TaskAwaiter.GetResult() in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 105
at ConsoleApp3.Logs.(String[] args)

This exception was originally thrown at this call stack:
k8s.Kubernetes.ReadNamespacedPodLogWithHttpMessagesAsync(string, string, string, bool?, bool?, int?, bool?, bool?, int?, int?, bool?, System.Collections.Generic.Dictionary<string, System.Collections.Generic.List>, System.Threading.CancellationToken)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in ExceptionDispatchInfo.cs
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) in TaskAwaiter.cs
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) in TaskAwaiter.cs
System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult() in TaskAwaiter.cs
ConsoleApp3.Logs.Main(string[]) in Program.cs
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in ExceptionDispatchInfo.cs
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) in TaskAwaiter.cs
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) in TaskAwaiter.cs
System.Runtime.CompilerServices.TaskAwaiter.GetResult() in TaskAwaiter.cs
...
[Call Stack Truncated]

For reference purposes, here's an older version of my app, when this functionality worked fine:

https://scubakiz.github.io/clusterinfo/

See attached screenshots.

[tg123]

catch HttpOperationException ex and dump its content. it should contains what returns from server side

[scubakiz]

Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'BadRequest'
at k8s.Kubernetes.ReadNamespacedPodLogWithHttpMessagesAsync(String name, String namespaceParameter, String container, Nullable1 follow, Nullable1 insecureSkipTLSVerifyBackend, Nullable1 limitBytes, Nullable1 pretty, Nullable1 previous, Nullable1 sinceSeconds, Nullable1 tailLines, Nullable1 timestamps, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at ConsoleApp3.Logs.Main(String[] args) in C:\Users\user\source\repos\ConsoleApp3\ConsoleApp3\Program.cs:line 27

[scubakiz]

To be clear, EVERY other part of the SDK (6.0.26) is working fine.

[tg123]

print HttpOperationException.Response.Content, it should contain the error from server
the error is 401 badrequest which means client side was sending requests server could not understand

[brendandburns]

I'm betting that some query parameter got added w/ a null/invalid value when the code was regenerated, but that's just a guess.

@tg123 is right that there should be more information about why the request is considered bad in the body of the response, that will help us understand what is wrong.

[lineville]

I'm having the same issue as well, the method ReadNamespacedPodLogWithHttpMessagesAsync is failing with the same error code. All other functions seem to be working as expected. I believe it may be an issue related to when the pod is not created successfully, it fails to read the logs. When the pod is created successfully this call works fine. However, it is still needed in some cases to inspect the logs of a failed pod similar to the behavior of calling kubectl describe -n <namespace> <pod>.

[tg123]

@lineville
what is the output of kubectl logs

[lineville]

@tg123 For context: running kubectl describe pod -n <namespace> <pod> shows me the pod, along with a list of events which include useful Warnings and Failures about the pod creation state -- that's really what I'm after, I'd like to get a useful error to surface about why the pod is not ready. The issue is that when the pod is created successfully the call to ReadNamespacedPodLogWithHttpMessagesAsync succeeds, but if there was an issue and the pod is left hung in "Pending" then this call fails and I can't inspect the logs.

To answer your question when I run kubectl logs this fails because it is an invalid command. When I run kubectl logs -n <my-namespace> -p <my-pod-name> I get the following error: Error from server (BadRequest): previous terminated container "my_container" in pod "my_pod_name" not found -- this seems similar to the exception that's being raised by the sdk.

Is there an sdk method that invokes kubectl describe pod -n <namespace> <pod> or does the ReadNamespacedPodLogWithHttpMessagesAsync call just use kubectl logs ?

[lineville]

@tg123 I've also tried calling ReadNamespacedEventWithHttpMessagesAsync to inspect the events of the pod after it's failed to create, and had similar results. It threw the same 'BadRequest' exception while attempting to read and watch the events from a pod.

[tg123]

i am porting kubectl style apis, but the process is a bit slow as i am busy with day time works

here is how describe po works, it is some data aggregation for some get apis
https://github.com/kubernetes/kubectl/blob/2d67b5a3674c9c661bc03bb96cb2c0841ccee90b/pkg/describe/describe.go#L705

[lineville]

Thanks, appreciate the work you're doing here 🙏 !

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale