exec hangs, is there something i need to do differently so that it returns?

[lknite]

Describe the bug
Running a command in a pod works, but it never returns. In the code below we never see "after exec". I saw another similar ticket, but that was resolved so not sure if its relevant: #1267

Kubernetes C# SDK Client Version
e.g. 11.0.44

Server Kubernetes Version
e.g. 1.25.9

Dotnet Runtime Version
e.g. net7

To Reproduce

            var cmds = new List<string>();
            cmds.Add("pwd");

            Console.WriteLine("before exec");
            await kubeclient.NamespacedPodExecAsync(
                "<pod_name_we_want_to_run_command_in>", "<namespace_where_pod_is>", "<container_in_pod>", cmds, false, CallbackHandle, Globals.cancellationToken);
            Console.WriteLine("after exec");

            return null;
        }

        public static Task CallbackHandle(Stream stdIn, Stream stdOut, Stream stdErr)
        {
            StreamReader sr = new StreamReader(stdOut);
            while (!sr.EndOfStream)
            {
                Console.WriteLine(sr.ReadLine());
            }

            return null;
        }

Expected behavior
I expect the command to execute in the pod and then return, allowing the program to continue, but it seems the await on the Exec never returns.

Where do you run your app with Kubernetes SDK (please complete the following information):

• OS: [e.g. Linux]
  linux, redhat9
• Environment [e.g. container]
  container
• Cloud [e.g. Azure]
  onprem, kubeadm

Additional context
I tried removing code from the CallbackHandle, to instead just 'return null', in case that might help, but it didn't seem to.

[tg123]

1267 is to fix exec in kubeconfig

exec is covered by e2e test

csharp/tests/E2E.Tests/MinikubeTests.cs

Line 718 in 66ad77d

var ws = await client.WebSocketNamespacedPodExecAsync(

let me know what cmd and i will try to repro

[lknite]

In my sample code I just run: pwd

I'll take a look at the link you provided.

Only difference I see is the NamespacePodExecAsync in the minikubetests.cs has a .ConfigureAwait(false); I'll test again with that addition.

[tg123]

catch exception to see if any

[lknite]

It is catching an exception, reports back the line number in my code, troubleshooting ...

It is executing the command successfully, so something about after it finishes running the command.

System.NullReferenceException: Object reference not set to an instance of an object.   at k8s.Kubernetes.NamespacedPodExecAsync(String name, String namespace, String container, IEnumerable`1 command, Boolean tty, ExecAsyncCallback action, CancellationToken cancellationToken)   at Method(String clusterName, String clusterNamespace) in /home/jenkins/agent/workspace/hidden-controller/hidden-controller/src/ ...

[lknite]

Tried using a 'string[] cmd = { "pwd" };' instead of a List, still the same result.

Maybe a permissions error? I have:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: role-argocd-exec
  namespace: argocd
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - get 
  - create

I'll try adding watch ... adding watch didn't help.

[lknite]

I tried the ExecInPod code from the example:
https://github.com/kubernetes-client/csharp/blob/master/examples/exec/Exec.cs

It's giving me more information when it catches Exception:

System.Net.WebSockets.WebSocketException (0x80004005): The server returned status code '403' when status code '101' was expected.   at System.Net.WebSockets.WebSocketHandle.ValidateResponse(HttpResponseMessage response, String secValue, ClientWebSocketOptions options)   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, HttpMessageInvoker invoker, CancellationToken cancellationToken, ClientWebSocketOptions options)   at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, HttpMessageInvoker invoker, CancellationToken cancellationToken)   at k8s.WebSocketBuilder.BuildAndConnectAsync(Uri uri, CancellationToken cancellationToken)   at k8s.Kubernetes.StreamConnectAsync(Uri uri, String webSocketSubProtocol, Dictionary`2 customHeaders, CancellationToken cancellationToken)   at gge.K8sControllers.ClusterK8sController.ExecInPod(IKubernetes client, V1Pod pod, String cmd) in 

maybe it is a permission issue

tried various permissions and eventually just granted everything, still getting same error

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: role-argocd-exec
  namespace: argocd
rules:
- apiGroups:
  - ""
  resources:
  - "*"
  verbs:
  - "*"

[lknite]

Well, this was blocking my development but with catching the exception I can get back to work. As long as the command is executing suppose it doesn't matter that it's throwing an exception, can come back to figure it out later.

[lknite]

Sorry for all the message, thinking out loud here, I think maybe it's my callback function returning null at the end. Need to return something more useful there maybe.

    /// A <see cref="Task"/> which represents the asynchronous processing of the process input, output and error streams. This task
    /// should complete once you're done interacting with the remote process.
    /// </returns>

... by returning a random task it doesn't crash now with an exception, but also it never returns:

        public static void PrintEvenNumbers()
        {
            //Console.WriteLine("all is done");
        }
        public static Task CallbackHandle(Stream stdIn, Stream stdOut, Stream stdErr)
        {
            StreamReader sr = new StreamReader(stdOut);
            while (!sr.EndOfStream)
            {
                Console.WriteLine(sr.ReadLine());
            }
            return new Task(PrintEvenNumbers);
        }

[tg123]

Task.ComplateTask or Task.Delay(0)

[lknite]

I never did get this working the right way, instead I used a try / catch so that I could continue on after the exception was raised.

In any case, not specifically due to this unexpected behavior, but partially ... I abandoned exec in favor of another technique.

Closing as I don't expect to have a chance to test for awhile. Will update here when I do get a chance.