pod exec not work #41

annProg · 2018-10-01T05:15:52Z

code

$k8sClient->setNamespace('dev');
$pods = $k8sClient->pods()->setLabelSelector(['app' => 'dev-router'])->find();

foreach($pods as $pod) {
	$arr = $pod->toArray();
	if($arr['status']['phase'] == 'Running') {
		print_r($k8sClient->pods()->exec($pod, ['cat','/etc/nginx/nginx.conf']));break;
	}
}

return 400 Bad Request with message Upgrade request required

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRe (truncated...)

The text was updated successfully, but these errors were encountered:

samuelhautcoeur · 2019-02-18T06:24:18Z

Same issue here. The exec endpoint wants a websocket connection.
This was handled nicely in godaddy/kubernetes-client: godaddy/kubernetes-client#319

momon · 2019-05-31T19:34:27Z

I've just encountered this issue. Would this ever be addressed? I am trying to figure out how to get an exec command working via the API like this. I have the exact same error as above.

Message: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

maclof · 2019-05-31T19:54:59Z

I can take a look at this soon

What version of kubernetes are you running?

momon · 2019-05-31T19:57:05Z

@maclof that would be awesome. I am exploring alternatives but if this is fixed here that would make my life a lot easier right away.

My cluster is currently on 1.12.6-gke.11 but I do tend to keep up to date (still in development).

momon · 2019-06-10T14:01:06Z

@maclof have you been able to take a look at this?

Thanks!

maclof · 2019-07-03T15:43:09Z

I've finally had time to start working on this a bit as I have a need for it in my production environment.

I do have a semi working implementation of this, basically instead of sending a HTTP request to the /exec endpoint it now uses a new websocket client dependency to send this request.

momon · 2019-07-03T23:22:12Z

That's great to hear. Much appreciate the effort.

I look forward to that commit so I can try it out.

…t working #41

maclof · 2019-07-04T15:24:28Z

The latest version (v0.14.0) should have fixed this.

I did have some difficulty getting it working on my production Kubernetes v1.9.6 cluster, but I tested it with a v1.14.1 Kubernetes cluster and all is working OK

See below for an example:

$pod = $kubernetesClient->pods()->setFieldSelector([
	'metadata.name' => 'mongodb-0',
])->first();

$response = $kubernetesClient->pods()->exec($pod, [
	'command' => ['ps'],
	'stdout'  => true,
	'stderr'  => true,
]);

If the command you want to execute has multiple arguments, you would define these as seperate elements of the array, e.g.

$command = ['ls', '-al'];

The $response will contain an array of outputted messages, if the message length is 0 it will not be added to the array.

array:1 [▼
  0 => array:2 [▼
    "channel" => "stdout"
    "message" => """
        PID TTY          TIME CMD
          1 ?        01:47:03 mongod
       2208 ?        00:00:00 ps
      """
  ]
]

momon · 2019-07-04T18:44:35Z

Much appreciate this @maclof but seems not to be working for me on 1.13.7-gke.8 (which is the latest one available on GKE).

I seem unable to select my pod using:

$pod = $kubernetesClient->pods()->setFieldSelector([
	'metadata.name' => 'myname',
])->first();

It returns null.

I also tried with labelselector and find but it returns an empty array (as if the deployment had no pods) but they are there because I can fetch them via CLI and see them working. So I am unsure if the latest commit or perhaps the cluster version could be an issue here. I don't see a way to upgrade to 1.14.1 on GKE

momon · 2019-07-05T03:39:44Z

Ok I think I figured out the problem but unsure how to solve it. It appears composer does not want to update past 0.13.0 from your repo. When I try to set 0.14.0 I get this error:

composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package maclof/kubernetes-client 0.14.0 exists as maclof/kubernetes-client[0.0.1, 0.0.10, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.10.0, 0.11.0, 0.12.0, 0.12.1, 0.13.0, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.9.0, 0.9.1, dev-master, dev-revert-23-master] but these are rejected by your constraint.

So as per above, it would appear that I can't upgrade to 0.14.0 ?? any clues?

Update: looks like this is the issue: https://packagist.org/packages/maclof/kubernetes-client packagist does not know about the 0.14.0 version.

momon · 2019-07-05T04:04:52Z

I was finally able to update to 0.14.0 by updating directly from VCS, however, I still can't get this working.

I am getting:

Fatal error: Call to a member function close() on null in /home/***/web/***/vendor/maclof/kubernetes-client/src/Client.php on line 478

Here is an example of my request:

                        $this->gke->setNamespace('default');
                        $pods = $this->gke->pods()->setLabelSelector(['app' => 'myappname'])->find();

                        foreach($pods as $pod) {
                            $arr = $pod->toArray();

                            if($arr['status']['phase'] == 'Running') {

                                $response = $this->gke->pods()->exec($pod, [
                                    'command' => ['ps'],
                                    'stdout'  => true,
                                    'stderr'  => true,
                                ]);

                            }
                        }

                        echo '<pre>';
                        print_r($response);
                        echo '</pre>';

maclof · 2019-07-05T04:57:15Z

I think my GitHub webhooks are out of date which is why you couldn’t update via composer without specifying the vcs url directly, I’ll get this fixed for the next release

I can fix your reported error about calling close on null, but that likely means it wasn’t able to connect for some reason.

What options are you specifying to the client for authentication? Client certificates / token etc? It’s been a while since I’ve used GKE

momon · 2019-07-05T13:15:42Z

@maclof on my sandbox (which is where I am testing this) I am using endpoint IP, username and password.

I am pretty sure it is connecting because I do get the data from the cluster on

$pods = $this->gke->pods()->setLabelSelector(['app' => 'myappname'])->find();

The error shows only on

$this->gke->pods()->exec($pod, [
                                    'command' => ['ps'],
                                    'stdout'  => true,
                                    'stderr'  => true,
                                ]);

maclof · 2019-07-08T09:33:14Z

@momon can you share the full schema for the pod? Remove any environment variable passwords, etc.

momon · 2019-07-08T15:27:32Z

@maclof here it is (I've removed / masked some sensitive info):

momon · 2019-07-18T22:48:43Z

@maclof how's it going? have you had a chance to look at this? Thanks!

momon · 2019-08-08T19:39:20Z

Hello @maclof just looking to check if you've had a change to look this over. Thanks!

maclof · 2019-08-08T20:00:03Z

@momon unfortunately not yet able to replicate on my clusters. Any chance of getting access to a cluster you’re working with? Perhaps setup a temporary gke cluster for a few days?

momon · 2019-08-08T20:06:41Z

@maclof happy to give you access to my sandbox, that's what it is intended for. Just unsure what details you would need exactly. Would the IP, usr and pwd for the actual cluster do so you can connect via the API or would you be needing me to add your service account to GKE?

maclof · 2019-08-08T20:09:31Z

@momon api access is perfect, if you could setup a demo pod that you’d try to use this with and I’ll get it all working

Drop the details via email to me at maclof@gmail.com

momon · 2019-08-13T00:41:55Z

@maclof sorry for the late response. I've sent you that email with the details.

Cheers,
Jose R. Lopez

maclof · 2019-08-13T18:22:46Z

@momon please try version 0.14.1, I've got it working within that pod you emailed me as an example :)

momon · 2019-08-14T15:12:56Z

@maclof can confirm it to be working now on 0.14.3 (which is what composer upgrade upgraded me to).

Much appreciate your help with this, I can now finish off some parts of the app.

Cheers,
Jose R. Lopez

maclof · 2019-08-14T15:54:44Z

@momon excellent news, let me know how you get on!

leonboot · 2019-11-14T15:02:31Z

I was very happy to see that version 0.14 introduced the ability to perform an exec on a pod, so I upgraded to the latest version (0.15.0 at the moment) and tried to execute a command. I'm having no luck, however :(

Running the following code results in an empty array being returned:

// This following line actually returns a Pod instance, not null
$pod = $this->client->pods()->setLabelSelector(['label' => 'somelabel'])->setFieldSelector(['status.phase' => 'Running'])->first();

$result = $client->pods()->exec($pod, [
    'command' => ['ls'],
    'container' => 'phpfpm',
    'stdout' => 'true',
    'stderr' => 'true',
]);

var_dump($result);

At first, I thought it might be because I'm talking to my cluster through a proxy (using kubectl proxy, with the --disable-filter flag). Then I created a service account, bound a role to it with enough privileges to perform a pod exec and tried again. Unfortunately the result was the same.

Are there any minimum requirements to the cluster that I'm talking to? In my case, it's a 1.12 GKE cluster.

maclof · 2019-11-14T16:15:59Z

@leonboot should work with GKE without issue, the cluster I tested was a GKE cluster.

If you can temporarily provide me access to the cluster and a code example to run, I'm sure I could get this working. If not you'll need to try and provide some more info to help debug what the problem is.

leonboot · 2019-11-15T08:47:42Z

@maclof I'm trying to create a test setup on our cluster with a service account which, annoyingly, works. I'm getting the expected output. So, on the bright side, the code works like a charm :) It's up to me now to figure out the difference between my live environment and the test setup. I'll share my findings if I think they might help others. Thanks for the input anyways!

maclof · 2019-11-15T09:10:25Z

@leonboot please share what the issue is if and when you find it, it could be something that we can put a check in place for to aid in debugging

leonboot · 2019-11-15T13:12:08Z

I've figured it out what the issue was: a trailing slash in the Kubernetes API endpoint URL 🤦‍♂

I used new Client(['master' => 'http://localhost:8001/']) to instantiate my client. While debugging, I added a var_dump($fullUrl) in the Client->sendUpgradeRequest() method, and noticed a double slash after the hostname (http://localhost:8001//api/v1/...) I removed the slash at the end of the endpoint URL and got the results I expected! 🎉

The trailing slash only seems to affect the React client. Normal requests all work as intended. So maybe it's worth to put some validation mechanism in place when performing an upgraded request?

meme · 2020-05-02T02:14:00Z

It seems that I can exec, but I'd like to interact with stdin to pipe files up to the node, like in kubectl cp. Is it possible to get a handle on this WebSocket connection so I can provide stdin input?

meme · 2020-05-02T03:22:41Z

Whipped something up (tar xf - -C /tmp/bar) using the code in the upgrade request function (assumes $data and $podName are in scope):

define("CHANNELS", [
    'stdin',
    'stdout',
    'stderr',
    'error',
    'resize'
]);

$connectorOptions = [
    'timeout' => 20,
    'tls' => [
        'cafile' => CA_CERT_PATH,
        'local_cert' => CLIENT_CERT_PATH,
        'local_pk' => CLIENT_KEY_PATH
    ]
];

$uri = 'wss://' . MASTER . '/api/v1/namespaces/default/pods/' . $podName . '/exec?command=tar&command=xf&command=-&command=-C&command=/tmp/bar&stdin=true&stderr=true&stdout=true';

$eventLoop = ReactFactory::create();
$connector = new WebSocketConnector($eventLoop, new ReactSocketConnector($eventLoop, $connectorOptions));
$connectionPromise = $connector($uri, ['channel.k8s.io'], []);

$socket = null;
$messages = [];

$connectionPromise->then(function ($connection) use ($data, &$socket, &$messages) {
    $socket = $connection;

    $connection->on('message', function ($message) use (&$messages) {
        /** @var Message $message */
        $data = $message->getPayload();

        $channel = CHANNELS[ord(substr($data, 0, 1))];
        $message = base64_decode(substr($data, 1));

        $messages[] = [
            'channel' => $channel,
            'message' => $message
        ];
    });

    $connection->send("\x00$data");
    $connection->close();
}, function ($e) {
    throw $e;
});

$eventLoop->run();

if ($socket) {
    $socket->close();
}

var_dump($messages);

Would be nice to have the ability to interact with the standard-in of the exec function, or perhaps a copy function for the pod object would be in order?

maclof · 2020-06-08T14:50:29Z

@meme apologies, for some reason I never saw your response to this.

Could you please open another issue for this? I'd love to get stdin support working correctly

meme · 2020-06-09T00:01:01Z

For sure: #66

maclof added a commit that referenced this issue Jul 4, 2019

Add support for upgrading a request for websockets, fixes pod exec no…

14b6a43

…t working #41

maclof closed this as completed Jul 4, 2019

maclof reopened this Jul 5, 2019

maclof closed this as completed Aug 14, 2019

meme mentioned this issue Jun 9, 2020

Feature Request: Interact with stdin (like pod exec) #66

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pod exec not work #41

pod exec not work #41

annProg commented Oct 1, 2018

samuelhautcoeur commented Feb 18, 2019

momon commented May 31, 2019

maclof commented May 31, 2019

momon commented May 31, 2019

momon commented Jun 10, 2019

maclof commented Jul 3, 2019

momon commented Jul 3, 2019

maclof commented Jul 4, 2019

momon commented Jul 4, 2019

momon commented Jul 5, 2019 •

edited

Loading

momon commented Jul 5, 2019

maclof commented Jul 5, 2019

momon commented Jul 5, 2019

maclof commented Jul 8, 2019

momon commented Jul 8, 2019

momon commented Jul 18, 2019

momon commented Aug 8, 2019

maclof commented Aug 8, 2019

momon commented Aug 8, 2019

maclof commented Aug 8, 2019

momon commented Aug 13, 2019

maclof commented Aug 13, 2019

momon commented Aug 14, 2019

maclof commented Aug 14, 2019

leonboot commented Nov 14, 2019

maclof commented Nov 14, 2019

leonboot commented Nov 15, 2019

maclof commented Nov 15, 2019

leonboot commented Nov 15, 2019

meme commented May 2, 2020

meme commented May 2, 2020

maclof commented Jun 8, 2020

meme commented Jun 9, 2020

pod exec not work #41

pod exec not work #41

Comments

annProg commented Oct 1, 2018

samuelhautcoeur commented Feb 18, 2019

momon commented May 31, 2019

maclof commented May 31, 2019

momon commented May 31, 2019

momon commented Jun 10, 2019

maclof commented Jul 3, 2019

momon commented Jul 3, 2019

maclof commented Jul 4, 2019

momon commented Jul 4, 2019

momon commented Jul 5, 2019 • edited Loading

momon commented Jul 5, 2019

maclof commented Jul 5, 2019

momon commented Jul 5, 2019

maclof commented Jul 8, 2019

momon commented Jul 8, 2019

momon commented Jul 18, 2019

momon commented Aug 8, 2019

maclof commented Aug 8, 2019

momon commented Aug 8, 2019

maclof commented Aug 8, 2019

momon commented Aug 13, 2019

maclof commented Aug 13, 2019

momon commented Aug 14, 2019

maclof commented Aug 14, 2019

leonboot commented Nov 14, 2019

maclof commented Nov 14, 2019

leonboot commented Nov 15, 2019

maclof commented Nov 15, 2019

leonboot commented Nov 15, 2019

meme commented May 2, 2020

meme commented May 2, 2020

maclof commented Jun 8, 2020

meme commented Jun 9, 2020

momon commented Jul 5, 2019 •

edited

Loading