Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support subprocess.Popen() style command execution for advanced uses #210

Open
Tracked by #169
jacobtomlinson opened this issue Nov 21, 2023 · 1 comment
Open
Tracked by #169
Labels
enhancement New feature or request kr8s

Comments

@jacobtomlinson
Copy link
Member

jacobtomlinson commented Nov 21, 2023

In #211 we added support for Pod.exec() which behaves like subprocess.run() and runs a command in a Pod and gathers the results.

It would also be useful to add a way of calling exec that behaves more like subprocess.Popen() and asynchronously opens the process. This would allow for finer control of communicating with the process via stdin/stdout/stderr.

We could call this command Pod.exec_open() which would return an Exec object.

@jacobtomlinson jacobtomlinson changed the title Support subprocess.POpen() style command execution for advanced uses Support subprocess.Popen() style command execution for advanced uses Nov 21, 2023
@jacobtomlinson jacobtomlinson added enhancement New feature or request kr8s labels Nov 21, 2023
@jacobtomlinson
Copy link
Member Author

jacobtomlinson commented Nov 21, 2023

One use case for this would be to stream files to/from a Pod.

With the current Pod.exec() implementation we wait for the remote command to finish before returning. Therefore if we want to mimic the behaviour of kubectl cp we need to call a tar command via exec which loads whatever files we are copying into memory, then we flush those files back to disk.

# Works today!
import kr8s, io, tarfile

pod = kr8s.objects.Pod.get("my-pod")

# Archive the contents of /tmp/foo in the container and pipe through stdout into a bytes buffer
# then extract those files from the buffer to /tmp/bar on the local system
with io.BytesIO() as archive_buffer:
    pod.exec(["tar", "cf", "-", "/tmp/foo"], stdout=archive_buffer)
    archive_buffer.seek(0)
    archive = tarfile.TarFile(fileobj=archive_buffer)
    archive.extractall("/tmp/bar")

However if the contents of /tmp/foo in the container are large we will run out of memory doing this.

Doubly because we store the stdout twice, once in the CompletedExec.stdout attribute and once in the BytesIO buffer Update: You can avoid this now with #213

It would be better if we could start the process with exec_open and use the stdout attribute as a readable stream object so that tarfile can start reading while it is still being written to which would help keep the buffer small.

# Potential solution
import kr8s, io, tarfile

pod = kr8s.objects.Pod.get("my-pod")

# Archive the contents of /tmp/foo in the container and pipe straight through to tarfile to
# then extract those files
with pod.exec_open(["tar", "cf", "-", "/tmp/foo"], stdout=archive_buffer) as exec:
    archive = tarfile.TarFile(fileobj=exec.stdout)
    archive.extractall("/tmp/bar")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request kr8s
Projects
None yet
Development

No branches or pull requests

1 participant