Open DataFrame in VD in Different Tmux Pane

[mohelm]

Hey guys -

I would be interested in transferring a pd.DataFrame from iPython running in a given tmux pane to another Tmux pane via some command - do you have an idea if that could be possible somehow?

Background is I am working in Neovim and if I open visidata from within a Neovim termianl running iPYthon it is not super comfortable to work with...

Thanks.

[saulpw]

Hi @mohelm, this is an interesting question. I don't know how it's possible, but I'll ask around and see if anyone has some ideas.

[mohelm]

Hey @saulpw - thanks for the reply. I already managed to find out that something like this:

x = df.to_csv()
!tmux split-window "echo '{x}' | vd -f csv"

is possible. But: the conversion to csv is not that appealing and also that only works for small dataframes as otherwise you run into a limit regarding the number of arguments shell commands can accept.

[saulpw]

It's a good start, and I think it's a decent approach. I'm thinking, what if Ctrl+S (save) could take an input like .arrows|tmux split-window "vd -f arrows"? Then the intermediate binary format would be more efficient, and not be limited by the CLI.

[mohelm]

hm - I guess I am not fully understanding it. Is Ctrl+S an IPython operation (I would like to move the dataframe "out of Ipython")? I guess I have to somehow save the dataframe to some buffer and then pass the buffer to visidata in the other tmux window/pane via something like this?

For instance I tried (example from the pyarrow documentation):

import pyarrow as pa

data = [
    pa.array([1, 2, 3, 4]),
    pa.array(['foo', 'bar', 'baz', None]),
    pa.array([True, None, False, True])
]


batch = pa.record_batch(data, names=['f0', 'f1', 'f2'])

sink = pa.BufferOutputStream()

with pa.ipc.new_stream(sink, batch.schema) as writer:
   for i in range(5):
      writer.write_batch(batch)

buf = sink.getvalue()

But then I am unsure how (if possible at all to move this to visidata in the seperate pane/window via something like:

.arrows|tmux split-window "vd -f arrows"

[mohelm]

Hey - thanks again.

It looks good. Can you elaborate a little bit what .arrows refers to here?

Also by Ctrl+S to what save are you referring there?

Thanks.

[saulpw]

.arrows is the Arrow IPC Streaming format. Ctrl+S is the normal save operation.

[mohelm]

also I tried something like this:

import io
buffer = io.BytesIO()
df.to_feather(buffer)
buffer.seek(0)
x = buffer.read()
!tmux split-window "{x} | vd -f arrow"

But not sure if that goes into the right direction - in anycase x is not recognized as arrow file format (which I guess is expected) - you somehow have to move the buffer right?

[saulpw]

Oh this wouldn't be meant for the public internet, more for intranet or on your own local computer.

[geekscrapy]

Yup, I am just well aware that people see a cool thing (like this) and will expose it 😅 so big warnings need to be had 😉

[ajkerrigan]

I dig the idea of big warnings @geekscrapy 👍 . When I started fiddling with a super basic form of remote control I used unix sockets in an effort to avoid some classes of oopses.

It's not exactly polished or powerful, but combining that sort of remote control with some shared storage seems like a useful thing to poke at. This felt promising:

I imagine it would be better and smarter to use memory mapped files but I couldn't figure that out 🙈

[geekscrapy]

Yea, that's kind of the thought I had @ajkerrigan , although the lower panel would be another instance of VD looking at the same data source, or even just shows (in detail) a full dump of the currently selected row (for e.g.)

This would allow you to move these separate windows around the screen and customise to your needs

[ajkerrigan]

Yeah that sounds neat. If this was a useful enough pattern, I could see refinements like:

• Use memory maps (or leave Ray running in the background, I can see pros/cons of each)
• Create vd commands for sending a sheet to another instance

I can see the "live update" cases being a bit thornier. That feels like the spot where the effort/reward tradeoff gets worse unless you really need that. But if you're treating any cross-process data as "fire and forget" that seems reasonable even if it's multiple zoom-in views of a main sheet 👍 .

[dufferzafar]

I wanted something similar but for integration with pgcli; I currently have vd -f csv set as my pager in pgcli.conf, so any queries I write open visidata with the results.

I too wanted a split screen view like:

Running queries on left should update the view on right!

[dufferzafar]

@saulpw You discussed an approach above where you talked about .arrows | vd but this would launch a new visidata process.

Would it make sense for vd to have a "control" mode? As in you start vd -server an empty visidata TUI opens but it starts listening to commands on an admin server TCP port & takes those actions?

Some other applications that have this control server:

tmux itself!

kitty terminal: https://sw.kovidgoyal.net/kitty/remote-control/

[ajkerrigan]

Hey @dufferzafar - first, I love the idea of using visidata as a pgcli pager 😍 . Can't believe I never thought of that before, heck yes though!

Second, I know the idea of remote control has come up before and does sound neat. This thread got me experimenting with a plugin that I've put up here. It's certainly rough/experimental/etc, but it does still feel neat.