The sidebar and guide sheets

[saulpw]

So I've been experimenting with a few things recently, as we consider how VisiData 3.0 is going to evolve. I've been trying out some ideas on develop, and the implementations are imperfect but tolerable (for me at least), and there are some pretty interesting developments. I want to share them early, with those of you who understand the depths of VisiData (@frosencrantz @geekscrapy @ajkerrigan to name a few), and get your perceptions and feedback.

I do want to apologize in advance if you get caught by something changing or breaking. If you use VisiData and don't want to see every bump along the way to our next version, you probably want to stick to a released version (latest is v2.11) for the time being.

Status messages on sidebar

You may have noticed that the status messages have moved to the sidebar, and in fact the sidebar is quickly becoming a baseline feature.

I'm not sure if I totally like having the status messages on the sidebar, but I think it's better (and more eye-catching) than the single line status messages we're used to. For a brief moment there was a separate status message panel, but it was too 'busy' and instead we're going to double down on the sidebar. I'd love to hear your thoughts on this.

Sidebar can be colorized

I added the ability to colorize text in with colorbox() (like clipbox()) with e.g. [:bold blue on 246] or [:warning], the latter of which will use the value of options.color_warning. And then you revert to the base settings of the sidebar with [:]. So like [:bold]my bolded text[:]. [Edit: Basic markdown also works now].

Guide Sheets

But the thing that I think will be most helpful for people, are Guides. A "Guide" is a sheet that explains how to use a small part of VisiData.

As it stands now, any sheet with a .help property will show that text in the sidebar. This help text can be a basic overview of the feature and what's required to get it working, and what options are available for tweaking. The idea is that b will toggle the sidebar, and Shift+B will push a sheet with the sidebar in it (so you can read it in its entirety if the sidebar is too small).

So now some Sheets, like the DirSheet or the SubredditSheet (which I just dusted off and pulled in tonight), with the simple addition of help text, can be their own Guides. Derived sheets like FreqTableSheet would need a special FreqTableGuide, whose .help would explain in the sidebar how to set up a frequency table, and likely include some sample data that you could practice with.

In the new visidata.experimental module (for functional but half-baked features which are not automatically imported), we also have started a GuidesGuide, a toplevel outline of the ~30 Guides we'd want to have for a decent comprehensive tutorial.

So the current vision is for vd without any arguments to launch the GuidesGuide, which should be a reasonable starting point to learn about VisiData, and you can read as much or as little as you want, and then the rest you can learn as you go.

Oh and the points columns are tentatively there to track your progress in the tutorial data hunt. Use the commands, find the cell, collect all the points. I think it would be fun anyway :)

Comments, ideas, suggestions?

I'd like to involve y'all earlier in the design process, which is why I posted here in Discussions. Feel free to comment here, or sync chat with me on Discord or IRC.

And I'll send a foil VisiData sticker to anyone who takes the initiative to write a Guide!

[frosencrantz]

@saulpw this is very exciting. I am looking forward to trying the Guides, adding some gaming could be fun. I hope this helps make the VisiData more accessible for people. It doesn't look like you've published your changes yet.

[frosencrantz]

@saulpw I just realized you added the guides as experimental. I missed that. How do we use this? Do we just copy this to our plugin directory? Or can you just import directly from your .visidatarc? It does not look like it.

[saulpw]

You can import experimental.guides in visidatarc. But, the only thing there is this GuidesGuide, the toplevel guide that eventually will be opened instead of the DirSheet when vd is launched with no arguments. The other guides I have been adding to individual sheets. I'm not quite pushing them yet because you can't yet toggle the sidebar--I want to integrate that from vdsql here soon.

[frosencrantz]

There are several minor UI issues with the guides:

• There is no way to close a guide. Maybe if there was a close button like an X or some help text saying how to close the guide.
• If the terminal height is too small, the guide is truncated, but there is no visual indication that it has been truncated. There does not appear to be a way to scroll the guide if it has been truncated.

[anjakefala]

There is no way to close a guide.

I'm not quite pushing them yet because you can't yet toggle the sidebar

It's on the way. =)

Thank you as always for testing the cutting edge so thoroughly!

[frosencrantz]

Thank you as always for testing the cutting edge so thoroughly!

I really appreciate that. Thank you!

One more thing, the sidebar/guide doesn't absorb mouse clicks. When there are mouse clicks on the sidebar, Visidata behaves as if the sidebar wasn't there, and moves the cursor within the current sheet.

[saulpw]

Okay, I pulled in the basics from vdsql: b to toggle the sidebar on/off, gb to expand the sidebar into its own sheet. I'll push it soon.

[geekscrapy]

Just updated, and love the sidebar, or at least the concept! A dump of my thoughts after using it for a few sessions.

• Its great having vis to the full message. Love. it. I used to have to dive into ctrl+P waaay too much
• If a lot of output is dumped to the sidebar, it can take up a lot of screen real-estate
• It is somewhat distracting for a power user who reads other cues from the screen to know if their commands are running. I find myself diving over to read it as its bright, and the content changes a lot which means the shape changes a lot too. This could be fixed with option 2 from my suggestion below.
• I'd love to see an option to have 3 levels of the guide:
  1. Full guide, as now
  2. Show a small subset of popular key combos for use on the current sheet. I think this would assist new users (and returning) a
     lot. Could this list be made up of the actual users top X commands for that sheet rather than a static list picked by the dev team? I think that would be massively powerful. Or maybe the flip-side of that, they are commonly used, therefore they dont need prompting?
  3. Off (but with the option, using b, to quickly enable level 2)

[saulpw]

Thanks for the feedback! I was hoping you'd chime in.

In my mind, we are already doing your option 2, as the few help/guides we've put together are just short sentence or two and then some subset of most useful commands. But maybe you mean something even smaller. Can you give a comparative example with a current sheet's help, and what you would prefer it said instead?

We could also have a way to toggle the sidebar per sheet, or per sheet type, and even persist this setting. So that once you "dismiss" the sidebar on e.g. the DirSheet, it will be invisible by default for future sessions, unless you manually use b to toggle it back on.

[saulpw]

I love the idea of different levels, too. Maybe b cycles through a set of possible disp_sidebar_fmt options. But then how to make that list more configurable?

[geekscrapy]

For the option 2, I'm thinking something like this in the sidebar:

(Maybe center justify the keystrokes so they are easier to see, i.e. select-col-regex)

|longname          |keystrokes  |
|------------------|------------|
|copy-cell         |zy          |
|freq-col          |Shift+F     |
|hide-col          |-           |
|key-col           |!           |
|rename-col        |^           |
|search-col        |/           |
|select-col-regex  ||           |

These suggestions could change depending on the number of times you use each command, but the number of suggestions and the size of the box is always static. Maybe this is facilitated by the command history file if present?

In regards to dismissing it per sheet: After I had read a few of them for different sheets, I wanted to disable globally. So maybe a global option is required? At least for vd old timers?

I love the cycle of the different levels, it would enable people to quickly test out which option they like best, and also quick and easy to for those that forget something to quickly get a prompt for it. However, I would still like a way to disable/enable and retain the same level

[saulpw]

Well us oldtimers should be able to just put options.disp_sidebar_fmt='' in the .visidatarc (this is what b does currently, it toggles between "{help}" and "")

[geekscrapy]

Well us oldtimers should be able to just put options.disp_sidebar_fmt='' in the .visidatarc (this is what b does currently, it toggles between "{help}" and "")

Yea, I just wanted to differentiate between an on/off toggle and level swapping 👍

[ajkerrigan]

I love the idea of a sidebar. In experimentation mode I'm likely to overuse it or do weird stuff just to get a feel for it. One of the first things that occurred to me (recapping a separate discussion) is to have sidebar content that varies based on the sheet. If I open a Cloud Custodian output directory in VisiData for example, it feels neat to show some summary information for the policy that produced the output.

Hacky sample up here - handwaving past the clunky implementation I think the idea feels useful.

Dynamic content aside, sidebars for more static guides also seem super useful (both for vanilla vd and also for plugin authors/users!)

[geekscrapy]

I love this idea. For flat files it's a little harder, but some details from the files DescribeSheet would be amazing! Some file formats like Parquet + sqlite store some summary data (I think), which could be shown, and loaders that use APIs (like the one that @ajkerrigan is using) can also quickly get this information without having to load an entire file.

Alternatively, the default_sample_size could be used in flat file cases, although safeguards should be put in place for those that have default_sample_size set to 0 (me 👈😀). But this may provide a quick win for most files?

If this was a thing, it might be best to include a "loading screen" as part of the user feedback while these things are retrieved.

[saulpw]

include a "loading screen" as part of the user feedback while these things are retrieved.

The individual items would just be empty strings until the item can resolve. DescribeSheet already deals with large sheets in an async way, you could pretty easily create some cached properties that tap into that code and display them in the sidebar.

I can see this being really sweet :)

[daviewales]

I really like the new status sidebar which shows all the status messages and automatically hides when I interact.

The help sidebar is cool, but I'm not so sure about it being on all the time.
I think I'd want the help sidebar turned off by default, but available with a toggle, and the auto-hiding status sidebar enabled.
Just checked, and options.disp_sidebar_fmt='' gives me exactly this behaviour. 😀
Is there a way to toggle the status sidebar back? Like ctrl+P, but bringing the status sidebar back rather than the status sheet?

The guide sheet on first startup looks great. And I like the gamification.
I assume if you want the old DirSheet behaviour you would just use vd .

[saulpw]

I think I'd want the help sidebar turned off by default, but available with a toggle, and the auto-hiding status sidebar enabled.

Working on it! Should be on the b key here soon.

I assume if you want the old DirSheet behaviour you would just use vd .

Yes, exactly. Doesn't seem too onerous :)

[frosencrantz]

I'm looking forward to seeing how the guide will work and how it could help with a guided tour or a self-directed journey. Like could Jeremy's tutorial be converted to a guide, helping with loading the sample data, and helping go through the steps that Jeremy has in his tutorial. Or could someone share their VisiData solution to the Hanukkah of Data with a guide?

I'm excited to see @ajkerrigan and @geekscrapy suggestions and experiments. It does feel like there is a lot of potential for what could be put in the sidebar. I like the idea of showing the top-N commands. Currently the history file doesn't save keystrokes or menu items, but it does for Space/exec-logname. I was thinking of something like that for dynamic menus when Saul introduced the menu system. @geekscrapy's markdown-style table would be very useful, if it could handle a markup-style table format, and VisiData could fill it in and display it in the sidebar.

In some ways I think these might be useful as plugins, as long as VisiData provides some very powerful formatting tools, like markdown/Python-fstring, users would have flexibility to add what they want.

So far I'm a big fan of the sidebar, I've already mentioned that it needs more visual indicators. One thing I keep wondering is if it makes sense for the sidebar to not be a popup/overlay, but actually a separate tile within Visidata. Think i3wm or any tiling window manager, where the windows are within their own non-overlapping tiles in a grid. One nice thing about it not being constrained, is that it can show data of different sizes without using more space than needed. But it can hide data, that requires you to turn off the sidebar to view. Or maybe it would be possible to move the sidebar somehow.

Using guides:

It looks like you can load the guide by adding this to your .visdatarc file: import visidata.experimental.guide

But it currently only has the top-level guide, which you can see via the open-guide command.

[frosencrantz]

@saulpw the sidebar functionality is enough that the ideas you talked about in the first message could be done. I was wondering what you think will be the next steps for this. Would it be possible to do an interactive version of Jeremy’s tutorial for a given dataset? Or a Hannukah of Data solution?

[saulpw]

A Hanukkah of Data walkthrough is a neat idea. I think you would get a noahsrug.zip file that had the data and the story text, and when you loaded up the datasets, different sidebars would show up depending on where you were in the game. So for the first puzzle, it would offer that might use regex (linked to the Regex Guide) or a custom Python function (link to the Column Expression Guide). You'd still have to figure out the answer, but you'd be effectively guided through the process. And then when you input the right answer (via custom command, which also can be a clickable item in the sidebar), it decrypts the next story, pushes a Canvas with the revealed art and a sidebar with your stats and how to get to the next puzzle. Your game state could be added to the .zip file at every puzzle, so if you quit and come back you can resume where you left off.

How's that sound?

[frosencrantz]

That would be brilliant! I really like the ability to resume from a saved state, and the ability to show the custom art.

To me that would be a compelling follow up to your PyCascades lightning talk from 2018.