Happy New Years! And some issues on trying to use the fill command.

[frosencrantz]

Happy New Year!

I tried something recently with VisiData and had some thoughts about issues I encountered, and thought worth mentioning.

On New Year's Day there was this interesting blog post from Michael Kennedy about numbers Python developers should know. It has a lot of tables in it, perfect for extracting with VisiData.

I found a couple things I consider issues when trying to look at this site. I really only looked at the biggest table Python numbers you should know (detailed version)

vd https://mkennedy.codes/posts/python-numbers-every-programmer-should-know/

The first column is the sparse “Category” column, because it is sparse I wanted to use the f (setcol-fill) command. I rarely use this, but have run it enough to realize you need to hold it correctly:

• The cells to update must be selected.
• The cells must be None, they can’t just be empty values.

So I select all the empty rows using , (select-equal-cell) and then use g= (setcol-expr) to set the value to None, status set 75 values = None. Then used f.

Did you catch my first mistake? I missed it. The setcol-expr reported success, but actually did nothing. Why? Because the Column type used for HTML tables is non-modifiable. To me, there are several issues:

• If it is not possible to modify the column, why isn’t that clearer in the UI, like Unicode 🔒, something in the ColumnsSheet, or maybe coloring. This is an issue VisiData wide.
• Why doesn’t VisiData report that the setcol-expr will not work on read-only columns?
• Why does the setcol-expr report success?!

So the actual steps are:

• ’ (freeze-col).
• , (select-equal-cell) on an empty cell.
• g= (setcol-expr) to None.
• f (setcol-fill)

To me the issues mentioned above are:

• VisiData should provide more feedback for non-writable columns.
  • Maybe visible in the UI.
• Column updating commands should provide helpful feedback when invoked on non-writable columns
• The setcol-fill should work on empty strings too. It is not clear to me how you can get None values in CSV or HTML sheets just by loading the data.

The other observation I have, that I don’t have a strong opinion about, is that the default columns used for HTML are non-modifiable. Should these be writable by default. Maybe? The default for CSV is writable.

[midichef]

That is a great idea, having the UI mark up columns that can't be written to! I often try to edit cells in uneditable columns, especially expression columns. Do you want to open an issue for this feature?

For fill behavior, I actually prefer that fill only works on None and not on empty strings. It lets me distinguish between the two. Even in a file that does not load to give None, using None as a special value is needed when I work with data that wasn't initially in the sheet, like columns that I create as intermediate steps in a calculation.

If fill only changes None, I can still target the empty string by translating it to None first. (by the way, you may want to use the shorter gzd instead of g= None.) But if we change fill so that fill changes the empty string too, it would take too much effort to emulate the current behavior by way of an intermediate column. You'd have to translate the empty strings to some special value that doesn't occur in the data, then translate that value back to '' in the result. (In researching special singleton values in Python that could be used for that as distinct alternatives to None, I found some interesting singletons, like the Ellipsis object aka ..., and NotImplemented.)

[frosencrantz]

TIL about gzd, which seems useful.

I've created a couple feature requests:

• [read-only columns]. Add a visual indicator that a column is read-only #2936
• [setcol-*] Column modifying commands should report errors if they can't update a column #2937

[frosencrantz]

@midichef how do you have options.some_selected_rows configured? It looks like the fill command has various different selections depending on whether vd is in batch mode or the values of that option.

visidata/visidata/selection.py

Lines 152 to 162 in 4af4abd

	@Sheet.property
	def someSelectedRows(self):
	'''Return a list of rows:
	(a) in batch mode, always return selectedRows
	(b) in interactive mode, if options.some_selected_rows is True, return selectedRows or all rows if none selected
	(c) in interactive mode, if options.some_selected_rows is False, return selectedRows or fail if none selected'''
	if options.batch:
	return self.selectedRows
	if options.some_selected_rows:
	return self.selectedRows or self.rows
	return self.onlySelectedRows

[midichef]

I do not change options.some_selected_rows, I leave it as the default (False). Thank you for pointing out that this option exists. It looks like when it's set to True, it applies various commands to all rows, when no rows are selected. It would save me the effort of doing gs to select all rows.

[midichef]

Also, as for how a csv file gets None by loading data, CSV rows get filled in with None when the CSV file had a blank line for that row, or when cell were left blank due to the absence of commas.

Like the third data row in this CSV, and every cell in the second column except for the one holding the value 3:

col1,col2
1
2,3

4

[midichef]

Good news! Visidata has a setting to enable the behavior you want, filling in both None and '' (the empty string). Set options.null_value to '', either by running visidata as vd --null_value='', or by adding to your .visidatarc:

options.null_value = ''
#Or you can instead apply the option only to specific types of sheets, like CsvSheet:
#CsvSheet.options.null_value == ''

setcol-fill always fills in cells with the value None, and also fills in cells holding whatever options.null_value is set to (which by default is None).

[midichef]

And if you ever want even more complicated fill behavior than that, you could override BaseSheet.isNullFunc() in your .visidatarc:

visidata/visidata/wrappers.py

Lines 9 to 15 in 02faae3

	@BaseSheet.api
	def isNullFunc(sheet):
	'Return func(value) which returns whether or not *value* is null.'
	nullv = sheet.options.null_value
	if nullv is None:
	return lambda v: v is None or isinstance(v, TypedWrapper)
	return lambda v, nullv=nullv: v is None or v == nullv or isinstance(v, TypedWrapper)