Data Cleaning Tip: Bulk Corrections with ge

[frosencrantz]

I recently used the ge / setcol-input (formerly edit-cells) command to clean some data for a friend. While I’ve used this command before, I appreciated how helpful it was this time and felt I should share. I think many users are already aware of this command.

This command is particularly useful for cleaning up inconsistent values. A common case is when there are inconsistent values for the same logical item (e.g. “visidata”, “Visidata”, “VisiData”, “vd”). The ge command allows you to set the contents of the current column for the selected rows to one value.

Here is how to use this command:

• The initial input used is from the current cell. In the example of the “visidata”, you would:
  1. Select the rows with inconsistent values.
  2. Move the cursor to a row with the correct/desired value. It doesn’t have to be a selected row.
  3. Type ge and then hit Enter. You can make edits before hitting Enter.
  4. All selected values in the column will be updated to the corrected value.
• Use ge with a FrequencySheet. Edits made to cells in the FrequencySheet will be propagated back to the original sheet, if the original column is modifiable. The FrequencySheet will make it easier to see the unique values in the original sheet.

The data I was cleaning has been scraped from a PDF file using AI. The data had some errors in the original data, and some added by the AI scraping. I found it very effective to use the FrequencySheet to identify inconsistencies and then the ge command to unify them, which made the cleanup process easier and faster.

[midichef]

I didn't know about ge. Combining it with a Frequency sheet is very helpful! Thank you!

[saulpw]

Thanks for sharing, @frosencrantz, you made my day. I appreciate your keen sense of how VisiData works (and sometimes how it should work, thank you for that too).