@saulpw saulpw released this Sep 25, 2018 · 6 commits to stable since this release

Assets 2

This is largely a bugfix and performance release. It was originally planned as a patch release (v1.3.2), but some features and interface changes crept in, so it is officially v1.4.

  • [pyobj] visidata.view(obj) and visidata.view_pandas(df)
  • [pandas] -f pandas loads file with pandas.read_<ext>
  • [date] date minus date now gives float number of days instead of seconds
  • [vdtui] internal colorizer API changed significantly
  • [vdtui] timeout and redraw only if other threads active

As usual, a more complete list of changes and features is in the CHANGELOG.

See the ROADMAP for the planned future direction of VisiData.

@saulpw saulpw released this Aug 31, 2018 · 95 commits to stable since this release

Assets 2

We found some issues with 1.3 (aggregators interacting with nulls/errors, primarily) that we didn't want to let sit until 1.4. So we fixed those issues and a couple others, and added a few 'minor' features. This patch version should be a definitive improvement over the base 1.3 version.

The complete list of changes is in the CHANGELOG. Here are the new options and features:

  • new extend join type keeps the type of the first sheet, extending it with columns from the other sheets
  • rename-sheet command (thanks to @jsvine for suggestion; what default keybinding should it have?)
  • [DirSheet] add reload-rows (gz^R) to undo modifications on all selected rows
  • remove all options.foo_is_null and add options.null_value
  • options.save_errors (default True) to include errors when saving
  • add options.json_indent for json pretty-printing
Aug 20, 2018
[test] Add json goldens since the save has been fixed
Assets 2

v1.3

It's been a productive 3 months since v1.2. The largest effort in this release was a commands/options reworking, which will hopefully pay dividends in the future. Many other so-called improvements were made as well. Here's a list of most of them:

commands

  • All commands were thoughtfully renamed, and the command longnames should be largely stable now.
  • commands.tsv is an exhaustive list of commands and their attributes and side effects.
  • The manpage has moved to Ctrl+H, which should be its final resting place.
    • F1 will still open the manpage if the terminal doesn't intercept it.
    • but z? has been repurposed (see below).
    • Note that because iTerm reports Ctrl+H as Backspace, these help commands are also available by using Backspace (backspace for help, a new trend).
  • z Ctrl+H opens a list of commands available on this sheet.
  • Keybindings and longnames are separated out. The cmdlog now records longnames as well.
  • See the new keyboard layouts page (thanks to @deinspanjer for inspiration).

changes to existing commands and options

  • The experimental menu system (was Space) has been removed.
  • Now Space (exec-longname) executes the command for the input longname (tab completion of available commands is supported). (This function was previously bound to Ctrl+A).
  • options.wrap (for TextSheet wrapping of lines) now defaults to False.
  • R (random-sheet) opens a new sheet instead of selecting random rows (reverting to former behavior).
  • za (addcol-empty) asks for column name
  • zd (delete-cell) moves value to clipboard ("cut", like other delete commands)

options

  • options can now be set on specific sheet types or even individual sheets. Shift+O opens options for the current sheet type, and g Shift+O opens the global options sheet.
  • See available colors by pressing Space and then typing the longname colors.

new safety options

  • Error messages are sorted before informational status messages, and colored by color_error and color_warning (thanks to @jsvine for suggestion)
  • options.quitguard (default False to keep old behavior) if True, will confirm before quitting last sheet.
  • The sheets-graveyard (gS) command opens a sheet that shows all discarded (but "precious") sheets. These are stored as weak references so they will be garbage collected eventually, but can be resurrected from the graveyard sheet until then.
  • options.safety_first (default False) makes loading/saving more robust, likely at the cost of performance which can become significant in large files.
    • Currently, only removes NULs from csv input.
  • options.tsv_safe_char is split into tsv_safe_newline and tsv_safe_tab.

new power features

  • z; (addcol-sh) adds new columns for stdout/stderr of a bash command, which uses $colname to substitute values from other columns (whole arguments only, so far).
  • z| (select-expr) and z\ (unselect-expr) select/unselect by Python expression (thanks to @jsvine for suggestion).
  • z/ (search-expr) and z? (searchr-expr) to search forward/backward by Python expression.
  • gI (describe-all) describes all columns in all sheeets (like gC (columns-all)).

nice things

  • The g(, z(, and gz( variants of ( ('expand-column') are filled out.
  • z# sets type of current column to len.

new loaders

minor changes

  • System clipboard command detection is more portable (thanks to @chocolateboy for the PR).
  • date supports adding a number of days (or like foo+6*hours, foo+9*months, etc).
  • Hidden columns are darkened on columns sheet.
  • Exceptions are rolled up properly.
  • options.motd_url now uses https by default (thanks to @jsvine for the warning).
  • [DirSheet] mode is editable (set to octal like 0o0644).
  • [internal dev] ProfileSheet is improved.

known issues

  • cmdlog replay with a Ctrl+S (save-sheet) to an existing file gets stuck in an infinite loop when options.confirm_overwrite is on.
  • After renaming a file on a DirSheet, Ctrl+R (reload-sheet) is required to refresh the ext column for that row.
  • n/N (next-search/prev-search) won't continue a previous search-expr and searchr-expr.
  • show-aggregate with mean errors on int columns.
  • Contracting (with )) a previously expanded column on a dup-ed (with ") sheet results it in disappearing on the source sheet.

Thanks to everyone who contributed to this release! As always, feedback and suggestions are welcome and appreciated.

@saulpw saulpw released this Jul 7, 2018 · 673 commits to stable since this release

Assets 2

@deinspanjer (issue #164) discovered that VisiData doesn't work with Python3.7 due to our use of the 'async' identifier, which is now off-limits as it is a formal keyword as of 3.7. The only change in 1.2.1 is changing 'async' to 'asyncthread' in order to work with Python 3.7.

Thanks to the illustrious @anjakefala for getting this release out quickly and without incident (knock on wood).

@saulpw saulpw released this May 3, 2018 · 687 commits to stable since this release

Assets 2

Here are the major feature enhancements for v1.2. Please see the CHANGELOG for the complete list.

Major features

  • DirSheet enhancements (modify file metadata, move/delete files)
  • multisave (single file multisave to html, md, and txt format, or into a directory for other formats)
  • prototype macro system with z^S on the commandlog
  • New conda package

New commands and options

  • T to push derived sheet with transposed rows/columns
  • zs/zt/zu to select/unselect/toggle rows from top of sheet to cursor (thanks @SteveJSteiner for the suggestion)
  • gzs/gzt/gzu to select/unselect/toggle rows from bottom to cursor
  • gv to unhide all columns
  • gM to melt into multiple value columns
  • g* to transform selected rows in place
  • z</`z>' to move up/down to next null cell
  • ^A to execute a command by its longname
  • gD to open the directory at options.visidata_dir (default ~/.visidata)

options

  • options.cmdlog_histfile to specify a file to auto-append commandlog to (default empty means disabled)
  • options.tsv_safe_char to replace tabs and newlines when writing .tsv format (default empty means disabled, for faster saving)
  • options.error_is_null to count errors as null (default false)

New supported formats and sources

  • sas7bda (SAS; requires sas7bdat)
  • xpt (SAS; requires xport)
  • sav (SPSS; requires savReaderWriter)
  • dta (Stata; requires pandas)
  • bz2 and xz (LZMA) compression (thanks @jpgrayson)

@saulpw saulpw released this Mar 6, 2018 · 882 commits to stable since this release

Assets 2

This is the first release since 1.0. The major additions and changes:

  • All external loader dependencies have been removed from the PyPI, brew, and Debian packages. This will make for much faster initial installation.

    • The json, csv, tsv, and sqlite loaders are still available with the base installation as they are supported by the Python standard library. Other loaders will need their dependencies to be installed manually. See requirements.txt for which external packages to install for each loader.
  • An experimental command menu is now available via Space. The goal is to make it easier for people to explore the available functionality, and to try commands without knowing their keybindings.

    • Use the standard VisiData movement keys (h/l/Arrows, q to back out)
    • The help strings are shown during navigation, along with the available keybindings, so this will hopefully help people learn the keybindings as they use commands, without having to break their flow to refer to the manpage.
    • Commands are organized in a hierarchy according to their longnames. This hierarchy may change in future releases, so these longnames are not yet stable enough to use for command aliases or in cmdlog replay. The actual command keys are more stable and should be preferred for the time being.
    • If you play with the menu system, please let me know what did or didn't work for you!
  • The Y command series was added to copy ("yank") to the system clipboard (with options.clipboard_copy_cmd, set to pbclip for MacOS by default) to conveniently paste data in any supported text output format.

    • Y copies the current row, zY copies the current cell, and gY copies all selected rows (or all rows).
    • This mirrors the existing y command series, which yanks to VisiData's internal clipboard.
  • - now works as a filename to specify stdin/stdout. Useful especially in batch mode to dump final sheet to stdout (-b -o -).

New supported formats:

  • markdown (md) is now supported for saving (but not loading, yet) to an org-mode compatible table format.

  • .png files can now be loaded and saved, and crudely viewed on the canvas with an overloaded .. The pixels can be edited on the source sheet like any other data.

  • .ttf and .otf (font) files can be loaded and viewed on the canvas. This is super useful for just about no one, but it was a great excuse to implement Canvas.qcurve().

These commands have been added or changed, and are expected to stay in future versions:

  • The za command adds an empty, editable column to any sheet. gza adds N new columns.

  • The ( and ) commands will expand/collapse list/dict columns (e.g. in nested json).

  • In the canvas, d deletes points from source sheet that are contained within the cursor. gd deletes all points shown on the screen.

  • The !@#$%-_ special actions on the Columns Sheet and Describe Sheet have been removed. They were sometimes convenient, but more often made it difficult to interact with the Columns Sheet itself.

    • The g forms of these commands are still available and will operate on the source columns (with the exception of g_, which now works consistently on all sheets as expected).
  • The Shift+Arrows are aliased to HJKL (though these may not work in all environments).

These additions are more experimental and may not stay in future versions:

  • The Backspace command drops the current sheet (like q), and also scrubs its history from the cmdlog.

  • ENTER is now aliased to modify-edit-cell by default.

Other minor changes:

  • search and select with no input now uses the most recent input. n/N do this already for row search (as before), but the new behavior works more like standard vim/less/etc tools, and also applies to non-row search (like c).

  • Many sheets (pivot, describe, melt, and many loaders) have improvements and bugfixes to make them even better.

Finally, some other news:

  • VisiData has been accepted into Debian for the next release! It is currently available if you've added the unstable repo; install with apt install visidata.

  • Jeremy Singer-Vine has put together a great tutorial for people who want a smoother path to start using VisiData.

Thanks to everyone who contributed to this release! As always, feedback and suggestions are welcome and appreciated.

@saulpw saulpw released this Jan 26, 2018 · 1073 commits to stable since this release

Assets 2

VisiData v1.0 (2018-01-25)

VisiData 1.0 has been officially released! Changes since the 0.99 release candidate:

  • a complete overhaul of the website and documentation;
  • a 2x across-the-board performance improvement for async threads (from disabling profiling by default);
  • removed support for .visidatarc in the current directory and via XDG; only $HOME/.visidatarc is used now;
  • many bugfixes, both functional and cosmetic;
  • removed the "visidata[full]" package from PyPI;
  • a homebrew package (so you can now do brew install saulpw/vd/visidata on MacOS), and an apt package that is under review for inclusion in Debian.

This is a huge milestone for VisiData. I devoted my sabbatical (the entire year of 2017) to creating an open-source tool that would be useful for myself and other terminal users, and it was very important for me to release a stable and lasting 1.0 so that it could be set aside when I resumed working.

We could have gotten away without certain features and with fewer supported formats, but there is such breadth and depth included in 1.0 (as shown by demos like this one at the PyCascades 2018 conference), that I'm hard-pressed to choose which features should have been omitted.

I owe a great deal to many people who have helped and supported me through this process. VisiData would not be what it is without them. The list of contributors includes everyone who has submitted to the repository, but there are some other groups to whom VisiData and I are eternally indebted:

  • the Recurse Center, for maintaining an awesome space and incredibly supportive community that I am lucky to have become a part of;
  • Office Nomads, my local co-working space where I could work without turning into Jack Torrance;
  • and the Python community, for creating such a broadly useful language and ecosystem, without which VisiData would not be nearly as powerful.

Finally, a very special thanks to Anja for her incredible help in putting together the website, documentation, tests, and various releases.

Share and enjoy!

Saul

@saulpw saulpw released this Dec 23, 2017 · 1108 commits to stable since this release

Assets 2

No changes will be made on stable before 1.0 except for targeted bugfixes with minimal likelihood of regressing anything else. If there are any more bugfixes to be applied, they will be cherry-picked from continuing develop. Either v0.99 or v0.99.1 will become the formal 1.0 in a few weeks.

User-facing changes in this version:

  • tab completion for filename and python expressions
  • load .html table elements (requires lxml)
    • http urls are now usable as sources (requires requests)
  • save as .json
  • ^W to erase a word in the line editor
  • gC views all columns on all sheets
  • median aggregator
  • v now 'visibility toggle' (moved from w)
  • --version (thanks to @jsvine)
  • options.use_default_colors (thanks to @wavexx)

@saulpw saulpw released this Dec 5, 2017 · 1238 commits to stable since this release

Assets 2

v0.98.1 is a patch release that fixes a couple of minor bugs. The primary change in this release, however, is that the 'visidata' package on PyPI no longer includes all of the loaders' dependencies by default.

This is a minor hassle for first-time users if they want to use certain formats, but since the goal is eventually supporting every possible data format, installing all dependencies for all users by default is not tenable. The base VisiData already includes support for tsv, csv, fixed width, sqlite, graphs, and more. Most users will only need one or two additional dependencies for their data sources.

To install VisiData with all dependencies for all loaders:

pip3 install "visidata[full]"

Of course, you can install just the dependencies you need. See "SUPPORTED SOURCES" in the manpage for which packages to install.