@shundhammer shundhammer released this Jun 4, 2017 · 43 commits to master since this release

Assets 2

Overview

The QDirStat project proudly presents its latest release: QDirStat 1.4.

This release brings a number of new features as well as some bug fixes:

  • File size statistics window with histogram

  • Filling the gaps in the treemap (directories with lots of small files)

  • Checked code with Coverity and fixed all issues it pointed out

  • Improved support for file and directory names with UTF-8 special characters (Qt 4 only issue)

  • Improved support for directories that have read, but not execute permissions

Details

New Features

File Size Statistics Window with Histogram

File Size Histogram Screenshot

This is a whole new kind of statistics in QDirStat showing how file sizes are distributed. You can start that for any selected directory (menu View -> File Size Statistics or F2) or from the File Type Statistics window if you select any filename suffix (extension) there and then File Type -> Size Statistics (or F2). In the latter case, you can see how large all your photos (.jpg), your videos (.mp4) or whatever are.

This new statistics window deals with a lot of things you might have come to hate at school or at university, and which your math teacher or your statistics professor never explained in a way that mere mortals can understand, so there are explanations of those things as a bonus. There is a landing page for that in that new window:

File Size Help

Or you might use this as a starting point.

Everybody thinking "I have no clue what this is all about", please have a look at the Median, Quartiles and Percentiles Explained document to convince yourself that really everybody can easily understand this.

There is also a GitHub issue to discuss this; comments are welcome. Feel free to reopen this for any discussions about that topic.

Filling the Gaps in the Treemap

GitHub issue #58 shows that users feel under-informed when there are grey areas in the treemap. The explanation is simple: Treemap tiles are only displayed when they have at least a certain minimum size (by default 3 pixels). Otherwise the treemap just gets cluttered with tiny things that don't show any information whatsoever.

The remaining space is taken by its parent directory's tile. They were rendered just flat grey which makes their boundaries against each other invisible, thus giving the impression that there is nothing.

So there were some experiments with visible borders, but that completely destroyed the visual impression of the treemap because those borders were everywhere. Fill patterns also didn't help: They were just ugly, and there was no way to tell where one directory tile ends and where the next one starts.

The next approach was using gradients. The first impression was good, but then it turned out that it was hard to tell which item was a (now over-emphasized) directory and which one a large file. Locating large files deep inside the directory hierarchy is the major strong point of the treemap visualization, so giving that up did not seem like a smart idea. After playing a bit with the gradient parameters (toning it down and giving it just a little blueish tint) this was the result:

Using Gradients

This appears to be a good compromise.

Of course this is configurable: Edit ~/.config/QDirStat/QDirStat.conf:

[Treemaps]
...
DirGradientEnd=#707080
DirGradientStart=#606070
...
UseDirGradient=true

Beware that QSettings sorts the entries alphabetically, so the start is after the end (how philosophical...).

Bugfixes

  • Fixed problem with directories that have read, but not execute permissions thanks to slodki:

    In that case, you'd get a warning in the log for every entry in such a directory, and it would get the wrong icon (a locked folder) and null values for all fields. Now checking for execute and read permission of the directory in advance and not even trying to read any contents (because the values would be bogus anyway).

  • Fixed GitHub Issue #61: Files and directories with UTF-8 special characters in the name not read correctly when built with Qt 4.x

    This happened only when QDirStat was built against Qt 4.x, but that is the default for NHellFire's PPA, so this affects all Ubuntu users who installed QDirStat from that PPA.

    Thanks to slodki who pointed this problem out!

Other Improvements

  • Checked code with Coverity

    Coverity offers free static code analysis for Open Source projects. This is really an outstanding tool, and it does a really thorough analysis.

    You might be glad to hear that while it complained about some minor things, there was not a single issue that would have been user relevant (let alone any security problems - there were none). Still, I did my best to fix the small complaints it had, and now we are down to zero outstanding defects reported by Coverity in QDirStat's 130,000 lines of code.

  • More consistency between file type and size statistics

    Like the new file size statistics window, the older file type statistics window now uses the currently selected directory (in the tree view), not always the tree's toplevel directory. If nothing is selected, it still uses the toplevel directory.

    That means that F3 no longer toggles that window, but re-populates it with the currently selected directory instead. This is consistent with the F2 view.

    Of course, the "Locate Files by Type" window now is also restricted to that subtree which actually gives it better functionality if you know that you want to locate files only there.

    This means that you can now select a subdirectory in the tree, open the file type statistics for it (F3 key), then select any filename extension (suffix) in that window and then open the file size statistics (F2 key) for that file type in that subtree.

    Previously, you would have to start QDirStat to show only that directory, then open the file type statistics window (F3), then the file size statistics window (F2) from there.

@shundhammer shundhammer released this Mar 5, 2017 · 167 commits to master since this release

Assets 2

Overview

The QDirStat project proudly presents its latest release: QDirStat 1.3.

This release brings a number of new features as well as some bug fixes:

  • File type statistics window

  • Locate files per directory by file type

  • Display path and size of the current item in the status line while hovering over a treemap tile

  • Optional alternate layout of the main window: Tree view and treemap side by side

  • Improved logging: More secure and rotating logs

  • New document: QDirStat for Headless Servers for server admins

  • New document: The GitHub Workflow for developers

Details

New Features

File Type Statistics Window

People who know WinDirStat inevitably want its File Type view in QDirStat, too. I was really reluctant to do that because I didn't quite see the point; in WinDirStat, it serves mostly as a legend to the treemap colors since they are constantly changing in WinDirStat: The file type that consumes most disk space always gets color #1, the next-most color #2 etc., so it depends which directory you scan what color each file type gets. In QDirStat, the colors are stable; they are predefined and configurable in the MIME Type Categories configuration dialog.

And as most of you probably know, filename extensions have a much stricter meaning in Windows than on Linux/Unix systems; Linux people get very creative when it comes to using dots in filenames. Sometimes those dots delimit a filename's extension (suffix) from its base name, sometimes they are used for entirely different purposes.

But there was one user who was insistent enough to make me reconsider and convinced me with valid use cases for this, and after some experimenting I found an approach that works for QDirStat -- both technically and in terms of usability:

File Type Statistics Window Screenshot

So it's not just a flat list of filename extensions (suffixes); rather, they are sorted into their respective MIME categories (yes, the ones you might know from the Treemap Colors config dialog) where appropriate and Other for everything that doesn't fit into a known category.

See the whole discussion with more screenshots at GitHub issue #45.

Limitations

Since filename extensions (suffixes) don't have as much semantics in Linux/Unix systems as they do in Windows, many files are categorized as "Other". This is a known limitation, but it's a limitation of the whole concept of using suffixes to categorize files by type. And no, checking file headers for magic byte sequences like the "file" command does is not an option here; QDirStat would have to do that for (at least) all the 30,000+ files typically listed under the "Other" category.

So we'll have to live with that limitation. More exactly, everybody will have to live with this; I will not spend hours upon hours anymore explaining that same thing to the umpteenth user demanding this to be more sophisticated.

This will also not become part of the main window (as was suggested before) because it just takes up too much space, and it would clutter the main window too much; See also GibHub issue #52 and GitHub issue #51.

Locating Files by Type

Locating FilesWindow

You can now locate files with a specific filename extension directly:

  • You select a file type (a filename extension) in the "File Type Statistics" window.

  • You click "Locate" or you double-click the item.

  • The "Locate Files" window opens.

  • You click a directory there.

  • In the main window, the branch for that directory opens, and all matching files are selected in the tree view and in the treemap.

  • You can now directly start cleanup actions for those files.

See also GitHub issue #48.

The File Type Statistics and the Locate window are both non-modal, i.e. you can still work in the QDirStat main window while they are open, and they both remember their size and position so you can arrange them on the screen to have everything visible at the same time that you need to see.

Improved Logging

Logging is now more secure, and the logs are automatically rotated, so there is no longer one large log file that keeps growing and growing.

QDirStat now uses its own log directory /tmp/qdirstat-$USER (where $USER is your user name; the numeric user ID is now only used if the user name cannot be obtained). It no longer keeps one single log file growing, but starts a new one each time it is started. 3 old logs are kept; any older ones are deleted.

The permissions for that directory are set up in a pretty restrictive way (0700, i.e. rwx------) when it is created. If it already exists, QDirStat checks the owner and creates a new one with a random name if it is owned by anyone else than the user who started QDirStat.

[sh @ balrog] ~ 68 % ls -ld /tmp/qdirstat-sh
drwx------ 2 sh sh 4096 Feb 24 18:29 /tmp/qdirstat-sh
[sh @ balrog] ~ 69 % ls -l /tmp/qdirstat-sh
total 16
-rw-rw-r-- 1 sh sh 2067 Feb 24 18:29 qdirstat-00.old
-rw-rw-r-- 1 sh sh 2067 Feb 24 18:07 qdirstat-01.old
-rw-rw-r-- 1 sh sh 2067 Feb 24 18:07 qdirstat-02.old
-rw-rw-r-- 1 sh sh 2067 Feb 24 18:29 qdirstat.log

For anybody regularly watching the log file this means they will now have to use tail -F qdirstat.log rather than tail -f since the latter does not realize when the file it watches is renamed and a new one is created under the same name.

Other New Features

  • Implemented GitHub issue #30:

    When hovering over a treemap tile, display the full path and the total size of that element in the status bar. When the hover ends (when the mouse cursor leaves the treemap tile), display the current selection again in the status bar.

  • Optional alternate layout for the main window: You can now switch from the traditional layout of tree view above and treemap below to a side-by-side layout with the tree view on the left and the treemap on the right (Menu Treemap -> Treemap as Side Panel).

    This was contributed by @flurbius.

New Documents

  • QDirStat for Headless Servers

    This document explains step by step how to use the qdirstat-cache-writer Perl script to collect data on your headless (without X server, without X libs) server somewhere in a rack in a data center and display the data on your local workstation machine with QDirStat.

    This information has been out there for at least 10 years, but still people keep asking that on sites like stackexchange.com and superuser.com; I lost count how many times I answered it. And still people come up with the most crazy and arcane command lines to do a shitty job for the same problem when they could also have the good solution.

  • The GitHub Workflow

    This document is a step-by-step instruction guide how to collaborate with projects such as QDirStat (but also other Open Source projects) with GitHub and git with upstream repositories and your personal fork and local working copies; how to set it up, how to fetch and rebase, how to work with pull requests and getting your merged contributions from upstream back to your fork.

    I wrote that when I caught myself explaining that in great detail in a pull request comment; I thought that might be useful for a lot more people out there. So here it is.

Bugfixes

  • Fixed GitHub issue #24:

    During directory reading, subdirectories would get out of sync when opening a tree branch.

    It looks like QDirStat's tree display was a bit too dynamic for the concepts of the underlying Qt classes (QTreeView / QAbstractItemModel): During reading, QDirStat would sort the tree by the number of pending read jobs. That number is constantly changing, so the sort order would also constantly need to change. This is very hard to do properly with the limitations those underlying classes impose; basically it would require a reset of all the data the QTreeView keeps, thus making it forget things like its current scrollbar position or which tree branches were expanded or collapsed. That would make the user interface pretty much unusable.

    So the fix for this is to not sort by read jobs, but by directory names instead since they don't change all the time. The user can still sort by any other column, but that sort is a momentary thing that might become invalid moments later as data (accumulated sizes, number of child items) are updated. Everybody please notice that this is a known limitation and any complaints about that will flatly be rejected. The alternative would be to not allow the user to sort at all during directory reading, and that is certainly a lot less desirable.

  • Fixed GitHub issue #54:

    Crash with cleanup action with "Refresh Policy: Refresh this item" when more than one file item was selected.

  • Fixed crash upon refresh when deleting the toplevel directory via cleanup.

Acknowledgements

This release was supported by my employer SUSE Linux GmbH with the SUSE Hack Week where R&D people are free to spend one whole working week on their favourite Open Source projects. SUSE does this roughly every 6-8 months to support the Open Source movement.

About one third of my time working on this release was covered by the Hack Week.

@shundhammer shundhammer released this Jan 3, 2017 · 242 commits to master since this release

Assets 2

Release Announcement

The QDirStat project proudly presents its latest release: QDirStat 1.2.

This release is focused on Btrfs improvements, so upgrading to this release is highly recommended for Btrfs users:

If you used QDirStat to scan a Btrfs partition, any subvolumes of that partition were not scanned (see GitHub issue #39).

Btrfs subvolumes were treated just like ordinary mount points (which, to all intents and purposes, they are). So you might have wondered why the df command shows your 40 GB root filesystem as 97% full, yet QDirStat shows only about 7 GB. The rest might be hidden in subvolumes.

QDirStat stops reading at mount points - which only makes sense because normally you want to know what eats up the disk space on that one partition that is filling up, not on any others like /home that are mounted there. Unfortunately, a Btrfs subvolume is also just another mount point, and QDirStat would stop reading there, too - at /var/log, at /var/spool, at /var/lib/libvirt etc.; a typical Btrfs root filesystem has about a dozen subvolumes, and all files in them were disregarded by QDirStat.

This is now fixed: Despite Btrfs doing its best to make this difficult (using one single privileged system call for all its functionality, including simple info calls), QDirStat now detects if a mount point is a Btrfs subvolume and continues reading if it is. QDirStat uses /proc/mounts (or, if this is not available, /etc/mtab) to find this out.

This is fixed in the qdirstat-cache-writer script, too.

Other Changes

  • Bernhard Walle contributed some patches for MacOS X support.
    Thanks, Bernhard!

  • Fixed GitHub issue #40:
    Crash without useful error message when no display available

    When ssh'ing without -X to a remote machine and starting QDirStat there, it would just dump core and not issue any meaningful message. The fatal error message was only in the log file:

    <ERROR> :0 (): QXcbConnection: Could not connect to display

    Now this message is also repeated on stderr, and in this particular case ("Could not connect to display"), it does not dump core any more, but just exits with error code 1.

About QDirStat

QDirStat is a graphical application to show where your disk space has gone and to help you to clean it up.

This is a Qt-only port of the old Qt3/KDE3-based KDirStat, now based on the latest Qt 5. It does not need any KDE libs or infrastructure. It runs on every X11-based desktop on Linux, BSD and other Unix-like systems.

QDirStat has a number of new features compared to KDirStat. To name a few:

  • Multi-selection in both the tree and the treemap.
  • Unlimited number of user-defined cleanup actions.
  • Properly show errors of cleanup actions (and their output, if desired).
  • File categories (MIME types) and their treemap color are now configurable.
  • Exclude rules for directories are easily configurable.
  • Desktop-agnostic; no longer relies on KDE or any other specific desktop.

Target Platforms

  • Linux
  • BSD
  • Unix-like systems

Target Audience

  • System administrators
  • Advanced users
  • Interested users who are not afraid of learning new things

Screenshots

Main Window Screenshot

Main window screenshot - notice the multi-selection in the tree and the treemap

Cleanup Action Output Screenshot

Screenshot of output during cleanup actions. Of course this window is purely optional.

Cleanup Action Configuration Screenshot

Screenshot of cleanup configuration.

MIME Categories Configuration Screenshot

Screenshot of MIME category configuration where you can set the treemap colors for different file types (MIME types), complete with a real treemap widget as preview.

Exclude Rules Configuration Screenshot

Screenshot of the exclude rules configuration where you can define rules which directories to exclude when reading directory trees.

More Information

https://github.com/shundhammer/qdirstat

@shundhammer shundhammer released this Oct 31, 2016 · 269 commits to master since this release

Assets 2

Release Announcement

The QDirStat project proudly presents its latest release: QDirStat 1.1-Pumpkin.

This is mostly a bug fix release with some small features thrown in.

It was about time for another official release to get the accumulated fixes and
small changes out into the world. Since the release day was Halloween, this release is named Pumpkin .

The last stable release, V1.0, was in mid-May (2016-05-16). Since then, there
were 5 bug fixes and one small feature (the config file split up into
independent parts so admins can provide presets to their users without
overwriting the complete configuration), all described in greater detail
below.

Changes

  • Fixed GitHub issue #32:
    %p does not escape single quotes properly

    If you have a file name like Don't do this.txt (with a quote character in
    the name), the shell used when executing a cleanup action with this would
    complain about unmatched single quotes.

    QDirStat had always escaped such single quotes, but not the way common
    shells (Bash, Zsh) expect it: They don't want a backslash in front of that
    embedded single quote. Rather, you need to terminate the string with a
    single quote, escape the embedded quote with a backslash (or put it into
    double quotes), and then re-open the old string with another single quote.

    Thus, 'Don't do this' becomes 'Don'\''t do this'.

    This is certainly not what most people expect. I just wonder how much other
    software is out there that does it the intuitive (yet wrong) way: Just
    escape the single quote with a backslash ('Don\'t do this').

    Of course, such file names should be avoided entirely, but you can't help
    some slightly broken MP3 ripper program doing it, so it needs to be handled
    correctly.

  • Fixed GitHub issue #31:
    Segfault with cleanup action while reading directories

    Now disabling cleanups that have a refresh policy other than "No Refresh"
    while directory reading is in progress; otherwise the re-read when the
    cleanup action has finished clashes with the directory read already in
    progress.

    This is not an optimal solution, but a very pragmatic one; the optimal
    solution might queue updates and execute them after the main read is done.

  • Fixed GitHub issue #33:
    Added command line option --slow-update (or -s) for slow remote X connections.

  • Fixed GitHub issue #23:

    The internal cache writer would sometimes generate incorrect cache files
    because of buggy URL escaping resulting in an empty file name and thus
    invalid cache file syntax. This affected file names with colons (which is
    weird, but legal).

    One of these days I'm going to throw out all that QUrl stuff and replace the
    few things that I need with something that actually works consistently and
    not just under optimum conditions.

  • Fixed GitHub issue #22:

    Cache files containing the root file system would not display correctly or
    segfault under certain conditions. This is now fixed.

  • Added "Refresh All" action to the main window tool bar. I had consciously
    avoided that because it's just too tempting to re-read the complete
    directory tree rather than think about what actually might have changed and
    then refresh just that, but it has become so common to use that action in
    web browsers that I found myself missing that more and more. And re-reading
    is not that expensive on today's mainstream PCs.

  • Fixed GitHub issue #21:

    When started from a desktop menu, i.e. without any command line parameters,
    QDirStat would not prompt for a directory to read, but read the current
    directory (typically the user's home directory) right away.

  • More graceful handling for nonexisting paths specified on the commmand
    line: It now no longer just throws an exception right after starting the
    program (which looks like a crash to the unwary user), but posts an error
    popup instead and then asks for a directory to read.

  • Split up config file into four separate ones below ~/.config/QDirStat:

    • QDirStat.conf
    • QDirStat-cleanup.conf
    • QDirStat-exclude.conf
    • QDirStat-mime.conf

    This should make it much easier for site administrators to provide their
    own site-wide cleanup actions, exclude rules, or MIME categories. I did
    this with this in mind:

    http://moo.nac.uci.edu/~hjm/kdirstat/kdirstat-for-clusters.html

    Here, they describe how users should overwrite their KDirStat config file
    with one provided by the site admin so all users have those carefully
    crafted cleanup actions. But that also means that all other settings get
    lost each time there is a change in any of those commands, and users have
    to update that config file again.

    With the latest change, it is now possible to only replace the cleanup
    action config (QDirStat-cleanup.conf) and leave everything else untouched.

    Notice that this is far from a perfect solution; all cleanup actions the
    user added himself still get lost. But doing this perfectly might pretty
    quickly become an overengineered solution that would be hard to understand
    for everybody.

    As for migration from previous single-file configurations, QDirStat does
    that automatically: It reads the single file and moves the respective parts
    where they belong. No need to bother with any migration scrips or anything
    like that.

@shundhammer shundhammer released this May 16, 2016 · 305 commits to master since this release

Assets 2

Release Announcement

The QDirStat project proudly presents its first official stable release: QDirStat 1.0.

In terms of source code, there were very little changes from the last Beta (0.98-Beta3 from 2016-04-08) and no real code change (only the version number increased) from the last check-in from 2016-04-11.

This version can really be considered stable in the truest sense of the word. It was not rushed out the door, and there were no hectic last minute changes. It is well tested, and the community had ample opportunity to report any problems.

More Information

https://github.com/shundhammer/qdirstat

@shundhammer shundhammer released this Apr 8, 2016 · 310 commits to master since this release

Assets 2

QDirStat 0.98-Beta3

Changes

  • Fixed GitHub issue #15:

    After a cleanup action is executed that needs refreshing the affected
    subtree, the parent directory is selected, which is intentional so the
    user's focus is not thrown off completely. There was a bug when you
    selected an item in the treemap afterwards, that change was not correctly
    propagated to the internal selection model: The parent directory remained
    selected (which was wrong), and the newly selected item was just added to
    the selection, i.e. that item and (typically) its parent directory was
    selected. When a potentially dangerous cleanup operation was now started,
    it would affect not only that item, but also the directory; and, worse,
    that directory often went out of the visible scope of the tree view. Yes,
    the confirmation popup would ask for both of them, but you all know how
    quickly users click away those popups without really reading them.
    This bug is now fixed.

  • Improved the confirmation popup. Now highlighting directories much more if
    there is a "mixed" selection, i.e., both directories and non-directories
    are selected at the same time:

    New cleanup confirmation popup

@shundhammer shundhammer released this Mar 20, 2016 · 319 commits to master since this release

Assets 2

QDirStat 0.92-Beta2

After 6 weeks of Beta 1, Beta 2 of QDirStat is now released.

Changes

  • Minor fixes
  • Documentation updates
  • Support for Debian and Ubuntu packaging contributed by Nathan Rennie-Waldock
  • Links to Nathan's Ubuntu PPA for QDirStat

@shundhammer shundhammer released this Feb 13, 2016 · 329 commits to master since this release

Assets 2

QDirStat 0.86-Beta1 announced

The QDirStat authors proudly present the first official public release of QDirStat, the brand new KDirStat replacement without any KDE dependencies.

Target Platforms: Linux, BSD, Unix-like systems

https://github.com/shundhammer/qdirstat

QDirStat is based on the code of the KDE 3 based KDirStat. It is an 80% rewrite, now based on the latest Qt5. QDirStat has a number of new features compared to KDirStat. To name a few:

  • Multi-selection in both the tree and the treemap.
  • Unlimited number of user-defined cleanup actions.
  • Properly show errors of cleanup actions (and their output, if desired).
  • File categories (MIME types) and their treemap color are now configurable.
  • Exclude rules for directories are easily configurable.
  • Desktop-agnostic; no longer relies on KDE or any other specific desktop.