Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement du shell command #277

Open
humphd opened this issue Sep 4, 2014 · 4 comments

Comments

Projects
None yet
2 participants
@humphd
Copy link
Contributor

commented Sep 4, 2014

NOTE: I'd like to do this as an exercise with my class, so please don't fix this :) Assigning to myself.

It would be helpful to know how much space a filesystem is occupying in storage. Unix has a way to do this already with du. We should have du in Filer's shell so we can get this info. See the du specification here:

The code to do this is similar to sh.ls (i.e., we need to use async to visit all the nodes recursively, and check their size property--don't visit the data blocks). We don't need to support all the various flags, and I maybe it's logical to always default to reporting sizes in bytes or a human-readable unit (K, M, G).

My initial proposal is as follows:

var sh = fs.shell();

// Example 1: du called on a file (or symlink'ed file) path
sh.du('/path/to/file', function(err, sizes) {
  if(err) throw err;
  // sizes is an object with a total size, and sizes for each entry:
  //  {
  //    total: 1024,
  //    entries: [
  //      {'path/to/file', 1024}
  //    ]
  //  }
});

// Example 2: du called on a dir (or symlink'ed dir) path
sh.du('/path/to/dir', function(err, sizes) {
  if(err) throw err;
  // sizes is an array with one element per child-entry, the file's path and size:
  // sizes is an object with a total size, and sizes for each entry:
  //  {
  //    total: 10345,
  //    entries: [
  //      {'path/to/dir/file1', 10},
  //      {'path/to/dir/file2', 103},
  //      {'path/to/dir/file3', 8210},
  //      {'path/to/dir/child-dir/file1', 10},
  //      {'path/to/dir/child-dir', 15},
  //      ...
  //    ]
  //  }
});

Obtaining the total usage for the filesystem would be done with sh.du('/', function(err, sizes){...}).

When called on a directory path, the entries array is a recursive, flattened, depth-first list of all entries contained in the directory, followed by the directory itself. The size for a directory is the sum total of all sizes for its children, recursively.

@humphd humphd self-assigned this Sep 4, 2014

@humphd

This comment has been minimized.

Copy link
Contributor Author

commented Sep 5, 2014

Fixing this bug requires at least the following:

  • Updates to README.md to include details of how sh.du works
  • Tests for the various ways sh.du can be called (on file, on empty dir, on dir with children, on symlink, on symlink to dir, ...).
  • Implementation of du

It is suggested that students work "top down" in the order described above (docs > tests > impl).

@modeswitch

This comment has been minimized.

Copy link
Member

commented Sep 5, 2014

At present, there's no way to measure the size of fs metadata, so the result will be wrong by that amount. You should be able to get everything you need using stat though. As a side-note, switching to binary metadata would allow us to measure everything.

@humphd

This comment has been minimized.

Copy link
Contributor Author

commented Sep 5, 2014

Yeah, good points. It think this is OK (metadata is of unknown, but mostly constant size) since du is mostly designed to give an estimate of disk usage. The real issue one is going to care about, and which can vary so widely, is the size of the data blocks.

@modeswitch

This comment has been minimized.

Copy link
Member

commented Sep 5, 2014

metadata is of unknown, but mostly constant size

Sortof. Extensive xattrs can make this much larger. Refactoring metadata and xattr data is on my list.

@humphd humphd removed their assignment Sep 18, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 24, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 24, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 24, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 24, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 24, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 24, 2014

jlaverty added a commit to jlaverty/filer that referenced this issue Sep 25, 2014

rdaoud1 added a commit to rdaoud1/filer that referenced this issue Sep 25, 2014

jlaverty added a commit to jlaverty/filer that referenced this issue Sep 25, 2014

yoavgurevich added a commit to yoavgurevich/filer that referenced this issue Sep 26, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 27, 2014

gideonthomas added a commit to gideonthomas/filer that referenced this issue Sep 27, 2014

@modeswitch modeswitch added this to the 0.2 milestone Nov 14, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.